Services
This post talks about the good, bad and ugly that we see around cybersecurity audits. It is informed by 4 years working for a company deeply embedded with PCI Compliance, and then about 8-10 years of experience helping dozens of companies with SOC 2SOC 2 is a type of Service Organization Control (SOC) report that evaluates the effectiveness of an organization's controls related to security, availability, processing integrity, confidentiality, and privacy. These reports are based on the Trust Services Criteria (TSC) developed by the AICPA and are commonly used by technology and cloud service providers to demonstrate compliance with industry standards and best practices., ISO 27001ISO 27001 is a globally recognized standard for information security management systems (ISMS). It provides a systematic approach to managing sensitive company information and mitigating cybersecurity risks. ISO 27001 includes a set of best practices for establishing, implementing, maintaining, and continually improving an ISMS, which helps organizations identify and manage information security risks and protect against cyber attacks., NISTNIST stands for the National Institute of Standards and Technology, which is a US government agency responsible for developing and publishing standards and guidelines related to information security and cybersecurity. 800-171A and other similar audits.
The TL;DR is that audits can be helpful for improving an organization's security posture. On the other hand, an audit doesn't prove security and the more you game the audit the less likely it is that the audit improves your security. Furthermore, unfortunately, most audits we have seen are inconsistent across auditing firms and don't mean what you might think they mean.
Audits force an organization to do some introspection. If you identify appropriate "controls" to check, then the audit activity can help you to make sure you are doing the right things.
For example, if you say you have endpoint protection in place then the auditor may ask for proof. While producing the proof, you may realize that Mac and Linux devices were not actually covered by the tool you had in mind. That might lead you to use a solution that works with those devices, reducing your actual risk materially.
In some cases, the audit introduce controls you didn't know you needed to have in place! This can be extremely helpful for organizations that haven't oriented to a standard (see: why we use NIST 800-53 or which security standard should I use? for more information) and may not have identified broad sets of controls they want to have in place.
Sometimes an audit just tells you what you need to know, but from an outside party. Well, if the respected expert XYZ firm says we need to do it, then we can justify the expense to upper management. This just happens sometimes.
So in some ways, audits can really help organizations improve their security.
For every customer we have that has had good experiences with audits, we have at least one that has had bad experiences with audits. That's < 50% positive feedback.
One huge problem is defining the audit Scope. I've seen audits where the auditors review all of the internal corporate IT as if internet facing applications didn't exist and sign off on an audit because the correct controls were in place for internal email and file sharing.
I've seen firms scope an audit a certain way and then come in and bleed customers after an initial audit agreement based on their lack of alignment to the controls.
I've also seen the opposite, audit firms that come in and charge a very low fee and basically check nothing.
Another problem with audits is that most of them aren't pass / fail and nobody reads audit reports. This means that so long as you can produce an audit report (eg. SOCService Organization Controls (SOC) are a set of standards developed by the American Institute of Certified Public Accountants (AICPA) to help organizations assess and report on the effectiveness of their internal controls. SOC reports provide assurance to customers and stakeholders that service organizations have appropriate controls in place to protect sensitive data and assets. 2), you can usually win business, even if the report says you didn't cover the main application the reviewer cares about or even if you had significant findings. Only very invested companies with strong teams are able to do this justice - and even then, they are subject to the whims of the auditor.
We have worked with a number of audit firms. You might be surprised at how inconsistent the process is. Some firms are quite technical and understand what controls fit in a given environment. Other firms have a list of controls and they are going to stick to them no matter what. Maybe the worst thing is that we've even seen significant variation in experience with different teams from the same firm!
Did you ever stop and think about how some of the most famous hacks were conducted against organizations that had been audited? Consider Target and PCI. Consider almost any recent hacking incident and SOC 2 Type 2SOC 2 Type 2 is a type of Service Organization Control (SOC) report that evaluates the effectiveness of an organization's controls related to security, availability, processing integrity, confidentiality, and privacy over a period of time, typically six to twelve months. These reports are based on the Trust Services Criteria (TSC) developed by the AICPA and cover both the design and operating effectiveness of a service organization's systems, processes, and procedures.. Was Heroku SOC 2 audited? GitHub? Okta?
It reminds me a bit of when I worked for a security company that helped build the PCI Compliance standard. I invite you to think for a moment about how good a business it is to define the standard for compliance, get the card brands to agree to your standard, then define the certification process for assessors, then be the main company that gates compliance to that standard! Can you say boondoggle!? But even PCI was at least a little bit prescriptive.
There is also a growing industry of tools to accelerate compliance. They typically integrate with a bunch of data sources and consolidate information. Whenever I see this, and the idea of automating compliance, I have to say - I do not think that works the way you think it works.
For example, you may have a tool that claims to confirm that you are using a password manager. Great. Does it know which passwords are in scope and that those passwords are stored ONLY in the password manager? Can it tell if you are actively using the password manager?
Let's take another example, suppose a tool tells you your AWS environment is SOC 2 compliant. What does that even mean? Does the tool understand anything about what your dataflows, user lists, and other details are? Do they think they can confirm ALL of your security settings are applied properly AUTOMATICALLY? Consider the question of who has access to what data? I've basically never seen a tool in any security domain that can effectively automate that.
So this whole idea that we can automate compliance is either jaw droppingly naive to me or reflects on the inadequacy of the audits - OR BOTH! And by the way, we build security tools that are directly adjacent to (but different from) both audit compliance (securityprogram.io) and AWS Compliance (jasp.cloud). One thing I will say, I'm all for using API's and open tools like steampipe.io that let you get the data you need for compliance. I will just insist that you have to be able to ask and answer questions and not just run a tool. Some auditors we work with won't accept CLI output as evidence, they need to see the AWS console. How do we reconcile that with a fully automated acceleration tool that is checking a bunch of things that may or may not matter?
So the idea that any audit (and particularly SOC 2) consistently demonstrates security is so obviously far from the truth that I can't believe that "the community" is using it as a standardized baseline these days.
The worst thing is, I don't believe it is ignorance so much as avarice on the part of the auditing community that we see this. Certainly many in the security community know that these audits are fundamentally flawed.
We work with some audit firms that we respect and like a lot. We believe they help our customers improve and hold themselves accountable for their internal security.
We also work with firms and tools that we think don't add a lot of value - let alone security - and which make a lot of work for the organization.
We recommend that you go into these relationships with eyes wide open about what you are trying to get and what your audit actually means based on the price and type of auditor that you have.
That being said, maybe it is time for a new audit standard. Maybe it is time for a better security standard. In either case, we believe collaboration and people will be a foundational part of it.
