Services
This is a post about how to get the most out of a penetration test and what to consider while you are in the buying process evaluating vendors.
It is inspired by a recurring theme with our penetration tests where a client will say: "Wow, you are amazing! Our past vendor didn't find all that!" Even if my first gut reaction is pride in our team and our process, I always have mixed feelings when I hear that.
On the one hand, penetration testing is always time scoped and therefore resource limited. There are always going to be people that are better at finding one thing and less successful finding other things. For good vendors it is a highly skilled human centered activity and it is natural that there may be things we find that the previous person didn't find - even if they were very very good! I'm sure sometimes firms follow us and find things that we missed.
On the other hand, most of the time I hear this comment it is because the client's last vendor was not very good and they just ran some scans and delivered the report from the scans. It's not that hard to tell this was the case, nor is it hard to avoid this type of vendor - and that is basically what this post is about.
Ultimately, because we provide excellent penetration testing, we want our clients to be informed about the buying process, the different issues that arise with vendors and what makes for a successful pen testing engagement.
Lots of folks before me have said this, so let's just repeat it here:
A scan is not a penetration test.
Lots of reputable people in the security industry.
If you walk away with nothing else from this post and you're going to go procure a pentest, just make sure you get an actual pentest and not a scan. We can do a scan too - but we call it a scan and we charge a fraction of the money to do it because it costs us very little to do it. It is so frustrating to follow a company (many are well known and trusted) that just ran a scan but charged even more than we did! Not to mention the injustice of leaving all of these low hanging fruit vulnerabilities in the client's application based on a misunderstanding of the task at hand.
When we scope penetration tests, because we are spending a lot of human time looking at the system, request payloads and other parts of the system, we care about a bunch of things. Understanding scoping can help you make sure you get the coverage you want.
How many roles are there in the application? The more roles there are, the more different types of authorization gaps there could be. Consider that as part of a penetration test, we're going to replay requests to ensure that User A can't see User B's data. We're also going to test that User A can't perform Manager C's function. The more different roles there are (User / Manager here), the more scenarios there are to test.
Another consideration is the number of screens or in an SPA the number of interaction points. The reason this matters is that each interaction point takes a chunk of time. If you are technical, you could think about each of these as an endpoint on an API or a POST. But the idea is that each interaction needs to be evaluated in a bunch of ways to identify issues. So the more interactions there are, the more time it will take.
Similarly, the complexity of the business function can add to the time it takes to perform a test. As can the number and types of fraud paths. In a recent scoping call, we saw that the application was 98% read only. That dramatically reduces the attack surface of the test.
To summarize the scoping consideration, we might ask:
Most of the tests we do are just one, two or three weeks and focus in on web applications and API's. We offer clients a choice of size because if they perceive high risk or want to go deeper, they should have that option. If they have limited budget and want to get the best test they can in a small scope, we try to accommodate that as well. If a client wants us to spend too small an amount of time on a large app, we reserve the right to decline because we just can't provide meaningful coverage.
Our goal is to scope so that clients get good coverage and check the things that need to be checked, without passing the point of diminishing returns where we're spending more time but not finding more things. There's no way to be sure you've covered everything and there's no certain way to know when you get to this point. Our method is based on a decade of real world experience.
We generally recommend allocating a healthy amount of time and budget for penetration testing in the spirit of conducting serious testing with a real goal of identifying real issues so they can be fixed.
A clean test can be a bad result because it doesn't signal thorough evaluation. In our experience, partners, vendors and auditors are generally ok with tests that have serious findings provided that an organization took appropriate action based on the findings and confirmed that they had been fixed.
When we do penetration testing, our goal is to provide everything you need to get everything possible out of the test. Most people know they need a report. The report is a large PDF (we just delivered one that was 107 pages) summarizing the findings and covering details about how to reproduce each finding and ideally how to fix them. We usually conduct a Read Out call with the client team to explain everything in the report and answer questions. The Read Out goes best when clients have had time to digest the report and ask questions about the detail. Often the preparation isn't complete and we just walk through the report to highlight important items.
When it comes to the report, even if you think you don't care, you probably need the report. Every once in a while we will do testing where the client doesn't think they need a formal report for audit purposes, they just care about the findings and want us to report the issues directly to Jira. That is fine with us, but usually someone needs to capture the output of the test at a point in time so that when you look back in 2-5 years you can see what you did and what was found back then. The best way to do that is to get the standard report.
That being said, the report is really just the beginning. For one, we don't recommend sharing the pen test report with anyone outside your company unless you have no other choice. Our reports basically always contain sensitive information about your security posture that you can reasonably say cannot be distributed. So what we recommend sharing is a one page executive summary that is truthful and describes the testing that was done, including the numbers of findings and key types of findings. We call this an Attestation LetterAn attestation letter is a document that provides assurance to a third party about the validity and accuracy of certain information or processes related to an organization's cyber security. An attestation letter may be provided by an independent auditor or security expert to confirm that an organization has implemented appropriate security policies, procedures, and training to protect sensitive data and systems.. This is intended to be shared with a third party as needed. It doesn't disclose any truly sensitive information other than the broad outcome of the testing. Usually that by itself isn't sensitive. It is implied that we would have a conversation with a partner if requested around the Attestation Letter but in 10 years nobody has ever asked us to do that.
We also include all of the raw evidence that is packagable in a supporting evidence folder. This includes the output of tools like Burp Suite, Amass, bucketfinder, nmap, dnstwist, sqlmap, etc. as we use them. We want developers that are interested to see the tools we used to collect the information. If you can do all of that part yourself, that is great. That being said, people rarely ask us anything about the raw evidence. Turns out understanding what those tools do requires some pretty expert level security knowledge...
One thing that is very informative is to look at the difference between the raw scan results in the evidence folder and the findings in the report. Since we find most of our most serious issues during manual testing, they don't show up in the tool reports. If your report matches the scan, you probably didn't get much manual analysis. Of course, vendors know this too so they don't provide the raw evidence - and on some level, you can't know that they provided all of it anyway.
After we do the readout call, most clients want to fix High and Medium severity findings and get an updated Report and Attestation Letter that reflects their progress. To do this, we need to Retest key items to confirm that the client has in fact fixed them. We don't remove things from the report but we do update the report to reflect that the item has been retested and confirmed to be fixed. We do this and included it in all of our pricing as part of something you always get - because we know most people will want it and we don't want to surprise you later with additional charges just because you didn't know to ask up front. We don't charge more for retesting or re-issuing a letter. We limit the time to retest to 90 days (with exceptions) to try to ensure that we have retestable scenarios. As code bases change, directly retesting an item can become impossible.
Summary of Deliverables:
There are a number of questions you can ask when you are looking at penetration testing vendors.
One thing that is important as you compare options is that you have a clear statement of Scope so that the different vendors you talk to are quoting on the same thing.
There are a few topics that are relevant and that I've gotten familiar with in the process of doing penetration testing for the last 10+ years as Jemurai and 4+ years before that I would like to call out a bit even though they are rarely talked about out loud.
Obviously it is in the pentest vendor's interest to have the least experienced, least highly paid person possible do the work. It is also in their interest to have a tool do most of the work for them. Pure penetration testing is a utilization business. The owners of the company get rich when utilization is high and resources are cheap.
When testing is heavily tools based, you might have a pen tester doing more than one test at once. If a tester can do more than one test at once, they can bill more clients per week. Of course, if the test is driven by expert human testers looking at how to abuse a system they can only focus on one client at a time. As a consumer, you are entitled to know which you are getting but you are unlikely to get a straight answer - so you'll have to ask good questions.
Sometimes people ask about "bug bounty" or if they can do "continuous pentesting". Frankly, I think this industry is fraught with misinformation. I have seen vendors in this space grandstand about "activity" based metrics where they produced millions of interactions over a year and had 600 researchers involved in a program. The problem is with incentives the way they are, you have 600 people that are mildly interested for a half day that all check the same things. Nobody is paying more for higher levels of attention or expertise in this arena. This doesn't do anything great for your security! My experience is that a good solid penetration test gives you much much better results than these types of programs. I'm not saying you shouldn't do bug bounty or even continuous pentesting, just know what you are paying for, what you are getting, where it fits in your program and that it is fundamentally different from penetration testing.
The goal of this post was to demystify the penetration testing process so that you can navigate it as smoothly as possible and get the most thorough useful test possible for your budget. We've written about similar things about The Truth About Audits on securityprogram.io. At the very least, we hope this information will help make sure you are getting what you are paying for - and not just a scan.
