This is the second in a series of posts looking at the current state of pen testing as I see it and presenting some ideas for the future. Part one is available here.
In this post I will explore some of the issues I see in pen testing, it’s something of a rant that I have regaled a number of pen testing friends with over the last couple of years. If you disagree violently with this let me know, if I’ve missed something let me know, we need to open up this conversation in the industry.
In the next post I start exploring why these problems exist and how they might be improved.
What is wrong with pentesting?
As an informed customer, and an ex-pentester, I see a number of problems with pentesting as delivered today;
- Too much focus on 0day as a measure of success
- Too much variation in quality and coverage between testers and between tests
- Too much unexplained and undefined ‘black magic’
I think those bullets describe the result of the ‘rock star pen tester’ mythology that has built up in the industry.
I comprehend that understanding a complex system in order to identify opportunities to subvert it and them successfully implementing those opportunities is an inherently creative task that requires great expertise.
However, I would also argue that the majority of pentesting undertaken is a combination of a network scan, a vulnerability scan, an application scan and a little creative chaining together of known platform-level vulnerabilities. There is rarely, in my experience, much in the way of architectural flaws identified and rarely much in the manner of truly innovative custom application attacks.
Much of what is completed isn’t as truly mind-blowing as it is billed, I’m sure many testers would recognise the forced march of a large-scope penetration test with a fixed deadline where the more interesting and unique subsystems are lightly glossed over to ensure as much coverage as possible in the timeframe on the things that can be tested with known tools and existing knowledge. Actually that is pretty valuable to many clients who aren’t trying to defeat advanced persistent foreigners, they’re just trying not to be another number in the ongoing cyber crime wave where making yourself more expensive to attack than the next guy still has some value.
I think the rock star pen tester mythos has distracted the pen testers and that they have little awareness of the outcomes the client is looking to achieve by conducting penetration testing, confusing the outcomes valued by the pen testing industry with the values valued by the customers.
I do understand this is a very broad brush generalisation, there is a growing awareness of this issue amongst some of the more experienced testers and there are exceptions in the industry but they are not that common and all the more valuable when you find them.
What is wrong with pen test customers?
There is a common refrain amongst pentesters after a repeat test that either the customers never fix the problems they’re told about or else the right people never read the report which languishes in some middle management audit function rather than influencing senior management and empowering IT staff to act.
I think there are a lot of customers that are way out of their depth when it comes to IT security and technical security testing but that isn’t the whole story, I think there are some fundamental issues in the pen testing process that I’ll address later.
Many customers have also bought into the industry ‘rock star pen tester’ mythology which confuses the issue as they also fall into fetishising 0days over the outcomes.
There are also many customers who are only buying pen tests because a different and powerful part of their business has told them they must. Their main interest being ticking the box to say such a test has been completed than any interest in the quality or coverage of the testing. These clients will happily pay less for a test that to the non-expert eye provides the same value but to the rest of us is barely worth the paper it’s printed on.
I think the ‘best practice’ of rotating through a number of pen testing companies by larger customers is an unfortunate response to the variance in quality and coverage. In order to minimise the chance that a particular pen tester or pen test company is providing bad quality or poor coverage it is common for a customer to switch pen test companies on a regular basis. This fails to address the issue of quality or coverage directly and actually harms the quality of the final report as pen testers have less opportunity to build up the experience and understanding of the systems they are testing.
Update: I must mention Haroon Meer’s ‘Penetration testing Considered Harmful’ presentation from 44con in 2011. Absolutely worth watching the video and downloading the slides if you have any concerns about the current state of pen testing.
As I described it recently, annual pentests without ongoing risk assessment is like not brushing your teeth and visiting the dentist once a year. But if the point is that pentesting alone won’t provide assurance (and I think that’s what it comes down to), it really doesn’t help that even the pen testing industry is kind of misinforming itself on its own role sometimes (caveat: I haven’t read that book and I musn’t judge a book by its cover, however …).
I’ve recently rejected a business case for a 0day detection module from a vendor because it looked and sounded too much like ‘heuristics, signatures and witchcraft’.
The customer is always right too! Oh yes she is. If that boxticking exercise wasn’t a requirement from senior management in some places, would we get better security?
Excuse the flippant comment, but good security analogies in infosec are hard to find, but “annual pentests without ongoing risk assessment is like not brushing your teeth and visiting the dentist once a year” is now in my repertoire.
Mainly I just wanted to say what a well written piece this is, and that you’ve illuminated a known issue very well.
However is this situation likely to change? Pentesters are generally interested in the more adventurous parts of their work, as are customers; no-one tells war stories about how they enumerated all the parameters on an entire website. Pentesting is a meritocracy based on your ability to work against systems, rather than on your ability to work for customers, so that will affect intake and time spent appropriately.
By the way “more valuable when you find them” is a reassuring comment, as I like to think I’m one of those exceptions, but I’ve yet to figure out a way to monetize that appropriately…
As for the issue of rotating companies, I always thought this was recommended in order to have fresh eyes looking at the “problem” of the target network, rather than using the same testing company each year with the the same tools, techniques and methodologies as were used the year before. I’ve always thought the ideal method was to use two companies, rotating one of them each year, so you have the combination of experience of your systems, and those fresh eyes… of course, as well as other issues, this does cost twice as much…
I have some ideas how this might change but I think the industry itself either has to lead the change or be disrupted from outside.
I don’t know which way it will go but the status quo is not working well and the growing number of pen testers making the jump to the customer side of the fence means the customers are increasingly going to become more demanding.
One other thing – there seems to be a growing push for introduction of and automation in vulnerability detection in commercial and some HMG organisations. Many are SaaS (which might not always fit well) but they all open up CVE in some way to give the customer intelligent data about their environment’s exploitability.
What does this mean for pentesters? Well I think its good and bad news. Bad for the boxticking outfits, good for the better companies that add value through crafted attacks or by attack path analysis etc.
The problem with these tools is they cannot provide any insight only an expert schema of gathered information.
Useful if you have a skilled and experienced person in house to analyse and understand but not so much use if you don’t.