Making sense of pen testing, part two

This is the second in a series of posts looking at the current state of pen testing as I see it and presenting some ideas for the future. Part one is available here.

In this post I will explore some of the issues I see in pen testing, it’s something of a rant that I have regaled a number of pen testing friends with over the last couple of years. If you disagree violently with this let me know, if I’ve missed something let me know, we need to open up this conversation in the industry.

In the next post I start exploring why these problems exist and how they might be improved.

What is wrong with pentesting?

As an informed customer, and an ex-pentester, I see a number of problems with pentesting as delivered today;

  • Too much focus on 0day as a measure of success
  • Too much variation in quality and coverage between testers and between tests
  • Too much unexplained and undefined ‘black magic’


I think those bullets describe the result of the ‘rock star pen tester’ mythology that has built up in the industry.

I comprehend that understanding a complex system in order to identify opportunities to subvert it and them successfully implementing those opportunities is an inherently creative task that requires great expertise.

However, I would also argue that the majority of pentesting undertaken is a combination of a network scan, a vulnerability scan, an application scan and a little creative chaining together of known platform-level vulnerabilities. There is rarely, in my experience, much in the way of architectural flaws identified and rarely much in the manner of truly innovative custom application attacks.

Much of what is completed isn’t as truly mind-blowing as it is billed, I’m sure many testers would recognise the forced march of a large-scope penetration test with a fixed deadline where the more interesting and unique subsystems are lightly glossed over to ensure as much coverage as possible in the timeframe on the things that can be tested with known tools and existing knowledge. Actually that is pretty valuable to many clients who aren’t trying to defeat advanced persistent foreigners, they’re just trying not to be another number in the ongoing cyber crime wave where making yourself more expensive to attack than the next guy still has some value.

I think the rock star pen tester mythos has distracted the pen testers and that they have little awareness of the outcomes the client is looking to achieve by conducting penetration testing, confusing the outcomes valued by the pen testing industry with the values valued by the customers.

I do understand this is a very broad brush generalisation, there is a growing awareness of this issue amongst some of the more experienced testers and there are exceptions in the industry but they are not that common and all the more valuable when you find them.

What is wrong with pen test customers?

There is a common refrain amongst pentesters after a repeat test that either the customers never fix the problems they’re told about or else the right people never read the report which languishes in some middle management audit function rather than influencing senior management and empowering IT staff to act.

I think there are a lot of customers that are way out of their depth when it comes to IT security and technical security testing but that isn’t the whole story, I think there are some fundamental issues in the pen testing process that I’ll address later.

Many customers have also bought into the industry ‘rock star pen tester’ mythology which confuses the issue as they also fall into fetishising 0days over the outcomes.

There are also many customers who are only buying pen tests because a different and powerful part of their business has told them they must. Their main interest being ticking the box to say such a test has been completed than any interest in the quality or coverage of the testing. These clients will happily pay less for a test that to the non-expert eye provides the same value but to the rest of us is barely worth the paper it’s printed on.

I think the ‘best practice’ of rotating through a number of pen testing companies by larger customers is an unfortunate response to the variance in quality and coverage. In order to minimise the chance that a particular pen tester or pen test company is providing bad quality or poor coverage it is common for a customer to switch pen test companies on a regular basis. This fails to address the issue of quality or coverage directly and actually harms the quality of the final report as pen testers have less opportunity to build up the experience and understanding of the systems they are testing.

Update: I must mention Haroon Meer’s ‘Penetration testing Considered Harmful’ presentation from 44con in 2011. Absolutely worth watching the video and downloading the slides if you have any concerns about the current state of pen testing.