What I need from pen test reports.

I get a lot of pen test reports to read. They vary from beautifully crafted prose extolling the skilled exploitation of the system by security testing artistes to functional dumps of tool output into a word format by jobbing vulnerability scanners.

Usually I read that report once, I use the summary to know what detail I need to understand and use the the risk or vulnerability tables to pinpoint the urgent issues to fix. Those vulnerability tables are then transfered to spreadsheets where extra columns tracking the management of the issues identified are added and populated.

These spreadsheets become the key tool for managing issues. Usually I manage a spreadsheet per system and include within it other outstanding security risks and issues discovered outside the penetration test affecting that system. The operational security meetings I hold with the senior security managers from my suppliers and the risk management representatives from my stakeholders are driven by the walk through of identified or outstanding issues, calling out those that don’t seem to be getting fixed or are receiving push-back from admin teams in the supplier environments. These meetings also provide stakeholder risk management representatives the opportunity to review proposed resolutions to see if their risks are being well managed (It is after all their risk not mine).

I remember lots of discussions when I was on the testing side of the industry trying to divine what customers actually wanted in their reports, usuall by digging through the guts of a failed or failing client relationship where technical reports ended up in auditors hands or vice versa.

Now I am an end user of these reports it is clear what I need:

    1) A high level summary that tells me the general feel for state of the system security is. I need this to answer the question; am I looking at a fundamental flaw stopping go-live or just a generally shoddy build process that needs tidying up? This needs to be aimed at a business user as that’s where this gets sent after I read it. If you’re going to add nice graphs or some sort of visualisation do it here (And read Edward Tufte first!)
    2) A tabular list of issues including a title, a description, a list of hosts affected, an estimation of how bad it is and a suggested fix or workaround. If you can give this to me in a spreadsheet format you’ve just bought me time and my gratitude.

The estimation of how bad an issue is could be as simple as high/medium/low, more actionable data for me to work with would include ease of exploit, complexity to fix and the extent of resulting compromise. Ease of exploit plays back into my risk model so I can see if I need to worry about the issue, complexity to fix plays into my remediation plan and the risk management decision on going live and the extent of resulting compromise is used to drive behaviours from non-security team members who don’t get the problem.

I hope the key message for this blog post is I need to fix or manage the problems found in pen tests and everything that makes my job easier means you were more effective at your job.

That’s what I need from a pen test report.

What I want is pretty much pie-in-the sky for pen tests these days but it would be great to get a test script describing how a particular issue was found. Which tool was used? Was it confirmed with multiple tools / techniques? If we want to see if we fixed it what command line switches do we need?

A major bonus for me would be an archive of the shell history from the pen testers laptop with regular time/date stamps or maybe a video of the testers desktop. This means I can test a single issue to see if it’s fixed without needing a full blown follow up pen test.

I would happily trade time spent writing the flowery report for an actionable and usable list of issues and a test script for each one.

8 thoughts on “What I need from pen test reports.

  1. I think the problem starts earlier than the report Phil.

    If the objectives of the test aren’t fully understood by both parties, then the tester doesn’t really know what he’s writing a report about. So you get a muddled and inconsistent account of the “clever stuff he just did”, much like you describe in para 1.

    I’m working on a objectives based approach to scoping pen-tests at the moment, which hopefully nails some of these fundamentals and just *maybe* will give you what you need from each assignment.


    1. Gaz,

      I absolutely agree, it’s my job as the customer of the test to specify my requirements properly and the pen testers job to develop a test plan against that scope within budget. This is an iterative process between the customer and the pen tester (Or should be!).

      Scoping seems to have fallen into a usual practice of identifying platforms and IP addresses and then working out numbers of days required rather than identifying where the most value will be added by applying the budget available.

      I tend towards a hybrid approach where I allow the normal ‘how wide is it?’ scope to be written but I call out specific functional security requirements that I need tested based on my understanding of the system.

      What is interesting is when I do this I tend to have to specify that the report contains the results no matter of what was found. The fact that a test was successful can be as important for me to know as that a test has failed.


      1. Hi Phil,

        What irks me is that other testing disciplines like unit, integration of functional testing generally already do all this stuff very well. They (generally) have automated test scripts available for every single test, you can run those tests at any time and see which passed and which failed. This is how it should be.
        You get test reports for free, you get regression testing for free and you get unlimited retesting for free.

        This would not be too difficult to do for web application testing using the browser drivers, but would add some extra time and expense to a standard web app test. Implementing it for inf testing would be even more time consuming and costly – but not impossible.

        A compromise could be to use a Compliance table, e.g. take something like the OWASP ASVS and create a list of expected controls and then a Pass/Fail/Partial for each item. E.g. The application correctly validated input data: Pass. The application implemented anti-brute force measures in the authentication mechanism: Fail. etc.

        This would go a long way to extracting exactly which tests the pentesters performed, rather than only seeing which tests failed.


        1. Hi Stephen,

          I absolutely agree! I spend a fair amount of time in the other side of testing during system delivery and the difference with security is like night and day. Security testing generally does not come out well from the comparison.

          I would argue that *most* of the security tests performed by less than rockstar security testers are entirely able to be automated. It places more emphasis on getting the scoping and test plan correct as is the case in non-security testing. The security testing for the low hanging fruit (what I would call non-functional security requirements) is eminently able to be automated.

          The testing for custom logic flaws (what I would call functional security requirements) is more esoteric and we are not at a place where we can easily automate the ‘craft’. Although Dinis Cruz O2 framework hints at a future where this might become more commonplace.


          1. Automation doesn’t have to be the holy grail that everyone searches for. In reality, what we, and our clients, want is repeatable quality. This can be achieved by ensuring that consultants (no matter how experienced) follow a well defined methodology and rely on a common knowledge base (i.e. test cases), which the testers/consultants should continually contribute to.

            So by recording the knowledge and experience of our employees, team members, etc. we can achieve repeatable quality, help train juniors, integrate/indoctrinate new hires into our companies test culture (no two consultancies are the same), and achieve world domination? (ambitions may vary). It seems like a small, but significant, investment to me in order to achieve relatively handsome rewards.

            The point regarding pass/fail in QA and /fail in Security QA is incredibly important. This is something I’ve pushed for a while and I believe it can be a valuable addition to the end deliverable. You’re both absolutely right in that the SQA process can be improved immensely by learning from the QA process, but again I would argue that whilst tools and automation are important, they don’t have to be a bottleneck.

          2. Hi Dave,

            I agree automation is less important than repeatability.

            However, the contentious point amongst my pen tester brethren is that I want and need reliable repeatability across different pen test companies. We pay through the project for our supplier to write custom code and produce a series of product deliverables including the test plans and test scripts. We retain copies of those and the rights to reuse them so that we can continue to operate even if we change suppliers.

            With pen testers there is a desire by the pen test suppliers to retain the detail of the tests performed as trade secrets, this has the effect of locking us into a single pen testing supplier if we want true repeatability. A good methodology helps but if it’s contents are secret and they are different between pen test providers it can be hard to argue in high assurance situations that the regression test by a different test company truly tested the same flaw.

            There’s a balance between the competitive advantage a good in-house research team brings a pen test company and what maintaining secret test scripts means for client repeatability which isn’t being resolved in favour of the client at the moment.

  2. Phil,

    They way I’m looking at twisting things won’t necessarily help with defining the scope of the test. Not in the sense of, “we test this”, “we won’t test that”. So that iterative process would still exist to define scope. However were I want to change things is the test objectives. The main crux of the problem for me, is that testers test systems. And then note any risks in that system. This is all wrong (I think).

    I propose that testers test controls not systems. Every test should be mapped to a control objective (be it patch management, config. management, SDLC components etc). So your report tells you how effective your controls are, or if any are completely missing.

    I’ll email you my notes when I’ve developed them a little bit more.


  3. I did consider recording my desktop during a pen-test and making it available but then I thought about all the typos and areas where I picked the wrong tool or wrong command line options then the googling around trying to find what turns out to be an easy option and thought I’d rather a client didn’t see all that as I’m sure everyone does it but to actually watch it it could make me look a bit bad.

Comments are closed.