This is the third in a series of posts describing how to put together a security testing stategy and the associated test plans. Part one is here and part two is here.
This is what I want to see covered in security test plans. Whenever I ask the supplier to specify or carry out the security tests I ensure I get to review and approve the test plans and the test outputs as part of the formal project deliverable process. I also try to make sure that the inputs to the test plan are made available to the actual security testers completing the test so they get a better feel for what the context of their test results is.
Automated static code analysis test plan
I would expect to complete automated static code analysis at unit Test or component test stages of the test cycle. I would also expect this to repeat for each release of custom developed components or bits of glue code during the development phase.
Automated static code analysis is used to cover compliance with the Secure coding standards and to identify common security flaws in the language used for the development. The supplier should specify the test plan and test scripts for any automated static code analysis using:
- Secure coding standards,
- Technical threat model.
The supplier should carry out Automated static code analysis. As the customer the only outputs I expect to see are the documented security defects (even if they are subsequently fixed) and a set of security defect metrics. The sorts of metrics I would want reported out of each release include:
- security defects by component per thousand lines of code,
- security defects by component by time to mitigate,
- security defects by component by type of vulnerability.
There are likely to be large numbers of both false positives thrown up by the automated tools, where these have been identified as such by the supplier they should be removed from the defect numbers.
Manual source code analysis test plan
The manual source code analysis should be completed at the component test stage of the test cycle. I would only require this on an ad-hoc basis for high-risk custom developed components identified through the development phase of each release.
Manual source code analysis is used to more accurately cover compliance with the Secure coding standards and to identify common security flaws in the language used for the development and flaws in the application business logic that might lead to a security breach. Manual code analysis is expensive and time consuming so it should be used sparingly.
the Supplier should specify the test plan and test scripts for the manual source code analysis. The inputs to this test plan should include:
- Secure coding standards,
- Technical threat model,
- System risk model.
The supplier should carry out Manual source code analysis on an ad-hoc basis as identified by the review of the defect counts from the automated static code analysis and the system risk model. The supplier should carry out manual source code analysis, as the customer the only outputs I expect to see are the documented security defects (even if they are subsequently fixed) and a set of security defect metrics. The sorts of metrics I would want reported out of each release include:
- security defects by component per thousand lines of code,
- security defects by component by time to mitigate,
- security defects by component by type of vulnerability.
Risk-based penetration test plans
I would expect to specify risk based penetration tests during the integration test stage of the test cycle. I would only require this on an ad-hoc basis for high-risk subsystems identified during the build phase of each release.
Risk based penetration tests focus on functional security requirements as defined in the security requirements for the system and negative requirements which are implied security requirements derived from the possible negative outcomes identified in the system risk model.
The supplier should specify the risk based penetration tests plans but in close collaboration with an independent testing company and me as the client. The inputs to the test plan and test scripts for the risk based penetration tests should include:
- Technical threat model,
- Security defect metrics,
- System security requirements,
- System threat model,
- System risk model.
The test should be performed by the independent testing company and the output should be the detailed documented security defects.
Programme penetration test plans
The programme penetration tests occur at the system test or end to end test stage for each release. I recommend this is completed twice, first internally by the supplier and second by an independent testing company.
The programme penetration test is the wrap up, it covers the functional security requirements defined in the system security requirements, the negative requirements implied from the negative outcomes in the system risk model and the non-functional security requirements where the performance and stress testing of security-specific components is completed.
The supplier should specify this test plan but as the client I must assure that scope before it is agreed with the security testing teams. The content of the programme penetration test plan and scripts is derived from:
- Technical Threat Model
- Security Defect Metrics
- System Security Requirements
- System Threat Model
- Infrastructure secure configuration guides
This should be performed first by the supplier to sweep up any glaring configuration errorss and missing patches and subsequently by an independent security testing company and the output should be the detailed documented security defects.