Measuring Security

For nearly a decade I have been regularly coming back to one of the hardest problems in security, measuring it. There are lots of opinions and no shortage of books full of candidate metrics and there are swathes of consultants prepared to give you a spreadsheet of metrics to go measure and develop a red/amber/green dashboard to understand them. It does seem to require practitioners to dig a bit deeper often to find a good approach to developing metrics and measurements that are actually of value to a particular organisation.

This post captures some of the thinking I’ve distilled from some of the big thinkers in the field. Talking of big thinkers…

“When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of science.” —Lord Kelvin, 1824-1907

“It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.” —Sir Arthur Conan Doyle, 1887

“Security is now so essential a concern that we can no longer use adjectives and adverbs but must instead use numbers.” — Dan Geer, 2008

Why measure security?

Measurement is fundamentally the reduction of uncertainty (with a hat tip to Doug Hubbard). The organisations we work for need to make decisions about the allocation of limited resources both to and within security. Getting these allocation decisions right means successfully delivering to the security drivers.

Getting these decisions wrong can be disastrous. The value of reducing uncertainty in security decisions is as big in IT budget terms of both avoiding unnecessary spend and avoiding breach costs.

Having a clear understanding of the security drivers for an organisation can be a good foundation for understanding what is worth measuring. Security drivers tend to be either Business Strategy or what the organisation wants to achieve and how security helps deliver that, Risk Management ensuring the organisation is taking an acceptable level of security risk or Compliance ensuring the organisation is meeting it’s regulatory and contractual requirements.

Key measures of security

There are a variety of different types of measurement available to us but the key measures tend to look at risks, controls or performance:

• A Key Risk Indicator (KRI) measures the outcomes we are concerned about and the factors that may enhance them.
• A Key Control Indicator (KCI) measures the effectiveness of the controls we deploy to mitigate and manage risk.
• A Key Performance Indicator (KPI) measures the output of activities we deliver in order to identify our relative performance in their delivery

Key security risks

There are many security risks every organisation needs to concerned with but experience tends to show us that the management of risk follows a pareto rule (ignoring black swan events) whereas actively managing 20% of the highest priority risks likely mitigates or manages 80% of the overall risk profile due to shared platforms and processes. Key security risks need to be identified but importantly need to be characterised in a reliable and useful manner for both practitioners and specialists. Security risks have characteristics of interest that include; likelihood, vulnerability and impact as well as factors and mitigations)

Having a fixed format for writing risks helps identify the common characteristics across risks and across different teams in an organisation:

• There is a risk that <impact> will occur as a result of <event type> caused by <threat source> exploiting <vulnerability> in order to achieve <goal>.
• This risk is amplified by <factors> and reduced by <mitigations>

Identifying the measurable characteristics of key security risks

Developing security measurements from key risks

An effective tool for developing organisationally relevant and useful measurements is the Goal, Question, Metric (or Concept, Object, Method) approach. The GQM approach focuses on identifying organisational goals, developing sets of natural language questions that would allow the intended audience to judge performance based on the answers and subsequently to identify a set of metrics that can drive the answers to the questions.

The GQM approach avoids the unfortunately common technology metrics approach of measuring everything that’s easy to measure assuming some of it might be useful to someone and replaces it with only measuring those things that help answer the audiences concerns.

Example

Assuming there has been a key risk identified for an organisation such that:

“There is a risk that reputational damage and regulatory fines will occur as a result of a large scale data breach caused by organised criminals exploiting unpatched systems in order to achieve theft of personally identifiable information.”

This would likely lead an organisation to consider a Security Goal related to Risk Management along the lines of reducing the financial and reputational damage from data breaches.

It would be reasonable for a senior executive to ask questions related to that goal along the lines of:

• Are we having more or less data breaches?
• Is our risk from a data breach increasing or decreasing?
• Are we doing all we can to prevent and limit data breaches?

Having a set of questions to answer we could look at possible KRIs such as:

• Number of data records held
• Number of data records breached
• Volume of outbound  comms traffic from the network
• Number of data records within 2 network hops of the Internet
• Staff churn in roles with privileged data record access
• Value of a single data record to the business
• Market value of a single data record to criminal groups
• Level of regulatory fine per data record

Tracking changes in these KRIs would potentially indicate a shift in our risk exposure up or down.

Similarly we could look for relevant KCIs perhaps in Data Leakage Prevention or Data Record Encryption controls or KPIs in data breach detection processes or privileged user access management processes. These could potentially indicate a reduction in our exposure (assuming the numbers are heading in the right direction.

It is unlikely that a Senior Executive will ever ask for KCIs on the effective implementation of encryption directly but by measuring that we can answer the question is our risk decreasing or increasing and importantly can provide a transparent and auditable metric for why we believe that is the answer.

Unfortunately there is no one size fits all for security measurement and that is because the Key Security Risks and the Security Goals of each organisation are different. An effective approach for developing management relevant measurements happily is something that can be copied and pasted… go ahead.