devices to the system?), the reliability of personnel, and the nature of the threat against the system.
Accordingly, a discussion cast simply in terms of whether a system is or is not secure is almost certainly misleading. Assessing the security of a system must include qualifiers such as, Security against what kind of threat? Under what circumstances? For what purpose? With what configuration? Under what security policy?
What does the discussion above imply for the development of cybersecurity metrics—measurable quantities whose value provides information about a system or network’s resistance to a hostile cyber operation? Metrics are intended to help individuals and companies make rational quantitative decisions about whether or not they have “done enough” with respect to cybersecurity. These parties would be able to quantify cost-benefit tradeoffs in implementing security features, and they would be able to determine if System A is more secure than System B. Good cybersecurity metrics would also support a more robust insurance market in cybersecurity founded on sound actuarial principles and knowledge.
The holy grail for cybersecurity analysts is an overall cybersecurity metric that is applicable to all systems and in all operating environments. The discussion above, not to mention several decades’ worth of research and operational experience, suggests that this holy grail will not be achieved for the foreseeable future. But other metrics may still be useful under some circumstances.
It is important to distinguish between input metrics (metrics for what system users or designers do to the system), output metrics (metrics for what the system produces), and outcome metrics (metrics for what users or designers are trying to achieve—the “why” for the output metrics).21
• Input metrics reflect system characteristics, operation, or environment that are believed to be associated with desirable cybersecurity outcomes. An example of an input metric could be the annual cybersecurity budget of an organization. In practice, many input metrics for cybersecurity are not validated in practice, and/or are established intuitively.
• Output metrics reflect system performance with respect to parameters that are believed to be associated with desirable cybersecurity outcomes. An output metric in a cybersecurity context could be the number of cybersecurity incidents in a given year. Output metrics can often be assessed through the use of a red team. Sometimes known as “white-hat”
21 See Republic of South Africa, “Key Performance Information Concepts,” Chapter 3 in Framework for Managing Programme Performance Information, National Treasury, Pretoria, South Africa, May 2007, available at http://www.thepresidency.gov.za/learning/reference/framework/part3.pdf.