3

Engineering, Operational, and Life-Cycle Challenges in Security Science

Improving cybersecurity requires that security considerations be integrated into the practice of hardware and software development, evolution, deployment, and evaluation—indeed, to reflect an understanding of the life cycle of a system. Research in many technical areas may embed assumptions (e.g., about agility and expected operations and maintenance) or may be focused explicitly on how to improve post-deployment activities related to systems. That is, research that focuses on how maintenance and system administration affect overall system performance would be part of a holistic approach to cybersecurity. Improved cybersecurity requires good technology control, resilience, and reliability, highlighting the importance of research in software quality assurance as well as more traditional notions of software security research.

LESSONS FROM DEVELOPMENT ORGANIZATIONS

In addition to what the traditional industrial research and academic communities contribute, software development organizations developing commodity systems have made substantial efforts to improve their own practices and the systems they build. These organizations have, over time, created a set of development practices aimed at reducing the prevalence of exploitable vulnerabilities in released software.¹ A critical

___________________

¹ For example, see, Microsoft, “What is the Security Development Lifecycle?,” 2016, https://www.microsoft.com/en-us/sdl/default.aspx, accessed September 2016. See also M. Howard and S. Lipner, The Security Development Lifecycle: A Process for Developing Demon-

component of these approaches is the use of feedback loops from discovered vulnerabilities or attacks that are traced to root causes. Applied systematically, these feedback loops can lead to new tools or techniques and are fundamental to improving cybersecurity. These efforts can be part of security science, integrating what is known about state-of-the-art software engineering practices; social, behavioral, and organizational theory; current understandings of the threat landscape; and models of attacks and defenses. Practical lessons from companies working at the cutting edge of secure system development can inform research approaches that incorporate scientific models. There are also opportunities to use well-established open-source projects as raw material for analysis (e.g., use of software life-cycle data). In academia, some of this sort of work has been done using open-source software as a source of information as well as working with large software companies.² The open-source stacks offer for study an abundance of components, libraries, frameworks, and tools. There is also a strong tradition of data analysis of open-source repository data and other data. And, most obviously, in most open-source projects, there is ready access to code, design documents, test cases, analysis models, process data, historical data, and so on.

Recently, the National Institute of Standards and Technology issued a report exploring how to reduce technical vulnerabilities in software.³ It offers an analysis of a number of technical approaches that can be applied in the software development process, their potential impacts, and suggestions for encouraging the development and use of a number of measurements and metrics to help assess software. Research on how security outcomes relate to development practices and on the use of particular tools and software languages is foundational. Cybersecurity outcomes ultimately rely on the properties of the systems developed and deployed in the real world. If it is known, for instance, that a certain class of programming languages and their compilers can prevent a wide class of

___________________

strably More Secure Software, Microsoft Press, Redmond, Wash., 2006. Microsoft has been especially open regarding its tooling and quality efforts—features include (1) diversity of models and tools, (2) clear pathways and incentives for development teams to adopt improved practices without coercion of a mandate, and (3) a creative approach to assessment of benefits and costs.

² For example, see Laurie Williams’ research group’s efforts to study software security using empirical analysis (A. Konovalov, “Project Zero,” blog, May 10, 2007, https://googleprojectzero.blogspot.com/ undertakes security analysis and publishes regularly on weaknesses in vendor projects).

³ P.E. Black, L. Badger, B. Guttman, and E. Fong, Dramatically Reducing Software Vulnerabilities: Report to the White House Office of Science and Technology Policy, National Institute of Standards and Technology, 2016, http://csrc.nist.gov/publications/drafts/nistir-8151/nistir8151_draft.pdf.

common errors⁴ that lead to security vulnerabilities, how can their wide adoption be incentivized? Managed languages and compilers have been explored and developed through research. Understanding which class of errors they prevent is related to the scientific foundations discussed in Chapter 1. Understanding how to incentivize their adoption would draw on social science knowledge and research. An additional challenge is that developers of systems frequently reuse software components developed by others, including components affecting security. Vulnerabilities in these components can result in common vulnerabilities across many systems.⁵ Even when the developers of the components fix vulnerabilities, the revisions may not result in updates to all the systems that depend on them. It would be worth understanding how often this occurs, especially in the Internet of Things (IoT) and other embedded devices. One might reverse-engineer the software in many different devices to estimate the distribution of shared components and how current they are.

Another operational challenge is virtualization. As the use of virtualization increases, it is important that security is addressed as an integral part of the technology and the architecture that is being virtualized. This spans the use of various hypervisors in infrastructures to the container approaches to application security. As security is increasingly integrated into virtualized environments, a recurring research challenge is how to assess and measure the effectiveness, completeness, and effective integration of the security measures.

LESSONS FROM REAL-WORLD DEPLOYMENTS

System administrators and other practitioners are often on the front lines of securing systems. It is important to develop mechanisms whereby

___________________

⁴ For instance, in the Defense Advanced Research Projects Agency’s clean-slate total-system architecture research and development effort on a new capability-based hardware, new operating systems and low-level compartmentalization kernels, and extensions of common programming languages that reflect the capability mechanism, it is the LLVM compiler extensions that address the hardware and inherently prevent buffer overflows and numerous other common security flaws. See D. Chisnall, C. Rothwell, B. Davis, R.N.M. Watson, J. Woodruff, S.W. Moore, P.G. Neumann, and M. Roe, “Beyond the PDP-11: Architectural Support for a Memory-Safe C Abstract Machine,” 20th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2015, Istanbul, Turkey, March 14-18, 2015; and R.N.M. Watson, J. Woodruff, P.G. Neumann, S.W. Moore, J. Anderson, D. Chisnall, N. Dave, B. Davis, B. Laurie, S.J. Murdoch, R. Norton, M. Roe, S. Son, and M. Vadera, “CHERI: A Hybrid Capability-System Architecture for Scalable Software Compartmentalization,” IEEE Symposium on Security and Privacy, San Jose, Calif., May 18-20, 2015.

⁵ One example is the continued use of Windows XP in some embedded applications. OpenSSL is another.

researchers can learn about the real problems being experienced in the field by practitioners. Example areas of overlap include work on resilient architectures, composition of components and properties, logging systems, variability, and configuration. Researchers in these areas benefit hugely from industry contacts and trust relationships with practitioner colleagues. In cybersecurity, however, knowledge transfer can be difficult, since so much is sensitive, including information about vulnerabilities, threats, events, data affected, organizational impacts, roles of human actors, and so on. In addition, understanding industry norms and expectations, and how they are evolving, is important. For instance, requirements for improved measurement and evaluation regimes will likely have an impact on practices and will need to be accounted for in research.

Related to the topic of observation, vulnerability research is an area that examines existing, deployed computer systems and networks with the aim of discovering new vulnerabilities. (See Box 3.1 for background on how this community developed.) The vulnerability “hacker” research ecosystem is often considered distinct from academic research, but it is clearly not. There are opportunities for academic researchers to learn from vulnerability research and to collaborate with vulnerability researchers. There are also open research questions—technical questions regarding the science of vulnerability finding (e.g., code-focused research based on binaries, directed fuzzing,⁶ and low-level analysis as well as such research based on source-level considerations, design of runtime application programming interfaces [APIs], and so on) as well as social science questions regarding motivating and rewarding vulnerability researchers to maximize the likelihood that newly found vulnerabilities or classes of vulnerabilities will be mitigated or eliminated from systems rather than being exploited. For instance, how effective are so-called bug bounties in terms of improving the security of systems? How do results from bug bounty efforts compare to other efforts? In terms of communications and reporting, vulnerability research results could be strengthened by including an estimate of the practical effect of what has been discovered. More generally, how can threat information play into this part of the process? There is a set of questions related to understanding attackers and deriving attack trees.⁷ Deriving attack trees can be helpful in considering whether an artifact satisfies its specification and how complete the specification itself is. For instance, are there implicit assumptions that lead to

___________________

⁶ Fuzzing or fuzz testing is a way of testing software by providing large amounts of random or unexpected data to a system to try to make it crash. Directed fuzzing is a variation by which key parts of well-formed input are determined, and those are modified (fuzzed)—for instance, to ensure that fuzzed input can get past basic checksums.

⁷ An attack tree is a diagram used to describe how a target might be attacked. It can be used to show the many different paths that could lead to a successful attack.

vulnerabilities? This sort of analysis connects vulnerability and life-cycle analyses with formal verification approaches to yield a more complete understanding of the security of a system.

CHALLENGES OF CYBERSECURITY METRICS

One cross-cutting technical challenge involves metrics.⁸ Developing a metric of the “overall cybersecurity” of a system is very difficult to do, not least because what is meant by “security” can vary significantly.⁹ At a minimum, such a global metric would require a comprehensive model of the system in question. But adversaries can exploit vulnerabilities in deployed systems, whose detailed characteristics may not all be captured by a model, meaning that a global cybersecurity metric defined on a model may not capture all possible attacks. Moreover, as Herley observes, the fact that there is no single observation that would allow an arbitrary system to be declared secure shows that claims of necessary conditions for security are unfalsifiable.¹⁰ Instead, one can exclude various vulnerabilities.

One can think in terms of what sorts of failures are allowed. For example, is the goal to minimize maximum regret, minimize average regret, maximize time between successful attacks of any sort, or something else? Regardless of the attack vector, is the goal to have a system where the attack surface is relatively large but where damage of a successful attack is limited, or a system where the attack surface is relatively small but where damage is catastrophic if an attack succeeds? Put another way, given fixed resources, should one set up a highly regimented, centrally controlled homogeneous infrastructure that is well administered or a federated heterogeneous infrastructure that may not be as well administered? The latter obviously has a larger attack surface, assuming appropriate independence, but the former is more susceptible to complete compromise once penetrated. The larger observation is that security is not only non-binary—it does not lend itself to a linear ordering. This makes the concept of security metrics even more difficult.

___________________

⁸ See S.M. Bellovin, On the brittleness of software and the infeasibility of security metrics, IEEE Security and Privacy, 2006.

⁹ Pfleeger and Cunningham have written at length about why security is difficult to measure. See S.L. Pfleeger and R. Cunningham, Why measuring security is hard, IEEE Security and Privacy 8(4):46-54, 2010, http://doi.ieeecomputersociety.org/10.1109/MSP.2010.60.

¹⁰ C. Herley, Unfalsifiability of security claims, Proceedings of the National Academies of Sciences 113(23):6415-6420, 2016. The ideas in this paper were followed up and expanded on in C. Herley and P. van Oorschot, “SoK: Science, Security, and the Elusive Science of Security,” Proceedings of the 2017 IEEE Symposium on Security and Privacy, forthcoming, http://people.scs.carleton.ca/~paulv/papers/oakland2017science.pdf.

There are properties of systems and of organizations that can be measured and can serve as a useful proxy against known classes of attacks. Proxy measures can be valuable as a way to distinguish between the cybersecurity equivalent of biomarkers (e.g., elements of blood chemistry) and health states (e.g., vision problems related to diabetes). Good biomarkers can be valuable, even essential, for monitoring latent conditions and monitoring conditions that are difficult to assess directly (whereas bad ones can be distractions or misleading in dangerous ways).

Metrics to measure capabilities of organizations could be developed. Examples of such metrics include the following: Does the organization use publicly available indicators of compromise? For what fraction of its systems and network? Does the organization collect data flows on its internal network and external connections? Another aspect that might be subject to measurement is commitment to continuous improvement. Does the organization learn from its own (and others’) mistakes and adapt? Over what kind of time frame?

Research is needed on metrics that make clear what could be done in terms of organizational practices. In many cases, which practices enhance security in a given context is not known. Most organizations have limited resources to devote to security. Researchers have an opportunity to investigate the degree to which each security practice is successful in a given context. One goal would be to provide practitioners with an understanding of what improvements and returns on investment an organization could expect for given security practices (and for how long and under what conditions). This kind of research needs the involvement of social scientists—and especially organizational psychologists, who can characterize the organizations, the people in them, and their practices and offer ways to assess the impacts of new practices. Another benefit would be to publicly describe the measures practiced by the most successful organizations in a way that makes them available for use as metrics by others. Further, this work would need to be kept up-to-date. Adversaries change tactics and approaches frequently, and the organizations who successfully defend themselves adapt continuously. Moving toward better ways to describe and measure security and security-related properties of a system and of organizations will involve understanding how science, models, attacks, and defenses interact; how systems are engineered, deployed, and maintained; and how organizations decide to invest in, develop, and promulgate technologies, practices, and policies regarding security.

ADDITIONAL RESEARCH TOPICS

Achieving effective outcomes stems from a combination of understanding models of systems, communicating that understanding well

across the research community, applying research results, high-quality engineering, understanding deployed systems, and a host of social, organizational, and behavioral considerations. In addition to development practices and engineering that prioritize security in the features and functionality of a system, designing systems for a complete life cycle that includes secure procurement, use, maintenance, operations, system administration, and aftermarket activities (such as notifications, updates, and repairs) is an important consideration. Legacy systems exist and will continue to be used—there is a research opportunity regarding how better to secure them and technical challenges that range from system-level reverse engineering to sandboxing to service-oriented architectures.

In addition to the ideas discussed above, potential topics of research and exploration include the following:

• Improving the inspectability of trusted subsystems—for example, by incorporating security checks (e.g., checksums of common software components across an enterprise) into regular operations and administration to watch for unexpected changes.
• Establishing and maintaining a root of trust and configuration integrity.
• Exploring architectural considerations such as coupling and minimization of the trusted computing bases, API and framework design issues, resiliency, pervasive monitoring and logging, and so on.
• Using a “DevOps” approach where developers and operators are collaborating to improve the security of cloud-based services. (The increased use of cloud-based software services has some security advantages—for instance, it is much easier to update the software as needed. On the other hand, the capability to roll out updates and changes rapidly can lead to mistakes that happen at scale, and the security and privacy risks at scale may weigh against cost savings.)
• Continuing to improve the ecosystem by which (sanitized) data sets from industry can be shared with cybersecurity experts in academia such that long-term empirical studies can be conducted that will allow a wide range of cybersecurity metrics to be tracked and for formal experiments that test particular research hypotheses to be conducted.
• Increasing the resilience of up-front engineering for embedded systems, SCADA systems, and the IoT. In these cases, updates are infrequent, if they are ever done. What incentives are available to make such engineering happen? Or, if these systems are made more easily “update-able,” how can their use as an attack channel

• be prevented? And how can users be provided with a high level of confidence that updating will not negatively impact their systems?
• Developing scientific models that incorporate assumptions about anticipated longevity and usage expectations.
• Designing and evaluating the usability of systems (readable user messages, careful selection of the options presented, careful design and selection of the metaphors through which policies and status are communicated to humans, and so on).
• Developing effective ways for cybersecurity experts, system administrators, and others who operate the systems to share insights and learn from each other regarding, for instance, designing and testing security protocols, data logging and analysis, and so on.
• Working with the operations and engineering communities to begin to understand “ground truth” about what is actually deployed and operating in the real world. For instance, to what extent do embedded systems or the IoT share security-relevant components and code? This would require systematic surveys, analysis, and reporting, and possibly include the need to reverse-engineer code.
• Developing better linkages between research and standards and evaluation criteria for cybersecurity.
• Considering observation itself as a fundamental scientific approach and developing an ongoing, realistic understanding of what the universe of deployed systems looks and behaves like through the use of monitoring, logging, and data analysis (real-time, lagged, and forensic) to inform ongoing cybersecurity research efforts and priorities.