How can software and the systems that rely on it be made dependable in a cost-effective manner, and how can one obtain assurance that dependability has been achieved? Rather than focusing narrowly on the question of software or system certification per se, this report adopts a broader perspective.
A system is dependable when it can be depended on to produce the consequences for which it was designed, and no adverse effects, in its intended environment. This means, first and foremost, that the term dependability has no useful meaning for a given system until these consequences and the intended environment are made explicit by a clear prioritization of the requirements of the system and an articulation of environmental assumptions. The effects of software are felt in the physical, human, and organizational environment in which it operates, so dependability should be understood in that context and cannot be reduced easily to local properties, such as resilience to crashing or conformance to a protocol. Humans who interact with the software should be viewed not as external and beyond the boundary of the software engineer’s concerns but as an integral part of the system. Failures involving human operators should not automatically be assumed to be the result of errors of usage; rather, the role of design flaws should be considered as well as the role of the human operator. As a consequence, a systems engineering approach— which views the software as one engineered artifact in a larger system of many components, some engineered and some given, and the pursuit of
dependability as a balancing of costs and benefits and a prioritization of risks—is vital.
Unfortunately, it is difficult to assess the dependability of software. The field of software engineering suffers from a pervasive lack of evidence about the incidence and severity of software failures; about the dependability of existing software systems; about the efficacy of existing and proposed development methods; about the benefits of certification schemes; and so on. There are many anecdotal reports, which—although often useful for indicating areas of concern or highlighting promising avenues of research—do little to establish a sound and complete basis for making policy decisions regarding dependability. Moreover, there is sometimes an implicit assumption that adhering to particular process strictures guarantees certain levels of dependability. The committee regards claims of extraordinary dependability that are sometimes made on this basis for the most critical of systems as unsubstantiated, and perhaps irresponsible. This difficulty regarding the lack of evidence for system dependability leads to two conclusions, reflected in the committee’s findings and recommendations below: (1) that better evidence is needed, so that approaches aimed at improving the dependability of software can be objectively assessed, and (2) that, for now, the pursuit of dependability in software systems should focus on the construction and evaluation of evidence.
The committee thus subscribes to the view that software is “guilty until proven innocent,” and that the burden of proof falls on the developer to convince the certifier or regulator that the software is dependable. This approach is not novel and is becoming standard in the world of systems safety, in which an explicit safety case (and not merely adherence to good practice) is usually required. Similarly, a software system should be regarded as dependable only if it has a credible dependability case, the elements of which are described below.
Meeting the burden of proof for dependability will be challenging. The demand for credible evidence will, in practice, make it infeasible to develop highly dependable systems in a cost-effective way without some radical changes in priorities. If very high dependability is to be achieved at reasonable cost, the needs of the dependability case will influence many aspects of the development, including the choice of programming language and the software architecture, and simplicity will be key. For high levels of dependability, the evidence provided by testing alone will rarely suffice and will have to be augmented by analysis. The ability to make independence arguments that allow global properties to be inferred from an analysis of a relatively small part of the system will be essential. Rigorous processes will be needed to ensure that the chain of evidence for dependability claims is preserved.
The committee also recognized the importance of adopting the practices that are already known and used by the best developers; this summary gives a sample of such practices in more detail below. Some of these (such as systematic configuration management and automated regression testing) are relatively easy to adopt; others (such as constructing hazard analyses and threat models, exploiting formal notations when appropriate, and applying static analysis to code) will require new training for many developers. However valuable, though, these practices are in themselves no silver bullet, and new techniques and methods will be required in order to build future software systems to the level of dependability that will be required.
Society is increasingly dependent on software. Software failures can cause or contribute to serious accidents that result in death, injury, significant environmental damage, or major financial loss. Such accidents have already occurred, and, without intervention, the increasingly pervasive use of software—especially in arenas such as transportation, health care, and the broader infrastructure—may make them more frequent and more serious. In the future, more pervasive deployment of software in the civic infrastructure could lead to more catastrophic failures unless improvements are made.
Software, according to a popular view, fails because of bugs: errors in the code that cause a program to fail to meet its specification. In fact, only a tiny proportion of failures can be attributed to bugs. As is well known to software engineers, by far the largest class of problems arises from errors made in the eliciting, recording, and analysis of requirements. A second major class of problems arises from poor human factors design. The two classes are related; bad user interfaces usually reflect an inadequate understanding of the user’s domain and the absence of a coherent and well-articulated conceptual model. Security vulnerabilities are to some extent an exception to this observation: The overwhelming majority of security vulnerabilities reported in software products—and exploited to attack the users of such products—are at the implementation level. The prevalence of code-related problems, however, is a direct consequence of higher-level decisions to use programming languages, design methods, and libraries that admit these problems.
In systems where software failure could have significant human or financial costs, it is crucial that software be dependable—that it can be depended upon to function as expected and to not cause or contribute to adverse events in the environment in which it operates. Improvements in dependability would allow such systems to be used more widely and
with greater confidence for the benefit of society. Moreover, software itself has great potential to bring improvements in safety in many areas.
Complete and reliable data about software-related system failures or the efficacy of particular software development approaches are hard to come by, making objective scientific evaluation difficult. Moreover, the lack of systematic reporting of software-related system failures is a serious problem that makes it more difficult to evaluate the risks and costs of such failures and to measure the effectiveness of proposed policies or interventions.
This lack of evidence has two direct consequences for this report. First, it has informed the key recommendations in this report regarding the need for evidence to be at the core of dependable software system development; for data collection efforts to be established; and for transparency and openness to be encouraged. Second, it has tempered the committee’s desire to provide prescriptive guidance: The approach recommended is therefore largely free of endorsements or criticisms of particular development approaches, tools, or techniques. Moreover, the report leaves to the developers and procurers of individual systems the question of what level of dependability is appropriate, and what costs are worth incurring to achieve it.
Nonetheless, the evidence available to the committee did support several qualitative conclusions. First, developing software to meet even existing dependability criteria is difficult and costly. Large software projects fail at a high rate, and the cost of projects that do succeed in delivering highly dependable software is often exorbitant. Second, the quality of software produced by the industry is extremely variable, and there is inadequate oversight in some critical areas. Today’s certification regimes and consensus standards have a mixed record. Some are largely ineffective, and some are counterproductive. They share a heavy reliance on testing, which cannot provide sufficient evidence for the high levels of dependability required in many critical applications.
A final observation is that the culture of an organization in which software is produced can have a dramatic effect on its quality and dependability. It seems likely that the excellent record of avionics software is due in large part to a safety culture in that industry that encourages meticulous attention to detail, high aversion to risk, and realistic assessment of software, staff, and process. Indeed, much of the benefit of standards such as DO-178B, Software Considerations in Airborne Systems and Equipment Certification, may be due to the safety culture that their strictures induce.
TOWARD CERTIFIABLY DEPENDABLE SOFTWARE
The focus of this report is a set of fundamental principles that underlie software system dependability and that suggest a different approach to the development and assessment of dependable software. Due to a lack of sufficient data to support or contradict any particular approach, a software system may not be declared “dependable” based on the method by which it was constructed. Rather, it should be regarded as dependable—certifiably dependable—only when adequate evidence has been marshaled in support of an argument for dependability that can be independently assessed. The goal of certifiably dependable software cannot therefore be achieved by mandating particular processes and approaches, regardless of their effectiveness in certain situations. Instead, software developers should marshal evidence to justify an explicit dependability claim that makes clear which properties in the real world the system is intended to establish. Such evidence forms a dependability case, and creating a dependability case is the cornerstone of the committee’s approach to developing certifiably dependable software systems.
Explicit Claims, Evidence, and Expertise
The committee’s proposed approach can be summarized in “the three Es”—explicit claims, evidence, and expertise:
Explicit claims. No system can be “dependable” in all respects and under all conditions. So to be useful, a claim of dependability must be explicit. It must articulate precisely the properties the system is expected to exhibit and the assumptions about the system’s environment upon which the claim is contingent. The claim should also indicate explicitly the level of dependability claimed, preferably in quantitative terms. Different properties may be assured to different levels of dependability.
Evidence. For a system to be regarded as dependable, concrete evidence must be present that substantiates the dependability claim. This evidence will take the form of a dependability case arguing that the required properties follow from the combination of the properties of the system itself (that is, the implementation) and the environmental assumptions. Because testing alone is usually insufficient to establish properties, the case will typically combine evidence from testing with evidence from analysis. In addition, the case will inevitably involve appeals to the process by which the software was developed—for example, to argue that the software deployed in the field is the same software that was subjected to analysis or testing.
Expertise. Expertise—in software development, in the domain under consideration, and in the broader systems context, among other things—is necessary to achieve dependable systems. Flexibility is an important advantage of the proposed approach; in particular the developer is not required to follow any particular process or use any particular method or technology. This flexibility allows experts freedom to employ new techniques and to tailor the approach to the system’s application and domain. But the requirement to produce evidence is highly demanding and likely to stretch today’s best practices to their limit. It will therefore be essential that developers are familiar with best practices and deviate from them only for good reason.
These prescriptions shape any particular development approach only in outline and give considerable freedom to developers in their choice of methods, languages, tools, and processes.
This approach is not, of course, a silver bullet. There are no easy solutions to the problem of developing dependable software, and there will always be systems that cannot be built to the required level of dependability even using the latest methods. But, the approach recommended is aimed at producing certifiably dependable systems today, and the committee believes it holds promise for developing the systems that will be needed in the future.
In the overall context of engineering, the basic tenets of the proposed approach are not controversial, so it may be a surprise to some that the approach is not already commonplace. Nor are the elements of the approach novel; they have been applied successfully for more than a decade. Nevertheless, this approach would require radical changes for most software development organizations and is likely to demand expertise that is currently in short supply.
Systems Engineering Approach
Complementing “the three Es” are several systems engineering ideas that provide an essential foundation for the building of dependable software systems:
Systems thinking. Engineering fields with long experience in building complex systems (for example, aerospace, chemical, and nuclear engineering) have developed approaches based on “systems thinking.” These approaches focus on properties of the system as a whole and on the interactions among its components, especially those interactions (often neglected) between a component being constructed and the components of its environment. As software has come to be deployed in—indeed has
enabled—increasingly complex systems, the system aspect has come to dominate in questions of software dependability.
Software as a system component. Dependability is not an intrinsic property of software. The committee strongly endorses the perspective of systems engineering, which views the software as one engineered artifact in a larger system of many components, some engineered and some given, and views the pursuit of dependability as a balancing of costs and benefits and a prioritization of risks. A software component that may be dependable in the context of one system might not be dependable in the context of another.
Humans as components. People—the operators and users (and even the developers and maintainers) of a system—may also be viewed as system components. If a system meets its dependability criteria only if people act in certain ways, then those people should be regarded as part of the system, and an estimate of the probability that they will behave as required should be part of the evidence for dependability.
Real-world properties. The properties of interest to the user of a system are typically located in the physical world: that a radiotherapy machine deliver a certain dose, that a telephone transmit a sound wave faithfully, that a printer make appropriate ink marks on paper, and so on. The software, on the other hand, is typically specified in terms of properties at its interfaces, which usually involve phenomena that are not of direct interest to the user: that the radiotherapy machine, telephone, or printer send or receive certain signals at certain ports, with the inputs related to the outputs according to some rules. It is important, therefore, to distinguish the requirements of a software system, which represent these properties in the physical world, from the specification of a software system, which characterizes the behavior of the software system at its interface with the environment. When the software system is itself only one component of a larger system, the other components in the system (including perhaps, as explained above, the people who work with the system) will be viewed as part of the environment. The dependability properties of a software system, therefore, should be expressed as requirements, and the dependability case should demonstrate how these properties follow from the combination of the specification and the environmental assumptions.
Coping with Complexity
The need for evidence of dependability and the difficulty of producing such evidence for complex systems have a straightforward but profound implication. Any component for which compelling evidence of dependability has been amassed at reasonable cost will likely be small by
the standards of most modern software systems. Every critical specification property, therefore, will have to be assured by one, or at most a few, small components. Sometimes it will not be possible to separate concerns so cleanly, and in that case, the dependability case may be less credible or more expensive to produce.
As a result, one key to achieving dependability at reasonable cost is a serious and sustained commitment to simplicity, including simplicity of critical functions and simplicity in system interactions. This commitment is often the mark of true expertise. An awareness of the need for simplicity usually comes only with bitter experience and the humility gained from years of practice. There is no alternative to simplicity. Advances in technology or development methods will not make simplicity redundant; on the contrary, they will give it greater leverage. To achieve high levels of dependability in the foreseeable future, striving for simplicity is likely to be by far the most cost-effective of all interventions. Simplicity is not easy or cheap, but its rewards far outweigh its costs.
The most important form of simplicity is that produced by independence, in which particular system-level properties are guaranteed by individual components much smaller than the system as a whole, which can preserve these properties despite failures in the rest of the system. Independence can be established in the overall design of the system, with the support of architectural mechanisms. Its effect is to dramatically reduce the cost of constructing a dependability case for a property, since only a relatively small part of the system needs to be considered.
Appropriate simplicity and independence cannot be accomplished without addressing the challenges of “interactive complexity” and “tight coupling.” Both interactive complexity, where components may interact in unanticipated ways, and tight coupling, wherein a single fault cannot be isolated but brings about other faults that cascade through the system, are correlated with the likelihood of system failure. Software-intensive systems tend to have both attributes. Careful attention should therefore be paid to the risks of interactive complexity and tight coupling and the advantages of modularity, isolation, and redundancy. The interdependences among components of critical software systems should be analyzed to ensure that there is no fault propagation path from less critical components to more critical components, that modes of failure are well understood, and that failures are localized to the greatest extent possible. The reduction of interactive complexity and tight coupling can contribute not only to the improvement of system dependability but also to the development of evidence and analysis in the service of a dependability case.
Rigorous Process and Preserving the Chain of Evidence
Generating a dependability case after the fact, when a development is largely complete, might be possible in theory. But in practice, at least with today’s technology, the costs of doing so would be high, and it will be practical to develop a dependability case only if the system is built with its construction in mind. Each step in developing the software needs to preserve the chain of evidence on which will be based the argument that the resulting system is dependable.
At the start, the domain and environmental assumptions and the required properties of the system should be made explicit; they should be expressed unambiguously and in a form that permits systematic analysis to ensure that there are no unresolvable conflicts between the required properties. Each subsequent stage of development should preserve the evidence chain—that these properties have been carried forward without being corrupted—so each form in which the requirements, design, or implementation is expressed should support analysis to permit checking that the required properties have been preserved. What is sufficient will vary with the required dependability, but preserving the evidence chain necessitates that the checks are carried out in a disciplined way, following a documented procedure, and leaving auditable records.
The Roles of Testing, Analysis, and Formal Methods
Testing is indispensable, and no software system can be regarded as dependable if it has not been extensively tested, even if its correctness has been proven mathematically. Testing may find flaws that elude analysis because it exercises the system in its entirety, whereas analysis must typically make assumptions about the execution platform, the external environment, and operator responses, any of which may turn out to be unwarranted. At the same time, it is important to realize that testing alone is rarely sufficient to establish high levels of dependability. It is erroneous to believe that a rigorous development process, in which testing and code review are the only verification techniques used, justifies claims of extraordinarily high levels of dependability. Some certification schemes, for example, associate higher safety integrity levels with more burdensome process prescriptions and imply that following the processes recommended for the highest integrity levels will ensure that the failure rate is minuscule. In the absence of a carefully constructed dependability case, such confidence is misplaced.
Because testing alone will not be sufficient for the foreseeable future, the dependability claim will also require evidence produced by analysis.
Moreover, because analysis links the software artifacts directly to the claimed properties, the analysis component of the dependability case will usually contribute confidence at a lower cost than testing for the highest levels of dependability. A dependability case will generally require many forms of analysis, including (1) the validation of environmental assumptions, use models, and fault models; (2) the analysis of fault tolerance measures against fault models; (3) schedulability analysis for temporal behaviors; (4) security analysis against attack models; (5) verification of code against module specifications; and (6) checking that modules in aggregate achieve appropriate system-level effects. These analyses will sometimes involve informal argument that is carefully reviewed; sometimes mechanical inference (as performed, for example, by “type checkers” that confirm that memory is used in a consistent way and that boundaries between modules are respected); and, sometimes, formal proof. Indeed, the dependability case for even a relatively simple system will usually require all of these kinds of analysis, and they will need to be fitted together into a coherent whole.
Traditional software development methods rely on human inspection and testing for validation and verification. Formal methods also use testing, but they employ notations and languages that are amenable to rigorous analysis, and they exploit mechanical tools for reasoning about the properties of requirements, specifications, designs, and code. Practitioners have been skeptical about the practicality of formal methods. Increasingly, however, there is evidence that formal methods can yield systems of very high dependability in a cost-effective manner, at least for small to medium-sized critical systems. Although formal methods are typically more expensive to apply when only low levels of dependability are required, the cost of traditional methods rises rapidly with the level of dependability and often becomes prohibitive. When a highly dependable system is required, therefore, a formal approach may be the most cost effective.
CERTIFICATION, TRANSPARENCY, AND ACCOUNTABILITY
A variety of certification regimes exist for software in particular application domains. For example, the Federal Aviation Authority (FAA) itself certifies new aircraft (and air-traffic management) systems that include software, and this certification is then relied on by the customers who buy and use the aircraft; the National Information Assurance Partnership (NIAP) licenses third-party laboratories to assess security software products for conformance to the Common Criteria. Some large organizations have their own regimes for certifying that the software products they buy meet the organization’s quality criteria, and many software product
manufacturers have their own criteria that each version of their product must pass before release.
Few, if any, existing certification regimes encompass the combination of characteristics recommended in this report—namely, explicit dependability claims, evidence for those claims, and a rigorous argument that demonstrates that the evidence is sufficient to establish the validity of the claims. To establish that a system is dependable will involve inspection and analysis of the dependability claim and the evidence offered in its support. Where the customer for the system is not able to carry out that work itself (for lack of time or lack of expertise) it may need to involve a third party whose judgment it can rely on to be independent of commercial pressures from the vendor. Certification can take many forms, from self-certification by the supplier at one extreme, to independent third-party certification by a licensed certification authority at the other. No single certification regime is suitable for all circumstances, so a suitable scheme should be chosen for each circumstance. Industry groups and professional societies should consider developing model certification schemes appropriate to their domains, taking account of the detailed recommendations in this report.
When choosing suppliers and products, customers and users can make informed judgments only if the claims are credible. Such claims are unlikely to be credible if the evidence underlying them is not transparent. Economists have established that if consumers cannot reliably observe quality before they buy, sellers may get little economic benefit from providing higher quality than their competitors, and overall quality can decline. Sellers are concerned about future sales, and “reputation effects” compel them to strive to maintain a minimum level of quality. If consumers rely heavily on branding, though, it becomes more difficult for new firms to enter the market, and quality innovations spread more slowly.
Those claiming dependability for their software should therefore make available the details of their claims, criteria, and evidence. To assess the credibility of such details effectively, an evaluator should be able to calibrate not only the technical claims and evidence but also the organization that produced them, because the integrity of the evidence chain is vital and cannot easily be assessed without supporting data. This suggests that in some cases data of a more general nature should be made available, including the qualifications of the personnel involved in the development; the track record of the organization in providing dependable software; and the process by which the software was developed. The willingness of a supplier to provide such data, and the clarity and integrity of the data that the supplier provides, will be a strong indication of its attitude to dependability.
Where there is a need to deploy software that satisfies a particular dependability claim, it should always be explicit who is accountable for any failure to achieve it. Such accountability can be made explicit in the purchase contract, or as part of certification of the software, or as part of a professional licensing scheme, or in other ways. Since no single solution will suit all the circumstances in which certifiably dependable software systems are deployed, accountability regimes should be tailored to particular circumstances. At present, it is common for software developers to disclaim, so far as possible, all liability for defects in their products, to a greater extent than customers and society expect from manufacturers in other industries. Clearly, no software should be considered dependable if it is supplied with a disclaimer that withholds the manufacturer’s commitment to provide a warranty or other remedies for software that fails to meet its dependability claims. Determining the appropriate scale of remedies, however, was beyond the scope of this study and would require a careful analysis of benefits and costs, taking into account not only the legal issues but also the state of software engineering, the various submarkets for software, the economic impact, and the effect on innovation.
KEY FINDINGS AND RECOMMENDATIONS
Presented below are the committee’s findings and recommendations, each of which is discussed in more detail in Chapter 4.
Improvements in software development are needed to keep pace with societal demands for software. Avoidable software failures have already been responsible for loss of life and for major economic losses. The quality of software produced by the industry is extremely variable, and there is inadequate oversight in several critical areas. More pervasive deployment of software in the civic infrastructure may lead to catastrophic failures unless improvements are made. Software has the potential to bring dramatic benefits to society, but it will not be possible to realize these benefits—especially in critical applications—unless software becomes more dependable.
More data are needed about software failures and the efficacy of development approaches. Assessment of the state of the software industry, the risks posed by software, and progress made is currently hampered by the lack of a coherent source of information about software failures.
Recommendations to Builders and Users of Software
Make the most of effective software development technologies and formal methods. A variety of modern technologies—in particular, safe programming languages, static analysis (analysis of software and source code done without actually executing the program), and formal methods—are likely to reduce the cost and difficulty of producing dependable software.
Follow proven principles for software development. The committee’s proposed approach also includes adherence to the following principles:
Take a systems perspective. Here the dependability of software is viewed not in terms of intrinsic properties (such as the incidence of bugs in the code) but rather in terms of the system as a whole, including interactions among people, process, and technology.
Exploit simplicity. If dependability is to be achieved at reasonable cost, simplicity should become a key goal, and developers and customers must be willing to accept the compromises it entails.
Make a dependability case for a given system and context: evidence, explicitness, and expertise. A software system should be regarded as dependable only if sufficient evidence of its explicitly articulated properties is presented to substantiate the dependability claim. This approach gives considerable leeway to developers to use whatever practices are best suited to the problem at hand. In practice the challenges of developing dependable software are sufficiently great that developers will need considerable expertise, and they will have to justify any deviations from best practices.
Demand more transparency, so that customers and users can make more informed judgments about dependability. Customers and users can make informed judgments when choosing suppliers and products only if the claims, criteria, and evidence for dependability are transparent.
Make use of but do not rely solely on process and testing. Testing will be an essential component of a dependability case, but will not in general suffice, because even the largest test suites typically used will not exercise enough paths to provide evidence that the software is correct nor will it have sufficient statistical significance for the levels of confidence usually desired. Rigorous process is essential for preserving the chain of dependability evidence but is not per se evidence of dependability.
Base certification on inspection and analysis of the dependability claim and the evidence offered in its support. Because testing and process alone are insufficient, the dependability claim will require, in addi-
tion, evidence produced by other modes of analysis. Security certification in particular should go beyond functional testing of the security components of a system and assess the effectiveness of measures the developer took to prevent the introduction of security vulnerabilities.
Include security considerations in the dependability case. Security vulnerabilities can undermine the case made for dependability properties by violating assumptions about how components behave, about their interactions, or about the expected behavior of users. The dependability case must therefore account explicitly for security risks that might compromise its other aspects. It is also important to ensure that security certifications give meaningful assurance of resistance to attack. New security certification regimes are needed that can provide confidence that most attacks against certified products or systems will fail. Such regimes can be built by applying the other findings and recommendations of this report, with an emphasis on the role of the environment—in particular, the assumptions made about the potential actions of a hostile attacker and the likelihood that new classes of vulnerabilities will be discovered and new attacks developed to exploit them.
Demand accountability and make it explicit. Where there is a need to deploy certifiably dependable software, it should always be made explicit who or what is accountable, professionally and legally, for any failure to achieve the declared dependability.
Recommendations to Agencies and Organizations That Support Software Education and Research
The committee was not constituted or charged to recommend budget levels or to assess trade-offs between software dependability and other priorities. However, it believes that the increasing importance of software to society and the extraordinary challenge currently faced in producing software of adequate dependability provide a strong rationale for investment in education and research initiatives.
Place greater emphasis on dependability—and its fundamental underpinnings—in the high school, undergraduate, and graduate education of software developers. Many practitioners do not have an adequate appreciation of the software dependability issues discussed in this report, are not aware of the most effective development practices available today, or are not capable of applying them appropriately. Wider implementation of the committee’s recommended approach, which goes beyond today’s state of the practice, implies a need for further education and training activities.
Federal agencies that support information technology research and development should give priority to basic research to further software-enabled system dependability, emphasizing a systems perspective and evidence. In keeping with this report’s approach, such research should emphasize a systems perspective and “the three Es” (explicit claims, evidence, and expertise) and should be informed by a systems view that attaches more importance to those advances that are likely to have an impact in a world of large systems interacting with other systems and operators in a complex physical environment and organizational context.