with one another. There did appear to be some consensus among panelists that certification processes should focus on the desired properties rather than on the manner in which the artifact is constructed. It was observed, however, that some properties that are not directly observable may require secondary evidence. There was substantial interest in certifying components for composability. There was also interest in certification processes that can be applied to a software component as it evolves.

There was some discussion of appropriate techniques for certification. Several participants contended that formal methods have a role to play in certification and that although there are limits to what they can achieve, “we should continue the long march of formal methods.” It was suggested that scale and adoptability should be the primary aim of continued research in this area, with a focus on “small theorems about large programs.” There was also controversy over the maturity of certification techniques and of software development in general. Some felt that generally reliable software can be achieved using existing software development techniques but that certification could be used to “raise the bar.” Others emphasized the need for more research on the use of tools and techniques to increase software quality.

The question of process-based versus product-based certification was discussed. There was some consensus that a combination of the two is necessary, and that “process evidence is only secondary to product evidence.” The new European standards ESARR 62 and SW013 were offered as examples of the combination approach. It was suggested that formal methods could offer support for “baking the evidence of process into the artifact itself,” using such techniques as proof-carrying code.

Panelists warned of several obstacles to effective certification. For example, standards that mandate adherence to untestable criteria are unlikely to be effective, as are certification processes that lack accountability. Standards that provide insufficient connection to the end goal can increase the cost of creating software without increasing its actual dependability. Standards may be too technology dependent, mandating methods of constructing (or not constructing) software. They may mandate the production of enormous documents that have little if any impact on the dependability of the software, or the measurement of “trivia” that is easily measurable but offers only a false sense of security.

Finally, participants briefly touched on some additional considerations, and the session concluded with suggestions for the committee in the preparation of its final report. There was some support for the notion that software development companies should be required to publish defect rates. It was noted that software products are often licensed without the “fitness-for-purpose” requirements that apply to other goods. The education of both professionals and consumers was a recurring topic: “We should make tools that give people best practices demos, so people can tell how to write good software, and can tell the difference. Consumer pressure [to produce dependable systems] is not as powerful as people had hoped.”

The committee was encouraged “not to be hidebound by current practices” and to take a longer-range view about what people involved in the development of complex software systems need to learn and know. What are the criteria for a good study in the area of certifiably dependable systems? What should the committee do to ensure that such a study provides value to the community and capitalizes on the wisdom and experience of the community at large?

It was suggested that the committee’s final report should, at least in part, be positive. Participants observed that the community tends to be negative but that much has been achieved. It

2  

The Eurocontrol Safety Regulatory Requirement (ESARR 6) provides a set of guidelines to ensure that risks associated with operating any software in safety-related ground-based air traffic management systems are reduced to a tolerable level. This guideline was developed by the European Organisation for the Safety of Air Navigation.

3  

SW01 is part of a larger set of civil aviation guidelines and standards called CAP 670 established by the Civil Aviation Authority in the United Kingdom. SW01 deals specifically with the design and assessment of ground-based services, including air traffic control systems.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement