1
Overview of Workshop Discussions
This report summarizes a workshop on software certification and dependability held April 19-20, 2004, in Washington, D.C., under the auspices of the Committee on Certifiably Dependable Software Systems. Several items should be kept in mind when reading this report:
-
The workshop focused on the subset of areas that the committee believed would best help frame the program of work for the remaining study period. There are areas of direct relevance for the study that are missing from the workshop agenda, either because of time constraints or because the panelists chose to address different areas. The committee plans to gather input on those areas in subsequent activities; feedback and additional input from readers of this report are welcome.
-
During the workshop, committee members deliberately refrained from questioning views expressed at the workshop—they preferred to use the time to gather input from workshop participants in an impartial manner. In addition, the committee chose not to extend the discussions in this first-phase report, instead reserving that task for the final report. Consequently, this report does not provide a free-standing overview of the current state of software development, of certification, or of anything other than the views expressed at this particular workshop.
-
The panel summaries have not been edited to make the terminology used by each panel consistent across the entire report. Meanings should be clear from the context. Deciding on appropriate and consistent terminology is a task for the committee as it prepares its final report.
Listed below are the main themes arising from each panel session. These themes are not conclusions or findings of the committee; they incorporate ideas extracted from each panel that seem to represent the major thrusts of each discussion. Each panel session discussion is elaborated in Chapter 2.
PANEL A THE STRENGTHS AND LIMITATIONS OF PROCESS
-
While following particular processes cannot alone guarantee certifiably dependable software, comprehensive engineering processes are nevertheless important to achieving this goal.
-
In evaluating a system, appropriate metrics should be used; it may be necessary to measure secondary artifacts (e.g., process quality) as surrogates if what is actually of interest cannot be measured.
-
Developing ways to determine how best to allocate resources (e.g., understanding where errors are likely to cluster) can improve both dependability and cost-effectiveness.
PANEL B LOOKING FORWARD: NEW CHALLENGES, NEW OPPORTUNITIES
-
Over the past several decades society has become increasingly dependent on software. While desktop systems are not generally regarded as safety-critical, these days they are often task-critical for their users. This is true not only at the individual level but also at the organizational level.
-
One increasingly sophisticated set of tools that can help in the software development process with respect to dependability is the set of tools related to programming languages, such as type checkers, static analyzers, and model checkers.
-
Systems integration is a growing and challenging problem. Additional tools and strategies are needed to cope with large-scale systems integration issues.
PANEL C CERTIFICATION AND REGULATION: EXPERIENCE TO DATE
-
The process of certification may add value in a collateral fashion because attention must be paid to issues that might not receive it otherwise; given that software and its uses and contexts change over time, any value that certification has decays over time as well.
-
Market forces and the cost structure of the software industry may create incentives to release flawed software.
-
Validation—determining what the software should do—is often harder than verification, or determining whether the software does it correctly, and may be more important. Despite the difficulties of achieving validation systematically, however, many critical systems seem to function well.
PANEL D ORGANIZATIONAL CONTEXT, INCENTIVES, SAFETY CULTURE, AND MANAGEMENT
-
Systems are certified only within a particular context and up to specified system boundaries; certification of a system may not guarantee the dependability and usefulness of a system over its lifetime.
-
As a system’s reliability increases or is demonstrated over long periods of time, dependence on that system may increase to an extent not anticipated in the original design.
-
Accountability, reporting, and communication are difficult issues that must be planned and managed in detail across an organization.
PANEL E COST-EFFECTIVENESS OF SOFTWARE ENGINEERING TECHNIQUES
-
There are interesting substantive overlaps in approaches to software development that seem philosophically opposed on the surface. In particular, agile methods such as “Extreme Programming” seem to share important elements with methods that employ formal notations for early modeling and analysis.
-
Understanding what is meant by “dependability” is critical; it was observed that, given its ubiquitous use and deployment, software is treated as though generally dependable.
-
Achieving dependable, certifiable software will require emphasis on process, people, and tools.
PANEL F CASE STUDY: ELECTRONIC VOTING
-
Structural flaws in the voting system go beyond the absence of voter-verifiable paper trails.
-
The lack of detailed risk analysis, coupled with a lack of openness in the voting system certification process, poses serious challenges to achieving a dependable voting infrastructure.
-
The current certification process does not seem to have resulted in secure or dependable electronic voting systems.