National Academies Press: OpenBook

Summary of a Workshop on Software Certification and Dependability (2004)

Chapter: Appendix A: Workshop Agenda

« Previous: 3 Summary of Closing Session
Suggested Citation:"Appendix A: Workshop Agenda." National Research Council. 2004. Summary of a Workshop on Software Certification and Dependability. Washington, DC: The National Academies Press. doi: 10.17226/11133.
×

Appendixes

Suggested Citation:"Appendix A: Workshop Agenda." National Research Council. 2004. Summary of a Workshop on Software Certification and Dependability. Washington, DC: The National Academies Press. doi: 10.17226/11133.
×

This page intentionally left blank.

Suggested Citation:"Appendix A: Workshop Agenda." National Research Council. 2004. Summary of a Workshop on Software Certification and Dependability. Washington, DC: The National Academies Press. doi: 10.17226/11133.
×

A
Workshop Agenda

MONDAY, APRIL 19, 2004

Welcome

Charles Brownstein, Director, Computer Science and Telecommunications Board

Daniel Jackson, Chair, Committee on Building Certifiably Dependable Systems

Panel A: The Strengths and Limitations of Process

Isaac Levendel, Independent Consultant

Gary McGraw, Cigital

Peter Neumann, SRI International

Moderator: Martyn Thomas

The focus of this panel is the contribution of particular processes and process characteristics to the successful development and effective certification of dependable systems.

  • What are the important characteristics of the processes that you believe should be followed when developing certifiably dependable systems? What evidence exists to support your opinion? How would it be possible to gain stronger evidence?

  • How important is evidence of the development process (or the absence of such evidence) to certification that a system meets its dependability objectives? Does your answer depend on the nature of the system under consideration? If so, in what way?

  • What specific processes, if carried out effectively, could provide sufficient evidence that a system meets its functional requirements? Does your answer change if the system is (a) preparing customer bills for a major utility company; (b) controlling a radiotherapy system; (c) providing flight-control for a fly-by-wire civil airliner; (d) protecting military secrets in a system accessible to staff with lower-level security clearances?

  • How would your answers change for the same question, applied to nonfunctional requirements, such as performance and usability?

  • How do you measure or demonstrate the correlation between process metrics and product metrics for attributes such as reliability and security?

Suggested Citation:"Appendix A: Workshop Agenda." National Research Council. 2004. Summary of a Workshop on Software Certification and Dependability. Washington, DC: The National Academies Press. doi: 10.17226/11133.
×
  • Can it ever be reasonable to argue that a system is more dependable than the evidence available can demonstrate scientifically? What should be the role of engineering judgment in certifying systems?

  • What do you consider to be the strengths and limitations of process metrics in assessing the dependability of a computer-based system?

Panel B: Looking Forward: New Challenges, New Opportunities

Robert Harper, Carnegie Mellon University

Shriram Krishnamurthi, Brown University

James Larus, Microsoft Research

André van Tilborg, Office of the Secretary of Defense

Moderators: John Rushby, Lui Sha

The focus of this panel is what has changed in the last 30 years with respect to certification.

  • How have the development of new technology and the spread of computing changed both the problems we face in certifying software, and the potential solutions to the certification problem?

  • How does the increasingly pervasive use of software in infrastructural systems affect the need for certification?

  • Does the greater sophistication of today’s users affect the problem?

  • What challenges and opportunities are presented by the widespread use of COTS software and outsourcing? How can we build and certify systems in which critical and noncritical components work together?

  • Should we move certification from a process-centric process to a product-centric process over time? If so, how?

  • What technologies are promising for aiding certification efforts? What role will there be for static methods such as static analysis, proof systems, and model checking? And for dynamic approaches involving, for example, runtime assertions and fault detection, masking, and recovery?

  • Is incremental certification in traditional safety-critical systems such as flight control an important goal to work toward? What is the technology barrier to success?

Panel C: Certification and Regulation: Experience to Date

Brent Goldfarb, University of Maryland

Mats Heimdahl, University of Minnesota

Charles Howell, MITRE Corporation

Robert Noel, MITRE Corporation

Moderators: Michael DeWalt, Scott Wallsten

The focus of this panel is to understand how certification and regulation affect software development.

  • How do regulation and certification affect current mission-critical software development?

  • What are the differences and similarities between industry-standard, self-imposed regulations, and government- or policy-imposed regulations and standards?

Suggested Citation:"Appendix A: Workshop Agenda." National Research Council. 2004. Summary of a Workshop on Software Certification and Dependability. Washington, DC: The National Academies Press. doi: 10.17226/11133.
×
  • How do developers consider trade-offs between improved safety/reliability and lower costs associated with less dependable (but perhaps more available) systems?

  • How do regulations and certification affect innovation?

  • Within your field of expertise, what are the top three issues in regulation or government oversight that hamper system dependability? What are the top three regulatory approaches that have provided significant improvements in system dependability?

  • How are regulations and guidance within your organization promulgated and approved?

  • What are the differences and similarities between developing regulations and guidance material for hardware dependability and developing them for software dependability?

  • What are possible future challenges (economic, technological, or otherwise) with respect to current regulatory and certification approaches?

  • In your answers to these questions, what supporting data are available and what supporting data are needed to buttress analyses?

Panel D: Organizational Context, Incentives, Safety Culture, and Management

Richard Cook, University of Chicago

Gene Rochlin, University of California, Berkeley

William Scherlis, Carnegie Mellon University

Moderators: Charles Perrow, David Woods

The focus of this panel is to explore the implications of certification within the organizational context.

  • How are software development organizations responsible for failures? How can software development organizations learn from failure? How can software development as a model of operations be integrated with operations?

  • How can software development better anticipate the reverberations of technology change?

  • Do we need to highlight particular problems with organizational performance in the certification area, distinct from dependability in general? Are the “mental models” of organizational routines more vulnerable in this area, thus requiring more demanding safeguards or personnel? How might this be achieved? By outsourcing, special training, incentives?

  • What role might insurance and liability play in achieving higher levels of certification? Might liability threats promote a better safety culture? Would the availability of insurance help (or make matters worse—the moral hazard problem)? Might insurers require evidence of reliability practices to make insurance available or reduce high premiums? Are there precedents for this in other areas of safety in low-probability/high-risk endeavors, and is there evidence the effort is successful?

  • To what extent should certification be left entirely to the producer? When should a firm hire specialists? When should a consumer require that an independent agency do the certification? Are there trade-secret issues with outside involvement?

  • Products are sold on the basis of performance and features. How can we make the promise of dependability attractive to consumers given its added cost?

Reactions to Panels

Suggested Citation:"Appendix A: Workshop Agenda." National Research Council. 2004. Summary of a Workshop on Software Certification and Dependability. Washington, DC: The National Academies Press. doi: 10.17226/11133.
×

TUESDAY, APRIL 20, 2004

Panel E: Cost-Effectiveness of Software Engineering Techniques

Kent Beck, Three Rivers Institute

Matthias Felleisen, Northeastern University

Anthony Hall, Praxis Critical Systems

Moderators: Peter Lee, Jon Pincus

The focus of this panel is to understand the cost-effectiveness of current software engineering techniques as they relate to dependability and certification.

  • What is the evidence for the cost-effectiveness of various software engineering techniques, either today or looking toward the future? Ideally, this would focus on the techniques’ roles in producing dependable software; however, strong evidence for cost-effectiveness in other domains is also interesting.

  • To the extent that evidence is currently limited, what kind of investigation could lead to strengthening it in the future?

  • Are there particularly promising directions that can lead to particular software engineering techniques becoming more cost-effective for creating dependable software?

Panel F: Case Study: Electronic Voting

David Dill, Stanford University

Douglas Jones, University of Iowa

Avi Rubin, Johns Hopkins University

Ted Selker, Massachusetts Institute of Technology

Moderators: Reed Gardner, Daniel Jackson

The focus of this panel is to explore a particular application domain within the context of certification, dependability, and regulation.

  • What role does software play in voting? How crucial is it? Does it make things worse or better?

  • What properties of the software might be certified? What current approaches might help?

  • What would the certification process, if any, be? Who would do it? What credibility would it have? Who has to be trusted? What ulterior motives are at play?

  • With respect to issues of dependability and certification, is this case study typical, or unique in some ways?

Group Brainstorm

Moderator: Daniel Jackson

What are the important questions that have come out of this workshop that the committee should address in the rest of its study?

Suggested Citation:"Appendix A: Workshop Agenda." National Research Council. 2004. Summary of a Workshop on Software Certification and Dependability. Washington, DC: The National Academies Press. doi: 10.17226/11133.
×
Page 25
Suggested Citation:"Appendix A: Workshop Agenda." National Research Council. 2004. Summary of a Workshop on Software Certification and Dependability. Washington, DC: The National Academies Press. doi: 10.17226/11133.
×
Page 26
Suggested Citation:"Appendix A: Workshop Agenda." National Research Council. 2004. Summary of a Workshop on Software Certification and Dependability. Washington, DC: The National Academies Press. doi: 10.17226/11133.
×
Page 27
Suggested Citation:"Appendix A: Workshop Agenda." National Research Council. 2004. Summary of a Workshop on Software Certification and Dependability. Washington, DC: The National Academies Press. doi: 10.17226/11133.
×
Page 28
Suggested Citation:"Appendix A: Workshop Agenda." National Research Council. 2004. Summary of a Workshop on Software Certification and Dependability. Washington, DC: The National Academies Press. doi: 10.17226/11133.
×
Page 29
Suggested Citation:"Appendix A: Workshop Agenda." National Research Council. 2004. Summary of a Workshop on Software Certification and Dependability. Washington, DC: The National Academies Press. doi: 10.17226/11133.
×
Page 30
Next: Appendix B: Panelist Biographies »
Summary of a Workshop on Software Certification and Dependability Get This Book
×
Buy Paperback | $29.00 Buy Ebook | $23.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Certification of critical software systems (e.g., for safety and security) is important to help ensure their dependability. Today, certification relies as much on evaluation of the software development process as it does on the system’s properties. While the latter are preferable, the complexity of these systems usually makes them extremely difficult to evaluate. To explore these and related issues, the National Coordination Office for Information technology Research and Development asked the NRC to undertake a study to assess the current state of certification in dependable systems. The study is in two phases: the first to frame the problem and the second to assess it. This report presents a summary of a workshop held as part of the first phase. The report presents a summary of workshop participants’ presentations and subsequent discussion. It covers, among other things, the strengths and limitations of process; new challenges and opportunities; experience to date; organization context; and cost-effectiveness of software engineering techniques. A consensus report will be issued upon completion of the second phase.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!