Summary of a Workshop on Software-Intensive Systems and Uncertainty at Scale (2007)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Joan D. Winston and Lynette I. Millett, Editors Committee on Advancing Software-Intensive Systems Producibility Computer Science and Telecommunications Board Division on Engineering and Physical Sciences 

THE NATIONAL ACADEMIES PRESS  500 Fifth Street, N.W.  Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Gov- erning Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engi- neering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. Support for this project was provided by the Office of the Secretary of Defense, Department of Defense, with assistance from the National Science Foundation under sponsor award number CNS-0541636 and by the Office of Naval Research under sponsor award number N00014-04-1-0736. Any opinions, findings, conclu- sions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the agencies and organizations that provided support for the project. International Standard Book Number-13:  978-0-309-10844-7 International Standard Book Number-10:  0-309-10844-6 Additional copies of this report are available from The National Academies Press 500 Fifth Street, N.W., Lockbox 285 Washington, DC 20055 800/624-6242 202/334-3313 (in the Washington metropolitan area) http://www.nap.edu Copyright 2007 by the National Academy of Sciences. All rights reserved. Printed in the United States of America 

The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal govern- ment on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the char- ter of the National Academy of Sciences, as a parallel organization of outstand- ing engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in pro- viding services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council. www.national-academies.org 



Committee on Advancing Software-Intensive Systems Producibility WILLIAM L. SCHERLIS, Carnegie Mellon University, Chair ROBERT F. BEHLER, The MITRE Corporation BARRY W. BOEHM, University of Southern California LORI A. CLARKE, University of Massachusetts, Amherst MICHAEL A. CUSUMANO, Massachusetts Institute of Technology MARY ANN DAVIDSON, Oracle Corporation LARRY DRUFFEL, Independent Consultant RUSSELL FREW, Lockheed Martin JAMES LARUS, Microsoft Corporation GREG MORRISETT, Harvard University WALKER ROYCE, IBM DOUGLAS C. SCHMIDT, Vanderbilt University JOHN P. STENBIT, Independent Consultant KEVIN J. SULLIVAN, University of Virginia Staff LYNETTE I. MILLETT, Study Director and Senior Program Officer JOAN D. WINSTON, Program Officer MARGARET MARSH HUYNH, Senior Program Assistant  

COMPUTER SCIENCE AND TELECOMMUNICATIONS BOARD JOSEPH F. TRAUB, Columbia University, Chair Eric Benhamou, Benhamou Global Ventures, LLC FREDERICK R. CHANG, University of Texas, Austin WILLIAM DALLY, Stanford University MARK E. DEAN, IBM Almaden Research Center DEBORAH ESTRIN, University of California, Los Angeles JOAN FEIGENBAUM, Yale University KEVIN KAHN, Intel Corporation JAMES KAJIYA, Microsoft Corporation MICHAEL KATZ, University of California, Berkeley RANDY H. KATZ, University of California, Berkeley SARA KIESLER, Carnegie Mellon University TERESA H. MENG, Stanford University PRABHAKAR RAGHAVAN, Yahoo! Research FRED B. SCHNEIDER, Cornell University ALFRED Z. SPECTOR, Independent Consultant, Pelham, New York WILLIAM STEAD, Vanderbilt University ANDREW J. VITERBI, Viterbi Group, LLC PETER WEINBERGER, Google, Inc. Staff JON EISENBERG, Director KRISTEN BATCH, Associate Program Officer RADHIKA CHARI, Administrative Coordinator RENEE HAWKINS, Financial Associate MARGARET MARSH HUYNH, Senior Program Assistant HERBERT S. LIN, Senior Scientist LYNETTE I. MILLETT, Senior Program Officer DAVID PADGHAM, Associate Program Officer JANICE M. SABUDA, Senior Program Assistant TED SCHMITT, Consultant BRANDYE WILLIAMS, Program Assistant JOAN D. WINSTON, Program Officer For more information on CSTB, see its Web site at <http://www.cstb.org>, write to CSTB, National Research Council, 500 Fifth Street, N.W., Wash- ington, DC 20001, call (202) 334-2605, or e-mail CSTB at cstb@nas.edu. vi 

Preface P ursuant to a request by the Department of Defense, the National Research Council (NRC) convened a study committee under the auspices of the Computer Science and Telecommunications Board (CSTB) to assess the nature of the U.S. national investment in software research and, in particular, to consider ways to enhance the knowledge and human resource base needed to design, produce, and employ software- intensive systems for tomorrow’s weapons and operations systems. Many organizations are facing the combination of increasing system scale and increasing complexity of software-intensive systems. However, the com- pelling need to interconnect them to realize DoD’s vision of “net-­centric warfare” exacerbates the challenges of uncertainty at scale for DoD. Several recent reports that highlight these challenges suggest that the scale and complexity of software-intensive systems introduce funda- mental new challenges and require augmentation of existing approaches by software practices and technologies that more explicitly address these challenges. Challenges of uncertainty and scale are faced in large-scale enterprise systems of all kinds but are particularly demanding in defense systems owing to the relative lack of precedent in both requirements and engineering designs and also to the need for high levels of quality, secu-   See, for example, Software Engineering Institute (2006), Ultra-large Scale Systems: The Software Challenge of the Future, which noted that current abstractions fail for the levels of complexity that systems require today. Also see Defense Science Board (2000), Task Force on Defense Software Report, which noted, among other things, that strengthening the technology base to rapidly adapt to fluid circumstances is important and that the complexity of DoD software applications is increasing more than linearly. vii 

viii SOFTWARE-INTENSIVE SYSTEMS AND UNCERTAINTY AT SCALE rity, and safety in an environment with well-resourced adversaries. This suggests that defense, while sharing many particular kinds of require- ments with large-scale enterprises and infrastructures, is nonetheless a demand leader with respect to many of these requirements, outpacing most enterprise and commercial projects. As part of its study, this committee organized a public workshop on January 17, 2007, to examine uncertainty at scale in current and future software-intensive systems. Workshop sessions examined the challenges related to engineering uncertainty, system complexity, and scale from a range of perspectives. Session speakers were given roughly 25 minutes to provide their views on issues identified in the workshop agenda (see Appendix A). There was substantial discussion and interaction among the session speakers and moderators, the committee, and others present. The purpose of the workshop was to inform the committee as it con- ducts its study. This report summarizes the workshop discussions, includ- ing speaker presentations and discussions with committee members and others present. It is not a compilation of quotations from particular indi- viduals nor is it a complete synthesis of conclusions or results. Although the summary was prepared by the committee based on presentations and discussion at the workshop, the comments do not necessarily reflect the views of the committee nor do they represent findings and recommenda- tions of the NRC. Moreover, the summary points for the sessions are a digest of both presentations and discussion. They should not be taken as remarks made solely by the scheduled session speakers because the discussions included remarks by the others in attendance. The committee’s broader consideration of advancing software- i ­ ntensive system producibility will appear in its final report, to be issued near the end of the study. That report will provide recommendations to the responsible agency, the executive branch, and legislative officials— and to the broader software community—about how to improve software development and achieve future goals. The committee thanks all the workshop participants for their thought- ful presentations and discussion. It also thanks the Computer Science and Telecommunications Board staff, particularly study director Lynette Millett and program officer Joan Winston, who have ably managed this project, coordinated the team, and contributed greatly to the development of this report, and to Margaret Huynh, who has facilitated our meetings and other project activities. William Scherlis, Chair Committee on Advancing Software-Intensive Systems Producibility 

Acknowledgment of Reviewers T his report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Research Council’s (NRC’s) Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evi- dence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: David Notkin, University of Washington, Alfred Spector, Independent Consultant, John Vu, Boeing Corporation, Peter Weinberger, Google, Inc., and Jeannette Wing, Carnegie Mellon University. Although the reviewers listed above have provided many construc- tive comments and suggestions, they were not asked to endorse the con- clusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was coordinated by Susan Graham of the University of California, Berkeley. Appointed by the NRC, ix 

 SOFTWARE-INTENSIVE SYSTEMS AND UNCERTAINTY AT SCALE she was responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring com- mittee and the institution. 

Contents 1 INTRODUCTION AND OVERVIEW 1 2 SUMMARY OF WORKSHOP DISCUSSIONS 4 Session 1: Process, Architecture, and the Grand Scale, 4 Session 2: DoD Software Challenges for Future Systems, 12 Session 3: Agility at Scale, 19 Session 4: Quality and Assurance with Scale and Uncertainty, 23 Session 5: Enterprise Scale and Beyond, 33 3 WRAP-UP DISCUSSION AND EMERGENT THEMES 40 Architectural Challenges in Large-Scale Systems, 40 The Need for Software Engineering Capability, 41 Open Questions and Research Opportunities, 42 APPENDIXES A Workshop Agenda 47 B Biosketches of Committee Members and Staff 50 C Biosketches of Workshop Speakers 60 xi