Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page R1
Statistical Software Engineering STATISTICAL SOFTWARE ENGINEERING Panel on Statistical Methods in Software Engineering Committee on Applied and Theoretical Statistics Board on Mathematical Sciences Commission on Physical Sciences, Mathematics, and Applications National Research Council National Academy Press Washington, D.C. 1996
OCR for page R2
Statistical Software Engineering NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce Alberts is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievement of engineers. Dr. Harold Liebowitz is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce Alberts and Dr. Harold Liebowitz are chairman and vice-chairman, respectively, of the National Research Council. This project was supported by the Advanced Research Projects Agency, Army Research Office, National Science Foundation, and Department of the Navy's Office of the Chief of Naval Research. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors. Furthermore, the content of the report does not necessarily reflect the position or the policy of the U.S. government, and no official endorsement should be inferred. Copyright 1996 by the National Academy of Sciences. All rights reserved. Library of Congress Catalog Card Number 95-71101 International Standard Book Number 0-309-05344-7 Additional copies of this report are available from: National Academy Press, Box 285 2101 Constitution Avenue, N.W. Washington, D.C. 20055 800-624-6242 202-334-3313 (in the Washington metropolitan area) B-676 Printed in the United States of America
OCR for page R3
Statistical Software Engineering PANEL ON STATISTICAL METHODS IN SOFTWARE ENGINEERING DARYL PREGIBON, AT&T Bell Laboratories, Chair HERMAN CHERNOFF, Harvard University BILL CURTIS, Carnegie Mellon University SIDDHARTHA R. DALAL, Bellcore GLORIA J. DAVIS, NASA-Ames Research Center RICHARD A. DEMILLO, Bellcore STEPHEN G. EICK, AT&T Bell Laboratories BEV LITTLEWOOD, City University, London, England CHITOOR V. RAMAMOORTHY, University of California, Berkeley Staff JOHN R. TUCKER, Director
OCR for page R4
Statistical Software Engineering COMMITTEE ON APPLIED AND THEORETICAL STATISTICS JON R. KETTENRING, Bellcore, Chair RICHARD A. BERK, University of California, Los Angeles LAWRENCE D. BROWN, University of Pennsylvania NICHOLAS P. JEWELL, University of California, Berkeley JAMES D. KUELBS, University of Wisconsin JOHN LEHOCZKY, Carnegie Mellon University DARYL PREGIBON, AT&T Bell Laboratories FRITZ SCHEUREN, George Washington University J. LAURIE SNELL, Dartmouth College ELIZABETH THOMPSON, University of Washington Staff JACK ALEXANDER, Program Officer
OCR for page R5
Statistical Software Engineering BOARD ON MATHEMATICAL SCIENCES AVNER FRIEDMAN, University of Minnesota, Chair LOUIS AUSLANDER, City University of New York HYMAN BASS, Columbia University MARY ELLEN BOCK, Purdue University PETER E. CASTRO, Eastman Kodak Company FAN R.K. CHUNG, University of Pennsylvania R. DUNCAN LUCE, University of California, Irvine SUSAN MONTGOMERY, University of Southern California GEORGE NEMHAUSER, Georgia Institute of Technology ANIL NERODE, Cornell University IMGRAM OLKIN, Stanford University RONALD F. PEIERLS, Brookhaven National Laboratory DONALD ST. P. RICHARDS, University of Virginia MARY F. WHEELER, Rice University WILLIAM P. ZIEMER, Indiana University Ex Officio Member JON R. KETTENRING, Bellcore Chair, Committee on Applied and Theoretical Statistics Staff JOHN R. TUCKER, Director JACK ALEXANDER, Program Officer RUTH E. O'BRIEN, Staff Associate BARBARA W. WRIGHT, Administrative Assistant
OCR for page R6
Statistical Software Engineering COMMISSION ON PHYSICAL SCIENCES, MATHEMATICS, AND APPLICATIONS ROBERT J. HERMANN, United Technologies Corporation, Chair STEPHEN L. ADLER, Institute for Advanced Study PETER M. BANKS, Environmental Research Institute of Michigan SYLVIA T. CEYER, Massachusetts Institute of Technology L. LOUIS HEGEDUS, W.R. Grace and Company JOHN E. HOPCROFT, Cornell University RHONDA J. HUGHES, Bryn Mawr College SHIRLEY A. JACKSON, U.S. Nuclear Regulatory Commission KENNETH I. KELLERMANN, National Radio Astronomy Observatory KEN KENNEDY, Rice University THOMAS A. PRINCE, California Institute of Technology JEROME SACKS, National Institute of Statistical Sciences L.E. SCRIVEN, University of Minnesota LEON T. SILVER, California Institute of Technology CHARLES P. SLICHTER, University of Illinois at Urbana-Champaign ALVIN W. TRIVELPIECE, Oak Ridge National Laboratory SHMUEL WINOGRAD, IBM T.J. Watson Research Center CHARLES A. ZRAKET, Mitre Corporation (retired) NORMAN METZGER, Executive Director
OCR for page R7
Statistical Software Engineering Preface The development and the production of high-quality, reliable, complex computer software have become critical issues in the enormous worldwide computer technology market. The capability to efficiently engineer computer software development and production processes is central to the future economic strength, competitiveness, and national security of the United States. However, problems related to software quality, reliability, and safety persist, a prominent example being the failure on several occasions of major local and national telecommunications networks. It is now acknowledged that the costs of producing and maintaining software greatly exceed the costs of developing, producing, and maintaining hardware. Thus the development and application of cost-saving tools, along with techniques for ensuring quality and reliability in software engineering, are primary goals in today's software industry. The enormity of this software production and maintenance activity is such that any tools contributing to serious cost savings will yield a tremendous payoff in absolute terms. At a meeting of the Committee on Applied and Theoretical Statistics (CATS) of the National Research Council (NRC), participants identified software engineering as an area presenting numerous opportunities for fruitful contributions from statistics and offering excellent potential for beneficial interactions between statisticians and software engineers that might promote improved software engineering practice and cost savings. To delineate these opportunities and focus attention on contexts promising useful interactions, CATS convened a study panel to gather information and produce a report that would (1) exhibit improved methods for assessing software productivity, quality, reliability, associated risk, and safety and for managing software development processes, (2) outline a program of research in the statistical sciences and their applications to software engineering with the aim of motivating and attracting new researchers from the mathematical sciences, statistics, and software engineering fields t o tackle these important and pressing problem areas, and (3) emphasize the relevance of using rigorous statistical and probabilistic techniques in software engineering contexts and suggest opportunities for further research in this direction. To help identify important issues and obtain a broad range of perspectives on them, the panel organized an information-gathering forum on October 11-12, 1993, at which 12 invited speakers addressed how statistical methods impinge on the software development process, software metrics, software dependability and testing, and software visualization. The forum also included consideration of nonstandard methods and select case studies (see the forum program in the appendix). The panel hopes that its report, which is based on the panel's expertise as well as information presented at the forum, will contribute to positive advances in software engineering and, as a subsidiary benefit, be a stimulus for other closely related disciplines, e.g., applied mathematics, operations research, computer science, and systems and industrial engineering. The panel is, in fact, very enthusiastic about the opportunities facing the statistical community and hopes to convey this enthusiasm in this report. The panel gratefully acknowledges the assistance and information provided by a number of individuals, including the 12 forum speakers—T.W. Keller, D. Card, V.R. Basili, J.C. Munson, J.C. Knight, R. Lipton, T. Yamaura, S. Zweben, M.S. Phadke, E.E. Sumne r, Jr., W. Hill, and J. Stasko—four anonymous reviewers, the NRC staff of the Board on Mathematical Sciences who supported the various facets of this project, and Susan Maurizi for her work in editing the manuscript.
OCR for page R8
Statistical Software Engineering This page in the original is blank.
OCR for page R9
Statistical Software Engineering Contents EXECUTIVE SUMMARY 1 1 INTRODUCTION 5 2 CASE STUDY: NASA SPACE SHUTTLE FLIGHT CONTROL SOFTWARE 9 Overview of Requirements 9 The Operational Life Cycle 10 A Statistical Approach to Managing the Software Production Process 10 Fault Detection 11 Safety Certification 12 3 A SOFTWARE PRODUCTION MODEL 13 Problem Formulation and Specification of Requirements 14 Design 14 Implementation 16 Testing 18 4 CRITIQUE OF SOME CURRENT APPLICATIONS OF STATISTICS IN SOFTWARE ENGINEERING 27 Cost Estimation 27 Statistical Inadequacies in Estimating 29 Process Volatility 30 Maturity and Data Granularity 30 Reliability of Model Inputs 31 Managing to Estimates 32 Assessment and Reliability 32 Reliability Growth Modeling 32 Influence of the Development Process on Software Dependability 36 Influence of the Operational Environment on Software Dependability 37 Safety-Critical Software and the Problem of Assuring Ultrahigh Dependability 38 Design Diversity, Fault Tolerance, and General Issues of Dependence 38 Judgment and Decision-making Framework 39 Structural Modeling Issues 40 Experimentation, Data Collection, and General Statistical Techniques 40 Software Measurement and Metrics 41 5 STATISTICAL CHALLENGES 43 Software Engineering Experimental Issues 43 Combining Information 46 Visualization in Software Engineering 48
OCR for page R10
Statistical Software Engineering Configuration Management Data 49 Function Call Graphs 50 Test Code Coverage 50 Code Metrics 50 Challenges for Visualization 52 Opportunities for Visualization 52 Orthogonal Defect Classification 59 6 SUMMARY AND CONCLUSIONS 61 Institutional Model for Research 62 Model for Data Collection and Analysis 62 Issues in Education 64 REFERENCES 67 APPENDIX: FORUM PROGRAM 72