| ||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page R1
Statistical Software Engineering
STATISTICAL SOFTWARE ENGINEERING
Panel on Statistical Methods in Software Engineering
Committee on Applied and Theoretical Statistics
Board on Mathematical Sciences
Commission on Physical Sciences, Mathematics, and Applications
National Research Council
National Academy Press
Washington, D.C. 1996
OCR for page R2
Statistical Software Engineering
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine.
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce Alberts is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievement of engineers. Dr. Harold Liebowitz is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce Alberts and Dr. Harold Liebowitz are chairman and vice-chairman, respectively, of the National Research Council.
This project was supported by the Advanced Research Projects Agency, Army Research Office, National Science Foundation, and Department of the Navy's Office of the Chief of Naval Research. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors. Furthermore, the content of the report does not necessarily reflect the position or the policy of the U.S. government, and no official endorsement should be inferred.
Copyright 1996 by the National Academy of Sciences. All rights reserved.
Library of Congress Catalog Card Number 95-71101
International Standard Book Number 0-309-05344-7
Additional copies of this report are available from:
National Academy Press,
Box 285 2101 Constitution Avenue, N.W. Washington, D.C. 20055 800-624-6242 202-334-3313 (in the Washington metropolitan area)
B-676
Printed in the United States of America
OCR for page R3
Statistical Software Engineering
PANEL ON STATISTICAL METHODS IN SOFTWARE ENGINEERING
DARYL PREGIBON,
AT&T Bell Laboratories,
Chair
HERMAN CHERNOFF,
Harvard University
BILL CURTIS,
Carnegie Mellon University
SIDDHARTHA R. DALAL,
Bellcore
GLORIA J. DAVIS,
NASA-Ames Research Center
RICHARD A. DEMILLO,
Bellcore
STEPHEN G. EICK,
AT&T Bell Laboratories
BEV LITTLEWOOD,
City University, London, England
CHITOOR V. RAMAMOORTHY,
University of California, Berkeley
Staff
JOHN R. TUCKER, Director
OCR for page R4
Statistical Software Engineering
COMMITTEE ON APPLIED AND THEORETICAL STATISTICS
JON R. KETTENRING,
Bellcore,
Chair
RICHARD A. BERK,
University of California, Los Angeles
LAWRENCE D. BROWN,
University of Pennsylvania
NICHOLAS P. JEWELL,
University of California, Berkeley
JAMES D. KUELBS,
University of Wisconsin
JOHN LEHOCZKY,
Carnegie Mellon University
DARYL PREGIBON,
AT&T Bell Laboratories
FRITZ SCHEUREN,
George Washington University
J. LAURIE SNELL,
Dartmouth College
ELIZABETH THOMPSON,
University of Washington
Staff
JACK ALEXANDER, Program Officer
OCR for page R5
Statistical Software Engineering
BOARD ON MATHEMATICAL SCIENCES
AVNER FRIEDMAN,
University of Minnesota,
Chair
LOUIS AUSLANDER,
City University of New York
HYMAN BASS,
Columbia University
MARY ELLEN BOCK,
Purdue University
PETER E. CASTRO,
Eastman Kodak Company
FAN R.K. CHUNG,
University of Pennsylvania
R. DUNCAN LUCE,
University of California, Irvine
SUSAN MONTGOMERY,
University of Southern California
GEORGE NEMHAUSER,
Georgia Institute of Technology
ANIL NERODE,
Cornell University
IMGRAM OLKIN,
Stanford University
RONALD F. PEIERLS,
Brookhaven National Laboratory
DONALD ST. P. RICHARDS,
University of Virginia
MARY F. WHEELER,
Rice University
WILLIAM P. ZIEMER,
Indiana University
Ex Officio Member
JON R. KETTENRING,
Bellcore
Chair,
Committee on Applied and Theoretical Statistics
Staff
JOHN R. TUCKER, Director
JACK ALEXANDER, Program Officer
RUTH E. O'BRIEN, Staff Associate
BARBARA W. WRIGHT, Administrative Assistant
OCR for page R6
Statistical Software Engineering
COMMISSION ON PHYSICAL SCIENCES, MATHEMATICS, AND APPLICATIONS
ROBERT J. HERMANN,
United Technologies Corporation,
Chair
STEPHEN L. ADLER,
Institute for Advanced Study
PETER M. BANKS,
Environmental Research Institute of Michigan
SYLVIA T. CEYER,
Massachusetts Institute of Technology
L. LOUIS HEGEDUS,
W.R. Grace and Company
JOHN E. HOPCROFT,
Cornell University
RHONDA J. HUGHES,
Bryn Mawr College
SHIRLEY A. JACKSON,
U.S. Nuclear Regulatory Commission
KENNETH I. KELLERMANN,
National Radio Astronomy Observatory
KEN KENNEDY,
Rice University
THOMAS A. PRINCE,
California Institute of Technology
JEROME SACKS,
National Institute of Statistical Sciences
L.E. SCRIVEN,
University of Minnesota
LEON T. SILVER,
California Institute of Technology
CHARLES P. SLICHTER,
University of Illinois at Urbana-Champaign
ALVIN W. TRIVELPIECE,
Oak Ridge National Laboratory
SHMUEL WINOGRAD,
IBM T.J. Watson Research Center
CHARLES A. ZRAKET,
Mitre Corporation (retired)
NORMAN METZGER, Executive Director
OCR for page R7
Statistical Software Engineering
Preface
The development and the production of high-quality, reliable, complex computer software have become critical issues in the enormous worldwide computer technology market. The capability to efficiently engineer computer software development and production processes is central to the future economic strength, competitiveness, and national security of the United States. However, problems related to software quality, reliability, and safety persist, a prominent example being the failure on several occasions of major local and national telecommunications networks. It is now acknowledged that the costs of producing and maintaining software greatly exceed the costs of developing, producing, and maintaining hardware. Thus the development and application of cost-saving tools, along with techniques for ensuring quality and reliability in software engineering, are primary goals in today's software industry. The enormity of this software production and maintenance activity is such that any tools contributing to serious cost savings will yield a tremendous payoff in absolute terms.
At a meeting of the Committee on Applied and Theoretical Statistics (CATS) of the National Research Council (NRC), participants identified software engineering as an area presenting numerous opportunities for fruitful contributions from statistics and offering excellent potential for beneficial interactions between statisticians and software engineers that might promote improved software engineering practice and cost savings. To delineate these opportunities and focus attention on contexts promising useful interactions, CATS convened a study panel to gather information and produce a report that would (1) exhibit improved methods for assessing software productivity, quality, reliability, associated risk, and safety and for managing software development processes, (2) outline a program of research in the statistical sciences and their applications to software engineering with the aim of motivating and attracting new researchers from the mathematical sciences, statistics, and software engineering fields t o tackle these important and pressing problem areas, and (3) emphasize the relevance of using rigorous statistical and probabilistic techniques in software engineering contexts and suggest opportunities for further research in this direction.
To help identify important issues and obtain a broad range of perspectives on them, the panel organized an information-gathering forum on October 11-12, 1993, at which 12 invited speakers addressed how statistical methods impinge on the software development process, software metrics, software dependability and testing, and software visualization. The forum also included consideration of nonstandard methods and select case studies (see the forum program in the appendix). The panel hopes that its report, which is based on the panel's expertise as well as information presented at the forum, will contribute to positive advances in software engineering and, as a subsidiary benefit, be a stimulus for other closely related disciplines, e.g., applied mathematics, operations research, computer science, and systems and industrial engineering. The panel is, in fact, very enthusiastic about the opportunities facing the statistical community and hopes to convey this enthusiasm in this report.
The panel gratefully acknowledges the assistance and information provided by a number of individuals, including the 12 forum speakers—T.W. Keller, D. Card, V.R. Basili, J.C. Munson, J.C. Knight, R. Lipton, T. Yamaura, S. Zweben, M.S. Phadke, E.E. Sumne r, Jr., W. Hill, and J. Stasko—four anonymous reviewers, the NRC staff of the Board on Mathematical Sciences who supported the various facets of this project, and Susan Maurizi for her work in editing the manuscript.
OCR for page R8
Statistical Software Engineering
This page in the original is blank.
OCR for page R9
Statistical Software Engineering
Contents
EXECUTIVE SUMMARY
1
1
INTRODUCTION
5
2
CASE STUDY: NASA SPACE SHUTTLE FLIGHT CONTROL SOFTWARE
9
Overview of Requirements
9
The Operational Life Cycle
10
A Statistical Approach to Managing the Software Production Process
10
Fault Detection
11
Safety Certification
12
3
A SOFTWARE PRODUCTION MODEL
13
Problem Formulation and Specification of Requirements
14
Design
14
Implementation
16
Testing
18
4
CRITIQUE OF SOME CURRENT APPLICATIONS OF STATISTICS IN SOFTWARE ENGINEERING
27
Cost Estimation
27
Statistical Inadequacies in Estimating
29
Process Volatility
30
Maturity and Data Granularity
30
Reliability of Model Inputs
31
Managing to Estimates
32
Assessment and Reliability
32
Reliability Growth Modeling
32
Influence of the Development Process on Software Dependability
36
Influence of the Operational Environment on Software Dependability
37
Safety-Critical Software and the Problem of Assuring Ultrahigh Dependability
38
Design Diversity, Fault Tolerance, and General Issues of Dependence
38
Judgment and Decision-making Framework
39
Structural Modeling Issues
40
Experimentation, Data Collection, and General Statistical Techniques
40
Software Measurement and Metrics
41
5
STATISTICAL CHALLENGES
43
Software Engineering Experimental Issues
43
Combining Information
46
Visualization in Software Engineering
48
OCR for page R10
Statistical Software Engineering
Configuration Management Data
49
Function Call Graphs
50
Test Code Coverage
50
Code Metrics
50
Challenges for Visualization
52
Opportunities for Visualization
52
Orthogonal Defect Classification
59
6
SUMMARY AND CONCLUSIONS
61
Institutional Model for Research
62
Model for Data Collection and Analysis
62
Issues in Education
64
REFERENCES
67
APPENDIX: FORUM PROGRAM
72
Items in cart [0]
