Reliability Growth
ENHANCING DEFENSE SYSTEM RELIABILITY
Panel on Reliability Growth Methods for Defense Systems
Committee on National Statistics
Division of Behavioral and Social Sciences and Education
NATIONAL RESEARCH COUNCIL
OF THE NATIONAL ACADEMIES
THE NATIONAL ACADEMIES PRESS
Washington, D.C.
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This study was supported by an award between the National Academy of Sciences and the U.S. Department of Defense through the National Science Foundation. Support for the work of the Committee on National Statistics is provided by a consortium of federal agencies through a grant from the National Science Foundation (award number SES-0453930). Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the organizations or agencies that provided support for this project.
International Standard Book Number-13: 978-0-309-31474-9
International Standard Book Number-10: 0-309-31474-7
Additional copies of this workshop summary are available for sale from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2015 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Research Council. (2015). Reliability Growth: Enhancing Defense System Reliability. Panel on Reliability Growth Methods for Defense Systems, Committee on National Statistics, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. C. D. Mote, Jr., is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Victor J. Dzau is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. C. D. Mote, Jr., are chair and vice chair, respectively, of the National Research Council.
This page intentionally left blank.
PANEL ON RELIABILITY GROWTH
METHODS FOR DEFENSE SYSTEMS
ARTHUR FRIES (Chair), Institute for Defense Analyses, Alexandria, VA
W. PETER CHERRY, Science Applications International Corporation (retired), Ann Arbor, MI
ROBERT G. EASTERLING, Statistical Consultant, Cedar Crest, NM
ELSAYED A. ELSAYED, Department of Industrial and Systems Engineering, Rutgers University
APARNA V. HUZURBAZAR, Statistical Sciences Group, Los Alamos National Laboratory, Los Alamos, NM
PATRICIA A. JACOBS, Operations Research Department, Naval Postgraduate School, Monterey, CA
WILLIAM Q. MEEKER, JR., Department of Statistics, Iowa State University
NACHI NAGAPPAN, Empirical Software Engineering Group, Microsoft Research, Redmond, WA
MICHAEL PECHT, Center for Advanced Life Cycle Engineering, University of Maryland
ANANDA SEN, Department of Family Medicine, University of Michigan Health System
SCOTT VANDER WIEL, Statistical Sciences Group, Los Alamos National Laboratory, Los Alamos, NM
MICHAEL L. COHEN, Study Director
ERNEST SEGLIE, Consultant
MICHAEL J. SIRI, Program Associate
COMMITTEE ON NATIONAL STATISTICS
2013-2014
LAWRENCE D. BROWN (Chair), Department of Statistics, The Wharton School, University of Pennsylvania
JOHN M. ABOWD, School of Industrial and Labor Relations, Cornell University
MARY ELLEN BOCK, Department of Statistics, Purdue University
DAVID CARD, Department of Economics, University of California, Berkeley
ALICIA CARRIQUIRY, Department of Statistics, Iowa State University
MICHAEL E. CHERNEW, Department of Health Care Policy, Harvard Medical School
CONSTANTINE GATSONIS, Center for Statistical Sciences, Brown University
JAMES S. HOUSE, Survey Research Center, Institute for Social Research, University of Michigan
MICHAEL HOUT, Department of Sociology, New York University
SALLIE KELLER, Virginia Bioinformatics Institute at Virginia Tech, Arlington, VA
LISA LYNCH, The Heller School for Social Policy and Management, Brandeis University
COLM O’MUIRCHEARTAIGH, Harris School of Public Policy Studies, University of Chicago
RUTH PETERSON, Criminal Justice Research Center, Ohio State University
EDWARD H. SHORTLIFFE, Department of Biomedical Informatics, Columbia University, and Department of Biomedical Informatics, Mayo Clinic Campus of Arizona State University
HAL STERN, Donald Bren School of Information and Computer Sciences, University of California, Irvine
CONSTANCE F. CITRO, Director
JACQUELINE R. SOVDE, Program Coordinator
Acknowledgments
We first thank Frank Kendall, the Under Secretary of Defense for Acquisition, Technology, and Logistics (AT&L) and Michael Gilmore, the Director of Operational Test and Evaluation (DOT&E) for their interest in and support for this study. Over the past 20 years, these two individuals and their predecessors have provided support for a series of related projects that have produced useful studies and, equally important, helped to establish a greater degree of collaboration between the defense testing community and leading members of the statistical, system engineering, and software engineering disciplines.
We also thank our primary contacts at the U.S. Department of Defense (DoD), Nancy Spruill, director of AT&L acquisition resources and analysis, and Catherine Warner, DOT&E science advisor, who were always ready to assist the panel’s work. They helped us clarify the issues for which we could have the greatest impact, they identified appropriate DoD staff to provide presentations at panel meetings to help us better understand the department’s current environment and operations, and they provided us with DoD documents (e.g., handbooks, guidances, memos, etc.) relevant to our study. We are greatly indebted to these four people for their help to the panel.
The panel members are also very indebted to the many experts who provided presentations at our first three panel meetings: Darryl Ahner (AFIT/ENS), Karen T. Bain (NAVAIR), Gary Bliss (USD AT&L), Albert (Bud) Boulter (USAF SAF/AQRE), Steve Brown (Lennox), David Burdick (Boeing), Michael J. Cushing (AEC, retired), Paul Ellner (AMSAA), Michael Gilmore (DOT&E), Martha Gardner (General Electric), Don Gaver (NPGS), Jerry Gibson (ASC/ENDR), Lou Gullo (Raytheon), Brian Hall (ATEC), Frank
Kendall (USD AT&L), Shirish Kher (Alcatel-Lucent), Eric Loeb (DOT&E), Andy Long (LMI), William McCarthy (OPTEVFOR), Stephan Meschter (BAE Systems), Andy Monje (DASD SE), Ken Neubeck (Exelisinc), David Nicholls (RIAC), Paul Shedlock (Raytheon), Tom Simms (USD AT&L), Nozer Singpurwalla (GWU), Jim Streilein (DOT&E), Patrick Sul (DOT&E), Daniel Telford (AFOTEC), Nicholas Torellis (OSD), Tom Wissink (Lockheed Martin), James Woodford (ASN(RD&A)), and Guangbin Yang (Ford).
We are grateful for the help from Michael Siri on administrative arrangements, and we thank Eugenia Grohman for extremely comprehensive technical editing.
The report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the Report Review Committee of the National Research Council. The purpose of this independent review is to provide candid and critical comments that assist the institution in making its reports as sound as possible and to ensure that the reports meet institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.
The panel thanks the following individuals for their review of the interim report: Karen T. Bain, Reliability and Maintainability, NAVAIR; Michael J. Cushing, U.S. Army Evaluation Center (Retired); Kathleen V. Diegert, Statistics and Human Factors, Sandia; Richard T. Durrett, Math Department, Duke University; Millard S. Firebaugh, Department of Mechanical Engineering, University of Maryland, College Park; Donald P. Gaver, Jr., Operations Research, Emeritus, U.S. Naval Postgraduate School; Pradeep Lall, Department of Mechanical Engineering, University of Auburn; Paul E. Shedlock, Reliability and System Safety Department, Engineering Product Support Directorate, Raytheon Company; Neil G. Siegel, Office of the Chief Technology Officer, Northrop Grumman Information Systems; and Marlin U. Thomas, Department of Industrial and Operations Engineering, University of Michigan.
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of the report was overseen by Thom J. Hodgson, Fitts Industrial and Systems Engineering Department, North Carolina State University and Roderick J. Little, Department of Biostatistics, School of Public Health, University of Michigan. Appointed by the National Research Council, they were responsible for making certain that the independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of the report rests entirely with the
authoring committee and the National Research Council. We also thank Ali Mosleh, Henry Samueli School of Engineering and Applied Science, University of California, Los Angeles, for his review of Appendix D. Finally, we thank the panel members who drafted large sections of this report and who devoted a great deal of time, energy, and expertise to this effort and always found time to contribute to the work.
Arthur Fries, Chair
Michael L. Cohen, Study Director
Panel on Reliability Growth Methods
for Defense Systems
This page intentionally left blank.
Contents
Panel Charge and Scope of Study
Achieving Reliability Requirements: Key History and Issues
Key Terms in Defense Acquisition
The Stages of Defense Acquisition
A Hard Problem, but Progress Is Possible
2 DEFENSE AND COMMERCIAL SYSTEM DEVELOPMENT: A COMPARISON
Issues in an Incentive System for Defense Acquisition
A Perspective on Commercial Best Practices
Continuously Operating Repairable Systems
5 SYSTEM DESIGN FOR RELIABILITY
Techniques to Assess Reliability Potential
Analysis of Failures and Their Root Causes
Two Approaches to Reliability Prediction
Redundancy, Risk Assessment, and Prognostics
6 RELIABILITY GROWTH THROUGH TESTING
Reliability Testing for Growth and Assessment
7 DEVELOPMENTAL TEST AND EVALUATION
Basic Elements of Developmental Testing
8 OPERATIONAL TEST AND EVALUATION
Timing and Role of Operational Testing
Software Reliability Growth Modeling
10 CONCLUSIONS AND RECOMMENDATIONS
An Outline Reliability Demonstration Plan
Raising the Priority of Reliability
Design for Reliability and Reliability Testing
Assessment of the Reliability of Electronic Components
Oversight of Software Development
Modeling in Conjunction with Accelerated Testing
Information on Operational Environments
Delivery of Prototypes for Developmental Testing
Intermediate Reliability Goals
A Recommendations of Previous Relevant Reports of the Committee on National Statistics
C Recent DoD Efforts to Enhance System Reliability in Development
D Critique of MIL-HDBK-217,
Anto Peter, Diganta Das, and Michael Pecht