Reliability Issues for DoD Systems
REPORT OF A WORKSHOP
Francisco Samaniego and Michael Cohen, Editors
THE NATIONAL ACADEMIES PRESS
Washington, D.C. www.nap.edu
THE NATIONAL ACADEMIES PRESS
500 Fifth Street, NW Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
The project that is the subject of this report was supported by contract DASW01-94-C-0119 between the National Academy of Sciences and the Director of Operational Test and Evaluation at the Department of Defense. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.
International Standard Book Number 0-309-08606-X
Additional copies of this report are available from the
National Academies Press,
500 Fifth Street, NW, Lockbox 285, Washington, DC 20055 (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu
Copyright 2002 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Research Council (2002) Reliability Issues for DoD Systems: Report of a Workshop. Committee on National Statistics. Francisco Samaniego and Michael Cohen, editors. Washington, DC: The National Academies Press.
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National Research Council.
COMMITTEE ON NATIONAL STATISTICS 2002
JOHN E. ROLPH (Chair),
Marshall School of Business, University of Southern California
JOSEPH G. ALTONJI,
Department of Economics, Northwestern University
ROBERT BELL,
AT&T Laboratories, Florham Park, New Jersey
LAWRENCE D. BROWN,
Department of Statistics, University of Pennsylvania
ROBERT M. GROVES,
Survey Research Center, The University of Michigan
HERMANN HABERMANN,
Statistics Division, United Nations, New York
JOEL HOROWITZ,
Department of Economics, Northwestern University
WILLIAM D. KALSBEEK,
Survey Research Unit, Department of Biostatistics, University of North Carolina
ARLEEN LEIBOWITZ,
Department of Policy Studies, School of Public Policy, University of California at Los Angeles
RODERICK J.A. LITTLE,
School of Public Health, University of Michigan
THOMAS A. LOUIS,
Division of Biostatistics, University of Minnesota
DARYL PREGIBON,
AT&T Laboratories, Florham Park, New Jersey
NORA CATE SCHAEFFER,
Department of Sociology, University of Wisconsin-Madison
MATTHEW D. SHAPIRO,
Department of Economics, University of Michigan
ANDREW A. WHITE, Director
Preface and Acknowledgments
The workshop with which this volume is concerned has a number of important historical antecedents. The Department of Defense (DoD) has long been keenly interested in quantitative analyses assessing the performance of the hardware and software used in its various activities. Its interest in the possibility of revisiting the statistical techniques emphasized in its handbooks and manuals since the early 1960s began in earnest with a DoD-sponsored workshop hosted by the National Academies in September 1992. That workshop served as the stimulus for a study by a panel of the Committee on National Statistics, National Research Council. The panel’s report, Statistics, Testing and Defense Acquisition: New Approaches and Methodological Improvements (1998) strongly advocates various forms of modernization in the statistical practices used in the various stages of the DoD acquisition process.
Two individuals within DoD played key roles in progressing to the essential next stage: active engagement between the statistical research community and DoD statisticians, engineers, and managers, with a view toward developing a common understanding of the agency’s current needs and the statistical research community’s most promising methods for addressing those needs. I wish to thank Nancy Spruill, Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics, and Ernest Seglie, Science Advisor to the Director of Operational Test and Evaluation, for recognizing the urgency of the problem facing DoD, for raising the level of consciousness about that urgency within the department and among
its own statistical contacts, and for their energetic work in developing the financial support for a new series of workshops on statistical topics of importance to DoD. In the workshop that is the subject of this report—the first in the series—they took a leading role in the identification of potential expert discussants from the defense community. Finally, Drs. Spruill and Seglie put the workshop organizers in touch with other scientists and administrators within DoD who served us as indispensable advisors on the organization and planning of the workshop. Among these, Allen Beckett, Dolores Etter, Hollis Hunter, Robert Nemetz, and Philip Rodgers merit special mention.
I would also like to acknowledge the very special contributions made by a distinguished group of statistical researchers who kindly accepted our invitation to make presentations at the workshop. Each of them took the time to engage in early discussions with DoD personnel working in the areas on which they would speak; this preliminary contact served to enhance both the relevance and the impact of their comments and contributions at the workshop. We gratefully acknowledge the contributions of each of the civilian presenters and discussants at the workshop: Wallace Blischke, University of Southern California; Jane Booker, Los Alamos National Laboratory; Frank Camm, the RAND Corporation; Larry Crow, General Dynamics; Siddhartha Dalal, Telcordia Technologies; Robert Easterling, Sandia National Laboratories; Donald Gaver, Naval Postgraduate School; William Meeker, Iowa State University; William Padgett, University of South Carolina; Stephen Pollock, University of Michigan; Jesse Poore, University of Tennessee; Ananda Sen, Oakland University; Samuel Saunders, Washington State University; Fritz Scholz, the Boeing Company; Duane Steffey, San Diego State University; and Michael Tortorella, Bell Laboratories. I also thank John Rolph and Bill Eddy of the Committee on National Statistics, who served with me as the organizing subcommittee for the workshop.
The workshop also benefitted greatly from extremely informative remarks by its defense community discussants: Allen Beckett, Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics; James Crouch, Wright-Patterson Air Force Base; Paul Ellner, Army Materiel System Analysis Activity; Jack Ferguson, Software Intensive Systems; Arthur Fries, Institute for Defense Analyses; Walter Hollis, Office of the Deputy Under Secretary of the Army; Fred Myers, Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics; Margaret Myers, Office of the Deputy Assistant Secretary of Defense; Ernest Seglie, Office of the Director of Operational Test and Evaluation; Nancy Spruill,
Office of the Under Secretary of Defense for Acquisitions, Resources and Analysis; and Marion Williams, Air Force Operational Test and Evaluation Command.
Several invited guests came to our rescue by serving as session chairs: Asit Basu, University of Missouri-Columbia; Henry Block, University of Pittsburgh; Philip Boland, University College, Dublin; Ronald Glaser, Lawrence Livermore National Laboratory; Patricia Jacobs, Naval Postgraduate School; and Simon Wilson, Trinity College, Dublin.
This workshop report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the Report Review Committee of the National Research Council. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the charge. The review comments and draft manuscript remain confidential to protect the integrity of the process.
We thank the following individuals for their review of this report: Luis Escobar, Department of Statistics, Louisiana State University; Hans Mark, Department of Aerospace Engineering, University of Texas; Vijay Nair, Department of Statistics, University of Michigan; and Nozer Singpurwalla, Department of Operations Research, George Washington University.
Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the content of the report nor did they see the final draft of the report before its release. The review of this report was overseen by William F. Eddy, Department of Statistics, Carnegie Mellon University. Appointed by the National Research Council, he was responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authors and the institution.
I would also like to thank Arthur Fries, Institute for Defense Analyses, who contributed greatly to the workshop by helping to identify speakers and to orient their presentations so they were as compatible as possible with the workshop goals. In addition, Art carefully read several drafts of the present report.
Finally, I thank Michael Cohen, Agnes Gaskin, and Julia Kisa, staff of the Committee on National Statistics. Michael Cohen did a yeoman’s job
as chief liaison between DoD and the Committee on National Statistics, and more than anyone else provided the impetus for the steady advance of both the scientific and practical issues associated with a successful workshop report. Michael developed the first draft of this report and has been a full partner with me since the beginning of this ambitious project. He is to be commended for his strong dedication to serving simultaneously the highest standards of the discipline and the most pressing needs of DoD. Agnes Gaskin, greatly assisted by Julia Kisa, handled all administrative details with great care and good humor. Finally, Eugenia Grohman managed the production of this report, including the enlistment of Rona Brier as technical editor, who served superbly in that role.
Francisco Samaniego, Chair
Workshop on Reliability Issues for Defense Systems