Reliability Issues for DOD Systems: Report of a Workshop (2002)

The final report of the National Research Council’s (NRC) Panel on Statistical Methods for Testing and Evaluating Defense Systems (National Research Council, 1998) was intended to provide broad advice to the U.S. Department of Defense (DoD) on current statistical methods and principles that could be applied to the developmental and operational testing and evaluation of defense systems. To that end, the report contained chapters on the use of testing as a tool of system development; current methods of experimental design; evaluation methods; methods for testing and assessing reliability, availability, and maintainability; software development and testing; and validation of modeling and simulation for use in operational test and evaluation. While the examination of such a wide variety of topics was useful in helping DoD understand the breadth of problems for which statistical methods could be applied and providing direction as to how the methods currently used could be improved, there was, quite naturally, a lack of detail in each area.

To address the need for further detail, two DoD agencies—the Office of the Director of Operational Test and Evaluation and the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics—asked the NRC’s Committee on National Statistics to initiate a series of workshops on statistical issues relevant to defense acquisition. The aim of each workshop is to inform DoD about the methods that represent the statistical state of the art and, through interactions of the statisti-

cal and defense communities, explore their relevance for DoD application.

The issue chosen for this first workshop was statistical methods for assessment of the reliability (including availability and maintainability) of defense systems. A list of potential topics for the workshop evolved from issues raised in Chapter 7 of the above-referenced NRC (1998) report; discussions with the sponsors identified other topics that are a high priority for DoD but were not specifically addressed in that report. Further discussions of the Committee on National Statistics subcommittee responsible for organizing the workshop narrowed these topics down to seven that were selected for focus: (1) global reliability test designs to direct defense system development; (2) recent developments in reliability growth modeling; (3) use of statistical modeling for combining information, with applications to developmental and operational testing; (4) methods for estimating reliability from field performance data; (5) modeling of fatigue; (6) reliability of software-intensive systems; and (7) assessment of life-cycle costs through the use of reliability analyses (see the Appendix for the workshop agenda).

For each topic, two speakers from either academia or industry were identified to provide presentations. In addition, one or more defense specialists with responsibilities directly relevant to each topic were identified. These individuals were asked to interact with the (nondefense) presenters in advance of the workshop to ensure that the presentations would reflect cognizance of the specific problems faced by DoD; the current methods used by DoD to address these problems; and the important, possibly unique, constraints under which DoD operates. They were also asked to serve as discussants for the relevant workshop session.

The Workshop on Reliability Issues for DoD Systems, held June 9-10, 2000, had multiple goals, partly because some of the described techniques are mature, while others are still undergoing active research. In addition, the intended audience for the workshop and this report comprises defense reliability experts, higher-level administrators who could help change the processes used in system development, and defense employees charged with the day-to-day responsibility of assessing the reliability of defense systems. With respect to this last segment of the intended audience, an important consideration for the workshop to address was how to communicate the more readily applied and broadly applicable techniques to those in the DoD

community who have the responsibility of evaluating the reliability of defense systems and who, by nature of other responsibilities and backgrounds, are generally less knowledgeable about statistical techniques than academic researchers. One possible way to facilitate such communication is to upgrade or replace DoD 3235.1-H, Test and Evaluation of System Reliability Availability and Maintainability: A Primer (issued in 1962, revised in 1982). This document, referred to henceforth as the RAM Primer, is commonly used (more so in some service test organizations, less in others) by those responsible for the design of tests of system reliability and the associated evaluation in the test service and other defense agencies.

Broadly considered, the goal of the workshop was to foster greater interaction between the academic and defense acquisition communities with regard to both those ideas whose applicability is still uncertain and those that are considered promising. A number of positive impacts for the DoD community were envisioned by the planners of the workshop. First, it was hoped that greater interest would be developed among the academic community in current issues of importance to the defense community involving reliability assessment. Another goal was to inform decision makers about ways of dealing with procedural or other constraints that may hamper the application of statistical techniques in the collection of and access to test and field data, and in the use of testing in the development of reliable systems. Finally, the workshop would acquaint the defense community with state-of-the-art techniques that are applicable to problems in defense system development.

The planned interactions between the statistical research community and DoD were expected to have, in equal measure, strong benefits for the participating researchers. Prominent among these was educating academics about the problems and constraints facing the defense acquisition community, which are often considerably different from those involved in analogous industrial applications.

The following eight key ideas represent a useful summary of the workshop sessions: (1) the advantages of methods for reliability growth management, (2) the benefits of broader understanding and use of modern methods for reliability estimation and testing, (3) the need for updating the RAM Primer, (4) gains from the use of alternative modeling approaches, (5) the advantages of state-of-the-art reliability growth models, (6) the po-

tential advantages of the application of methods for combining developmental and operational test information, (7) the development of statistical models of fatigue of materials, and (8) the need for greater use of physics-of-failure models and for modeling some failure sources separately. A short synopsis of each of these eight key ideas is presented below.

A general issue noted by many workshop participants is that defense systems do not satisfy their operational suitability requirements sufficiently often, and as a result DoD is spending too much for system redesigns, spares management, and maintenance. Speakers stressed that a change in emphasis is needed to address this problem, including greater focus on test and evaluation for suitability, but more important, use of a number of techniques that can help identify design flaws and provide assessments of reliability performance much earlier in system development. This problem relates to the following NRC (1998:105) recommendation:

A number of speakers stressed that much progress could result not only from technical changes, but also from management changes that would support a more comprehensive approach to reliability improvement. Such changes would include the following: (1) a change in the function of reliability assessment from that of a statistic used to support promotion decisions to an early and continuing objective measurement (combining a wide variety of types of information) that supports system development by helping to identify components in need of redesign or maturation; (2) the collection of information on system performance in the field to support assessment of life-cycle costs and therefore future decisions on system acquisition; (3) cataloguing of test information and field performance to support feedback loops and thereby improve system performance, as well as the design of future tests; (4) early detection of processes in trouble or “bad actors” (defined below); and (5) development of a better understanding of the relationship between reliability performance in developmental and op-

erational test. Many of these issues relate to the following NRC (1998:120-21) recommendation:

Various speakers pointed out areas in which the DoD test and evaluation community could make greater use of modern methods for modeling various aspects of reliability and the advantages of doing so. Examples included (1) methods for combining information across test environments; (2) methods for incorporating subjective assessments; (3) fatigue modeling; (4) statistical methods for software engineering; (5) wider use of nonparametric methods, specifically for reliability growth but also more generally (e.g., for models for combining information and for uncertainty analysis); and (6) alternative methods for modeling reliability growth.

Given that the RAM Primer has been used to disseminate reliability methods throughout the defense test community for nearly 40 years and that its current version is substantially out of date with respect to a wide variety of currently accepted techniques, a number of speakers strongly suggested that it be updated, possibly in a substantially different format. One possibility mentioned was to have the RAM Primer be a web-based document with embedded software for carrying out the variety of calculations necessitated by modern methods. (This concern about datedness applies to other reliability-related military handbooks and military standards that are of similar vintage.) These suggestions relate to the following NRC (1998:126) recommendation:

Many speakers pointed out that the defense test community relies on particular models for specific purposes, the key examples being the power law process in reliability growth estimation and the exponential time-to-failure distribution for a wide variety of reliability questions. In these and other areas, use of alternative models would have the benefits of helping to identify when current models can be improved, to identify situations in which the inference is and is not robust across the different modeling approaches, and to provide an indication of the uncertainty due to model misspecification. Nonparametric models are particularly useful for these purposes since they make relatively few distributional assumptions and are therefore generally applicable. Since it is important that estimates of the uncertainty of reliability estimates take into account as many sources of uncertainty as possible for the benefit of decision makers, it is valuable to use alternative modeling approaches, which can be used to provide an assessment of the uncertainty of estimates due to model misspecification. These suggestions reinforce the following NRC (1998:113) recommendation:

Speakers suggested the wider use of reliability growth models that are consistent with the maturation process of test, analyze, and fix. Otherwise, since the power law process and related approaches do not explicitly take into account the results of the process of discovering and fixing the faults found in testing, reliability growth models are liable to produce inaccurate predictions. Two models were proposed that assign probabilities of detection to various system faults. These models use the natural assumption that earlier testing is more likely to find errors that have a higher probability of being discovered, whereas subsequent testing is more likely to discover the less probable errors. Such approaches are consistent with the test, analyze, and fix process. They can therefore can be used to examine the maturation that would result from a specific test design given various characteristics of the system under test, and they can provide an estimate of the reliability of the “matured” system.

Speakers described several new approaches based on the use of models for combining information from developmental and operational test when the failure modes in these separate environments of use are well understood (or otherwise satisfy the necessary assumptions underlying the models). In those instances, use of these models can make operational tests much more informative and thereby save test funds in comparison with methods that do not combine information. Use of this approach relates to the following NRC (1998:119) recommendation:

Especially given the recent use of defense systems for longer periods of time, fatigue modeling is clearly of critical importance to DoD. Speakers pointed out that the use of fatigue modeling that derives from a statistical approach provides useful estimates in a variety of important and common DoD contexts. Special emphasis was placed on the need for understanding the science in applications involving fatigue. Presentations during the session on fatigue modeling included both illustrations of the necessary statistics/physical science partnership and descriptions of a variety of models motivated by recent research in materials science. Speakers suggested that fatigue modeling could be one of the techniques included in a revised RAM Primer.

Several speakers supported the greater use of physics-of-failure models (i.e., models that directly represent the physical basis for failure) whenever these approaches are applicable. Use of such models would generally provide better estimates of various characteristics of system failure as compared with models not linked to specific failure modes. Discussion of the benefits of physics-of-failure models, along with leading examples, could also be included in a revised RAM Primer.

Further, several speakers proposed dividing failures into those of mature components and those of immature components, the latter through either design or production flaws. For example, the Integrated Reliability Growth Strategy classifies design failures into type A and type B modes, i.e., associated with mature and immature design components. Further, failures in the field are typically overrepresented by poorly produced components, referred to as “bad actors.” The identification and separate modeling of failures from type B modes and from bad actors is an approach to reliability estimation that could be effective in providing better estimates of system and fleet reliability. One speaker also mentioned “special-cause” failures—those that are unpredictable and are due, for example, to changes

in manufacturing processes. The study of special-cause failures could also benefit from separate treatment for modeling and prediction.

As is true with most workshop reports, this report is intended to capture the flavor of the workshop and highlight its primary and most useful ideas and discussion. While the report does not represent a comprehensive transcript of the proceedings, it should certainly serve as a helpful guide to current trends in reliability research and practice that have special relevance to DoD applications. We recognize that much more could be said on statistical modeling, reliability, and system development, perhaps delving into such topics as experimental design for reliability measurement, estimation of reliability for highly reliable systems (so that the probability of not observing a failure during testing is sizable), complications posed by destructive measurements, repeated measure degradation studies, analysis of recurrent events, and use of simulation-based techniques. Perhaps some or all of these topics will be addressed in future workshops in this proposed series.

The next chapter examines methods for measuring and managing reliability growth that were presented at the workshop. This is followed by a chapter on important areas of current research in reliability modeling and inference. The final chapter presents a general discussion of reliability issues and examines the need, expressed on several occasions during the workshop, to disseminate well-understood, broadly applicable methods for reliability test and evaluation, possibly through a comprehensive overhaul of the RAM Primer and other DoD documents that focus on reliability.