tional test results. He suggested that the failure modes are so distinct that this linkage is unlikely to be useful for many systems. He added that he thought there was a place for Bayesian models in the evaluation of defense systems in development. A key issue for him is the justification of test sizes for operational testing. Williams’ comments elicited a discussion of the place of Bayes methods in operational evaluation. David Olwell said that the key issue is that priors need to be selected objectively. Francisco Samaniego added that a sensitivity analysis using an appropriately broad collection of possible priors would be especially important in DoD applications since assessment of the influence of prior assumptions should be part of the subsequent decision-making process concerning a system’s suitability.

A fourth panelist, Jack Ferguson (substituting for Hans Mark), returned to the theme of reliability management. He is convinced that testing and analysis must be moved upstream so that the system design is improved with respect to its operational performance as early as possible. These are the types of systems that generally work well in the field. Further, field data need to be used more often to update estimates of the costs of spares, maintenance, and so on.

Finally, we summarize discussions concerning the RAM Primer’s current value in disseminating state-of-the-art reliability methods to the DoD test and evaluation community and the possible form of an updated version. These discussions occurred primarily in the concluding panel session of the workshop.

The DoD test and evaluation community currently has limited access to expert statistical advice (see, e.g., National Research Council, 1998). It is typical (and has been for decades) in all four services for both operational test planning and operational test evaluation to be carried out by individuals with relatively limited statistical training. For this reason, the RAM Primer served an important function for many years in communicating omnibus, easily applied techniques for test planning and test evaluation with respect to measurement of system reliability, availability, and maintainability. The chapters of the RAM Primer cover basic definitions, reliability measures, test planning, reliability models and estimation, hypothesis testing and confidence intervals for reliability performance, data analysis, and reliability growth estimation. Also included are tables and charts for assistance in applying the methods described.

Steve Pollock pointed out a number of areas in which the RAM Primer is currently deficient. These include the lack of discussion of physics-of-



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement