ponent level. Clearly, a direct way to address this issue is to use developmental testing that is more representative of operational use, to the extent possible.
Ellner pointed out some assumptions that limit the applicability of the exponential reliability growth model, though he was relatively confident that these could be addressed. For Gaver’s model, the greatest challenge is that of initial input, that is, the number of faults in each stage of the system and the probability of discovering a fault during a test. Ellner suggested that the number of faults can be assumed to be quite large for complicated systems, and that giving the probabilities of discovering a fault a hierarchical structure is also a promising generalization of the model.
Ellner also strongly supported Sen’s proposal regarding the use of many alternative models that are consistent with the data to assess model misspecification. If these models agree with respect to decisions, one can be confident; if they disagree, the discrepancy will have to be analyzed.
Ellner remarked that an AMSAA website2 and an Institute of Electrical and Electronics Engineers working group are both concerned with updating Military Handbook 189 on reliability growth management (U.S. Department of Defense, 1981). He suggested that efforts to update this handbook would be more successful if the responsibility were assigned to a specific organization.
In his discussion of the papers by Scholz and Meeker, James Crouch pointed out that DoD already makes considerable use of operational test and field performance data, at least in the area of reliability testing of jet engines. Performance data are used to manage and control various aspects of turbine engine reliability, specifically (1) the engine in-flight shutdown rate; (2) the rate at which the engine needs to be repaired; and (3) the line replaceable unit rate, the maintenance rate for replacement of the components that surround the engine. The use of operational test data is complicated by engine-to-engine variations (it is typical to develop only three or four prototypes for operational test), and the use of both operational test and field performance data is complicated by variations in operational use on which data are not easily collected. These problems are currently being addressed using modeling and simulation.
The Air Force uses Pareto charts (histograms of the number of failure occurrence reports by root-cause category) based on field performance data