some of the input variables may be very difficult to control. A great deal of progress has been made on the simple linear version of this problem, but this is not the case for highly complicated nonlinear versions. Further, if one is required to make a prediction in a region of the x space where no tests have been run, one needs to extrapolate from the prediction error model, which necessitates a rather complete understanding of the underlying physics. This response-surface modeling could be equivalent in difficulty to building the computer model in the first place.

In the concluding panel session, Rob Easterling described some approaches that might address this problem: (1) leaving some x’s out of the model, (2) using simplified variable and parameter spaces, (3) using simplified prediction error models, and (4) using fractional two-level factorial experiments. Easterling said that it can be shown that

where the subscript x under the variance and expectation operators indicates averaging over x as it varies according to its multivariate distribution. Since mx is likely to vary much less than y*, this equation can be typically modified as follows:

There is a large body of research exploring how much variance there is in y* given the random variation in the inputs, using such methods as Latin Hypercube sampling. However, that term is a good estimate of the variance of y only if the second term, the variance of the prediction errors, is negligible. This is currently a relatively unexplored area of research.

A second panelist, Steve Pollock, stated that the DoD community needs to determine how best to apply the various methods described at the workshop. Doing so would entail directly implementing some methods when applicable and otherwise tailoring them, if necessary, to DoD’s specific needs. Pollock added that additional workshops (structured similarly to this one) should be organized at regular intervals to help keep DoD abreast of recent advances in reliability methods.

A third panelist, Marion Williams, recommended that great care be exercised in using developmental test results in combination with opera-



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement