. "Appendix B: Combining Information for Test Design with Staged Development." Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press, 2006.
The following HTML text is provided to enhance online
readability. Many aspects of typography translate only awkwardly to HTML.
Please use the page image
as the authoritative form to ensure accuracy.
Testing of Defense Systems in an Evolutionary Acquisition Environment
ditions and that the only major change will be for day-versus-night test scenarios. Then, it is intuitively clear that most of the test resources in stage k should be used to assess the improvement in night-versus-day scenarios, with substantially fewer resources allocated to testing the effect of sunny-versus-rainy conditions for the new system. The actual allocation can be decided informally, depending on the level of confidence in the knowledge that the addition of the new night vision capability will not affect the rainy-versus-sunny comparison.
A more formal way of allocating test resources can be based on a Bayesian framework for design of experiments (Chaloner and Verdinelli, 1995). The estimates and Var () from stage (k – 1) can be viewed as prior information for the unknown parameters in stage k. Since we do not expect the new capability to affect the sunny-versus-rainy test comparison, we can expect the parameter β10 to remain relatively unchanged in stage k. So the prior information from stage (k – 1) for β10 is quite reliable, and and Var () will serve as good estimates of the mean and variance of the prior distribution. However, we expect β20 to change considerably, so we have to place a lot less weight on the prior information from stage (k – 1) for this parameter. The exact decision will depend on our belief on how relevant the data from the previous stage are. For example, if we believe that the data from the previous stage still provides unbiased information, we can use as the mean and an inflated value of Var with the inflation factor depending on our judgment of the relevance. In the extreme case in which the previous data provide no useful information, we can use a “noninformative” prior for β20. Alternatively, if the performance is expected to improve with the new night vision capability, the comparison of night-versus-day from the previous stage can be used as a lower bound for the prior distribution. The Bayesian theory for optimal design can now be used to obtain appropriate allocation of test resources to the various scenarios. There are several optimality criteria, such as D-, A-, and G-criteria. A good review can be found in Chaloner and Verdinelli (1995).
The same ideas can be used if we have additional interaction terms of the form
or a model with factors or test scenarios with more than two levels and quadratic terms: