tion. However, there are also important differences with traditional use of sequential designs. Evolutionary acquisition represents a “block” sequential case, in which experiments corresponding to a particular stage are carried out in blocks. Furthermore, the system under study changes from stage to stage; unlike many other sequential studies, the decision on what to do in the next stage does not necessarily depend on results from the previous stage. Rather, this decision is based on strategic consideration about the new additional capabilities that are needed for the system in the field.

In each block there are several types of experimental strategies that could be considered, such as screening experiments for identifying important factors, response-surface designs for determining the optimal factor combinations, and so on.1 Their usefulness for developmental and operational testing is discussed in Statistics, Testing and Defense Acquisition: New Approaches and Methodological Improvements (National Research Council, 1998; see also National Research Council, 2004; Box, Hunter, and Hunter, 1978; Box and Draper, 1987; Myers and Montgomery, 2001; Wu and Hamada, 2000). This section provides a qualitative description of how one could use information from past stages to improve test design in the current stage. A more technical description of some of these ideas is given in Appendix B.

One can use test results and field performance of the system from the previous stage in both qualitative and quantitative ways in test design. These involve obtaining information on (a) factors that were found to be unimportant in previous tests; (b) how the performance (response surface) varied as a function of changes to key inputs or test scenarios; (c) hazard rate behavior of failure data, such as increasing hazard rate, infant mortality, etc., for reliability test design; (d) estimates of variability that are needed for allocating the test resources to different test scenarios in the current stage; and so on. Results from the earlier stages can also suggest whether to oversample or undersample certain test scenarios in the current stage of development or to push the testing envelope in certain directions. The system in a subsequent stage in development could be tested in problematic circumstances to see whether reliability growth or the addition of new com-

1  

The naïve use of both screening experiments and response-surface models will almost always be inferior to the use of techniques that are developed in collaboration with system experts, who can help guide the choice of variables and model forms.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement