As in industrial applications of experimental design for product testing, various circumstances can conspire to cause some runs to abort or cause the operational test designer to choose factors that were not planned in advance, thus compromising features such as the orthogonality of a design, causing confounding, and reducing the efficiency of the test. General discussion of these issues may be found in Hahn (1984). The panel is interested in understanding these real-life complications in DoD operational testing. As an example, the expense involved prohibits the use of more than one battleship in operational testing of naval systems, so it is impossible to measure the effects of differences in crew training on ship-to-ship variation.
A further complication is that there are typically a large number of system requirements or specifications, referred to as measures of performance or effectiveness, for DoD systems. It is not uncommon to have as many as several dozen important specifications, and thus it is unclear which single specification should be used in optimizing the design. One could choose a “most important” specification, such as the hit rate, and select a design that would be efficient in measuring that output. However, the result might be inefficiency in measuring system reliability, since reliability measurement typically involves a broader distribution of system ages and a larger test sample size than a test of hit rate. Various statistical methods can be applied in an attempt to compromise across outputs. The panel does not address this problem in this interim report (though it is discussed briefly in Appendix B), but we intend to examine it for our final report.
This chapter first presents the progress made by the panel in understanding how experimental design is currently used in DoD operational testing. This is followed by discussion of some concerns about certain aspects of the use of experimental design that we have investigated, although only cursorily. We next describe a novel approach proposed in an operational test conducted by the Army Operational Test and Evaluation Command (OPTEC) for the Army Tactical Missile System/Brilliant Anti-Tank system (a missile system that targets moving vehicles, especially tanks), which involves the use of experimental design in designing a small number of field tests to be used in calibrating a simulation model for the system. The panel is interested in understanding this approach, which may become much more common in the future as a method for combining simulations and field tests. The final section of the chapter describes the future work we plan to undertake before issuing our final report. As background material for the discussion in this chapter, a short history of experimental design is provided in Appendix B.
Some elements of our discussion—for example, testing at the center versus the edge of the operating envelope—focus on the question of how to define the design space (and associated factor levels). Other elements—for example, possible uses of fractional factorial designs—are concerned with approaches for efficiently testing a well-defined design space. Such considerations are thus complementary and are among the diverse issues that must be addressed in designing a sound operational test that yields accurate and valuable information about system performance.
To become better acquainted with the environment, characteristics, and constraints of DoD operational testing, the panel began by investigating a single system and following it from the test design, through the testing process, and finally to evaluation of the test results. After considering the current systems under test by OPTEC, the panel decided to examine the operational testing of the Apache Longbow helicopter. 1
The panel is extremely grateful for the cooperation of OPTEC in this activity, especially Henry Dubin, Harold Pasini, Carl Russell, and Michael Hall.