principles. For each workshop session, one or more defense analysts prepared papers that were circulated in advance. These papers were a primer on the topic of the session and its associated statistical problems. Appendix A lists the sessions and their participants. For each of the six sessions, two statisticians were asked to prepare written comments in advance on that session's papers. Appendix B lists the briefing papers prepared for the workshop; these papers are occasionally referred to explicitly throughout this report. All 12 invited statistical commentators were also encouraged to read all the papers and the written comments in advance. After a brief period of learning one another's terminology, analysts from both the defense and the statistical communities succeeded to a remarkable degree in communicating effectively with each other about a variety of statistical issues in weapon system evaluation and testing.
Key steps in the evaluation and testing process are (1) cost and operational effectiveness analysis, known as COEA; (2) developmental testing; and (3) operational test and evaluation, called OT&E. These occur as a weapon system is being developed, but before it goes into full-scale production. Regular reviews by decision makers at designated steps, called milestones, assess the performance potential and costs of the system.
COEAs are submitted or updated by the military services at the milestones in the acquisition process to evaluate the costs and benefits of new systems and their alternatives in terms of operational effectiveness in meeting defense needs. COEAs frequently include simulation experiments, analysis of which requires the use of appropriate statistical methods to make comparisons between alternatives.
Early testing during a weapon system development program (developmental testing) covers a wide range that includes component testing, modeling and simulation of anticipated performance, and engineering systems testing. Developmental testing presents the first opportunity to measure the performance and effectiveness of the system against the criteria developed during the COEA. This stage may include a period of combined operational and developmental testing (OT/DT) that is conducted by the interested branch of service, thus providing input from the prospective user during system development.
OT&E is the process that the Department of Defense uses to assess whether the weapon system actually meets its planned capability before deciding whether to begin full-scale production. By law, the director of OT &E must report to Congress on these tests and, in particular, state whether the testing and evaluation performed were adequate and whether the results confirm that the items or components actually tested are effective and suit-