APPENDIX A
Overview of the Current Milestone Development Process
The current process used for the development of a single-stage defense system consists of several well-defined steps, each ending with a milestone, that is, a decision point (see Figure A-1). These represent established criteria as to whether the system design is ready to be promoted to the next step in the process. Two primary types of defense testing are associated with this process: developmental testing and operational testing (discussed in detail below). We first describe the milestone development process.
Milestone Development: The first phase of system development is concept refinement, and it culminates in Milestone A: new acquisition program approval. The second phase, technology development, ends in Milestone B, at which stage a well-defined system design has been agreed on for addressing an identified need. The third phase of development is system development and demonstration. Once the system design is verified using developmental testing and appropriate operational assessments, Milestone C authorizes production of prototypes that are likely to satisfy the operational requirements. During this phase, production verification testing as well as other tests are carried out. It concludes with operational testing and evaluation in support of the decision to enter into full-rate production. Passing the operational test generally promotes the system to full-rate production and deployment (fielding). The last phase of acquisition is operations and support, which includes maintenance of production, system support in the field, and any follow-on product improvements and any additional operational testing and evaluation that is needed.
We describe next the two types of testing—developmental and operational testing—that take place during the development process. Developmental testing includes testing of components, subsystems, and software to ensure that performance capability and reliability are designed into the system early. Developmental testing then proceeds to system-level and system-of-systems–level testing to ensure that the system has matured to the point at which it can be expected to meet initial operational test and evaluation and operational employment requirements. Operational testing is a test of the full production representative system in as operationally realistic a setting as possible, with representation of opposition forces, countermeasures, and participation by users with training equivalent to what would be available in the field under circumstances of typical use.
These two types of defense testing have different objectives. Developmental testing is used to determine the capabilities of components and of the full system against individual stress factors, to help identify failure modes. The intent is to discover changes that can greatly improve the overall design. In addition, developmental testing is intended to help produce a system that would be expected to pass operational testing in proceeding to full-rate production. To examine whether a system design is mature, it is vital to learn about the performance of each component and, to some extent, their interactions. Therefore, during developmental testing, testing under more strenuous circumstances, that is, testing at the edge of the envelope, may be conducted.
In operational testing, an implicit assumption that is often made is that the system design is basically mature and that, barring unusual circumstances, the system will soon be purchased and fielded. Therefore, the system is typically not stressed at excessive levels. Instead, the testing roughly is guided by the typical scenarios of use (e.g., as promulgated by the U.S. Army’s formal Operational Mode Summary/Mission Profile document). The objective of operational testing is usually stated as to confirm that the system satisfies its operational requirements. This means that significance tests at standard statistical levels would fail to reject the null hypothesis that the system’s performance satisfied its requirements. (The requirements are often stated in terms of averages over scenarios of use, in which the scenarios and their frequencies are governed by the Operational Mode Summary/Mission Profile document.)
Early operational assessments occupy a middle ground between developmental and operational testing. They may involve component testing with typical users or other means of gleaning operationally relevant insights.