National Academies Press: OpenBook

Testing of Defense Systems in an Evolutionary Acquisition Environment (2006)

Chapter: Appendix A: Overview of the Current Milestone Development Process

« Previous: References
Suggested Citation:"Appendix A: Overview of the Current Milestone Development Process." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×

APPENDIX A
Overview of the Current Milestone Development Process

The current process used for the development of a single-stage defense system consists of several well-defined steps, each ending with a milestone, that is, a decision point (see Figure A-1). These represent established criteria as to whether the system design is ready to be promoted to the next step in the process. Two primary types of defense testing are associated with this process: developmental testing and operational testing (discussed in detail below). We first describe the milestone development process.

Milestone Development: The first phase of system development is concept refinement, and it culminates in Milestone A: new acquisition program approval. The second phase, technology development, ends in Milestone B, at which stage a well-defined system design has been agreed on for addressing an identified need. The third phase of development is system development and demonstration. Once the system design is verified using developmental testing and appropriate operational assessments, Milestone C authorizes production of prototypes that are likely to satisfy the operational requirements. During this phase, production verification testing as well as other tests are carried out. It concludes with operational testing and evaluation in support of the decision to enter into full-rate production. Passing the operational test generally promotes the system to full-rate production and deployment (fielding). The last phase of acquisition is operations and support, which includes maintenance of production, system support in the field, and any follow-on product improvements and any additional operational testing and evaluation that is needed.

Suggested Citation:"Appendix A: Overview of the Current Milestone Development Process." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×

FIGURE A-1 The defense milestone development process.

SOURCE: U.S. Department of Defense Instruction Number 5000.2. May 12, 2003 Under Secretary of Defense for Acquisition, Technology and Logistics. Available: <http://akss.dau.mil/dag/DoD5002/Figure1.asp>.

We describe next the two types of testing—developmental and operational testing—that take place during the development process. Developmental testing includes testing of components, subsystems, and software to ensure that performance capability and reliability are designed into the system early. Developmental testing then proceeds to system-level and system-of-systems–level testing to ensure that the system has matured to the point at which it can be expected to meet initial operational test and evaluation and operational employment requirements. Operational testing is a test of the full production representative system in as operationally realistic a setting as possible, with representation of opposition forces, countermeasures, and participation by users with training equivalent to what would be available in the field under circumstances of typical use.

These two types of defense testing have different objectives. Developmental testing is used to determine the capabilities of components and of the full system against individual stress factors, to help identify failure modes. The intent is to discover changes that can greatly improve the overall design. In addition, developmental testing is intended to help produce a system that would be expected to pass operational testing in proceeding to full-rate production. To examine whether a system design is mature, it is vital to learn about the performance of each component and, to some extent, their interactions. Therefore, during developmental testing, testing under more strenuous circumstances, that is, testing at the edge of the envelope, may be conducted.

Suggested Citation:"Appendix A: Overview of the Current Milestone Development Process." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×

In operational testing, an implicit assumption that is often made is that the system design is basically mature and that, barring unusual circumstances, the system will soon be purchased and fielded. Therefore, the system is typically not stressed at excessive levels. Instead, the testing roughly is guided by the typical scenarios of use (e.g., as promulgated by the U.S. Army’s formal Operational Mode Summary/Mission Profile document). The objective of operational testing is usually stated as to confirm that the system satisfies its operational requirements. This means that significance tests at standard statistical levels would fail to reject the null hypothesis that the system’s performance satisfied its requirements. (The requirements are often stated in terms of averages over scenarios of use, in which the scenarios and their frequencies are governed by the Operational Mode Summary/Mission Profile document.)

Early operational assessments occupy a middle ground between developmental and operational testing. They may involve component testing with typical users or other means of gleaning operationally relevant insights.

Suggested Citation:"Appendix A: Overview of the Current Milestone Development Process." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×
Page 45
Suggested Citation:"Appendix A: Overview of the Current Milestone Development Process." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×
Page 46
Suggested Citation:"Appendix A: Overview of the Current Milestone Development Process." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×
Page 47
Next: Appendix B: Combining Information for Test Design with Staged Development »
Testing of Defense Systems in an Evolutionary Acquisition Environment Get This Book
×
Buy Paperback | $29.00 Buy Ebook | $23.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Department of Defense (DoD) recently adopted evolutionary acquisition, a dynamic strategy for the development and acquisition of its defense systems. Evolutionary defense systems are planned, in advance, to be developed through several stages in a single procurement program. Each stage is planned to produce a viable system which could be fielded. The system requirements for each stage of development may be specified in advance of a given stage or may be decided at the outset of that stage's development. Due to the different stages that comprise an evolutionary system, there exists a need for careful reexamination of current testing and evaluation policies and processes, which were designed for single-stage developments.

The Office of the Under Secretary of Defense for Acquisition, Technology and Logistics (USD-AT&L) and the Director of Operational Testing and Evaluation (DOT&E) asked the Committee on National Statistics (CNSTAT) of the National Academies to examine the key issues and implications for defense testing from the introduction of evolutionary acquisition. The CNSTAT was charged with planning and conducting a workshop to study test strategies for the evolutionary acquisition. The committee reviewed defense materials defining evolutionary acquisition and interviewed test officials from the three major test service agencies to understand the current approaches used in testing systems procured through evolutionary acquisition. The committee also examined possible alternatives to identify problems in implementation.

At the workshop that took place on December 13-14, 2004, the committee tried to answer many questions including: What are the appropriate roles and objectives for testing in an evolutionary environment?, Can a systematic, disciplined process be developed for testing and evaluation in such a fluid and flexible environment?, and Is there adequate technical expertise within the acquisition community to fully exploit data gathered from previous stages to effectively combine information from various sources for test design and analysis?. Testing of Defense Systems in an Evolutionary Acquisition Environment provides the conclusions and recommendations of the CNSTAT following the workshop and its other investigations.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!