National Academies Press: OpenBook
« Previous: Executive Summary
Suggested Citation:"1 Introduction." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×

1
Introduction

EVOLUTIONARY ACQUISITION: DEFINITION AND RATIONALE

The Department of Defense (DoD) recently adopted evolutionary acquisition, a dynamic strategy for the development and acquisition of its defense systems. While the term has been used in various ways in DoD documents, the following statement is consistent with its description in the official DoD Instruction 5000.2:

Evolutionary defense systems are those that are planned, in advance, to be developed in several stages through a single procurement program. Each stage is planned to produce a viable system which, if shown to be superior to existing capabilities, could be fielded. The system requirements for each stage of development may be specified in advance of a given stage or may be decided at the outset of that stage’s development.

According to E.C. Aldridge, Jr. (2002), “[evolutionary acquisition] will allow us to reduce our cycle time and speed the delivery of advanced capability to our war fighters.” The term “evolutionary acquisition” is used somewhat inconsistently within DoD, and the resulting confusion has served to obscure the original goal and intent of the process. The implica-

Suggested Citation:"1 Introduction." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×

tions of this issue are discussed in Chapter 4. In this report, we use only the terms “evolutionary acquisition” and “staged acquisition.”

Reducing acquisition time and costs is a problem of great importance to DoD, as it often takes a decade or longer to complete the development and delivery of new major military systems. Even a small reduction in the costs of an ACAT I (acquisition category I) system provides significant savings in program costs. One reason given for delay in the development of complex defense systems is the phenomenon of “excessive requirements.” These can be overly optimistic initial expectations of what can be achieved, or they can be add-ons to the initial requirements, revisions of specifications, or changes in designs that occur during development and production. While most of these requirements are well intentioned, the net result is often significant slippage in development schedules (thereby delaying delivery of the capability to users in the battlefield) and substantial increases in procurement costs.

Evolutionary acquisition has been advocated as a way to address these problems. The stages of evolutionary acquisition may correspond to new hardware and software capabilities, increased performance, or to system improvements traced to technological maturity and reliability growth advances. If the acquisition program actually encompasses a “system of systems,” then the stages can provide for an increase in the capabilities or performance levels of individual systems, or they can represent new systems that are incrementally added to the system of systems. With evolutionary acquisition, initial and intermediate versions of the system will be released to the field if the decision is made that the system at that stage is superior to those currently in use. For this process to work, however, the initial system architecture must be designed so as to allow for the incorporation of changes dictated by the capabilities or improvements at later stages.

The evolutionary acquisition framework ideally allows for a fielded system to undergo further improvement without initiating a new procurement program. Thus, new technologies (particularly software) can be quickly deployed to enhance capabilities in the field, providing flexibility in responding to changing military missions, threats, and operating environments. By comparison, the traditional acquisition process, with its long lead times, often results in buying systems containing technologies that are outdated when finally fielded.

The idea of modifying fielded defense systems to improve their performance is, of course, not new. The Services have been developing substantial upgrades to existing defense systems for many years. However, many of

Suggested Citation:"1 Introduction." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×

these improvements were not planned for in advance in traditional single-stage development programs. As a result, the major block upgrades often have taken substantial amounts of time, redesign work, and cost. In evolutionary acquisition, in contrast, the planning for such major system improvements is supposed to be undertaken in advance to minimize the costs of redesign and retrofitting and to expedite the delivery to the field.

GOALS OF THE REPORT

The Office of the Under Secretary of Defense for Acquisition, Technology and Logistics (USD-AT&L) and the Director of Operational Testing and Evaluation (DOT&E) asked the Committee on National Statistics (CNSTAT) of the National Academies to examine the key issues and implications for defense testing from the introduction of evolutionary acquisition. The specific charge was as follows: “This study will involve the planning and conduct of a workshop to study test strategies for evolutionary acquisition. The study committee will review defense materials defining evolutionary acquisition and will interview test officials from the three major test service agencies to better understand the current approaches used to test systems procured using evolutionary acquisition. Possible alternatives will be examined to identify problems in implementation.”

CNSTAT set up an oversight committee to plan and conduct the workshop and write a report based on the presentations and discussions during the workshop and subsequent deliberations among committee members. The members were selected for their expertise in defense acquisition, software system development, industrial system development, operations research, and experimental design. Several teleconferences and meetings at the Pentagon assisted in the planning of the workshop. The committee also examined information in relevant defense documents, the academic literature, reports of the Government Accountability Office (see, e.g., U.S. General Accounting Office, 2002; U.S. Government Accountability Office, 2004), and commercial industrial processes.

The workshop took place on December 13-14, 2004. Participants represented a broad range of expertise from the DoD community, industry, and academia. The workshop agenda and participants are presented in Appendix D. The workshop was organized to learn about the evolutionary acquisition process itself, to examine the role of testing in an evolutionary environment, and to identify relevant best practices in industry and state-of-the-art methodology in the research literature.

Suggested Citation:"1 Introduction." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×

Some of the specific questions were: What are the appropriate roles and objectives for testing in an evolutionary environment? Can a systematic, disciplined process be developed for testing and evaluation in such a fluid and flexible environment? How can information from the earlier stages of the evolutionary acquisition process be used effectively in developing test designs for subsequent stages? Are there methodologies, either in the academic literature (statistics, operations research, management science, etc.) or best practices in industry that can be adapted for use in the evolutionary acquisition environment in DoD? Are there advantages to data archiving and documenting results from past stages of development? Is there adequate technical expertise within the acquisition community to fully exploit data gathered from previous stages and to effectively combine information from various sources for test design and analysis?

In the discussion of these questions, it became apparent that there are several broader, contextual issues that must also be addressed if the recommendations on test design are to be effective. Among these broader questions, the committee decided to focus on the following in the report: Is the meaning and intent of evolutionary acquisitions sufficiently clear within DoD, or is there a need for clarity and consistency in the terminology and enforcement of policies and procedures? Can the culture and organization of defense test and acquisition fully support the effective implementation of evolutionary acquisition? If not, what changes are needed in the DoD environment, the acquisition process, and incentives to ensure that the full benefits of testing in the evolutionary environment can be realized? Is the current level of cooperation among the program manager, contractors, and the developmental and operational testing communities adequate for supporting evolutionary acquisition?

Some of these questions raise issues beyond this study’s original scope or would have required more resources and expertise than were available. Nevertheless, we have addressed them, since they are critical to carrying out our overall charge. We make a number of general recommendations. Many of these will require substantial changes in the way program managers, test officials, and contractors carry out their responsibilities—changes that will be difficult to implement and institutionalize. Although developing the details is beyond the charge and the expertise of the committee, we stress that this organizational redesign will be critical to achieving the potential benefits of the evolutionary acquisition process and therefore requires the serious attention of DoD’s top management and technical leadership.

Suggested Citation:"1 Introduction." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×

ORGANIZATION OF THE REPORT

The rest of the report is organized as follows. Following this introduction, Chapter 2 addresses the role of testing and evaluation, broadly speaking, in contributing to evolutionary system development. Chapter 3 examines efficient strategies for test design and analysis by combining information in the context of staged system development. Chapter 4 addresses changes in the process and culture and infrastructure changes that are needed to support a more effective implementation of evolutionary acquisition, and in particular the methods proposed in Chapters 2 and 3.

Appendix A is an overview of the current single-stage milestone development process used in DoD. Appendix B provides more technical details in support of Chapter 3. Appendix C examines the special case of software development methods in the context of evolutionary acquisition. Appendix D presents the workshop agenda and the attendees. Appendix E presents biographical sketches of committee members and staff. Box 1-1 is a list of acronyms and abbreviations related to the topics of this report.

Box 1-1
Acronyms and Abbreviations

ACAT I

acquisition category I

ACTDs

advanced concept technology demonstrations

AoA

analysis of alternatives

APBs

acquisition program baselines

ATDs

advanced technology demonstrations

CNA

Center for Naval Analyses

CNSTAT

Committee on National Statistics

DoD

U.S. Department of Defense

DOT&E

Office of the Director of Operational Testing and Evaluation

FOC

full operating capability

FRP

full rate production

IOC

initial operating conditions

IOT&E

initial operational test and evaluation

LRIP

low rate initial production

OSD

Office of the Secretary of Defense

POS

parametric operational situation

RDT&E

research, development, test, and evaluation

USD-AT&L

Under Secretary of Defense for Acquisition, Technology and Logistics

Suggested Citation:"1 Introduction." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×
Page 9
Suggested Citation:"1 Introduction." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×
Page 10
Suggested Citation:"1 Introduction." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×
Page 11
Suggested Citation:"1 Introduction." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×
Page 12
Suggested Citation:"1 Introduction." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×
Page 13
Next: 2 Testing and Evaluation in an Evolutionary Acquisition Environment »
Testing of Defense Systems in an Evolutionary Acquisition Environment Get This Book
×
Buy Paperback | $29.00 Buy Ebook | $23.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Department of Defense (DoD) recently adopted evolutionary acquisition, a dynamic strategy for the development and acquisition of its defense systems. Evolutionary defense systems are planned, in advance, to be developed through several stages in a single procurement program. Each stage is planned to produce a viable system which could be fielded. The system requirements for each stage of development may be specified in advance of a given stage or may be decided at the outset of that stage's development. Due to the different stages that comprise an evolutionary system, there exists a need for careful reexamination of current testing and evaluation policies and processes, which were designed for single-stage developments.

The Office of the Under Secretary of Defense for Acquisition, Technology and Logistics (USD-AT&L) and the Director of Operational Testing and Evaluation (DOT&E) asked the Committee on National Statistics (CNSTAT) of the National Academies to examine the key issues and implications for defense testing from the introduction of evolutionary acquisition. The CNSTAT was charged with planning and conducting a workshop to study test strategies for the evolutionary acquisition. The committee reviewed defense materials defining evolutionary acquisition and interviewed test officials from the three major test service agencies to understand the current approaches used in testing systems procured through evolutionary acquisition. The committee also examined possible alternatives to identify problems in implementation.

At the workshop that took place on December 13-14, 2004, the committee tried to answer many questions including: What are the appropriate roles and objectives for testing in an evolutionary environment?, Can a systematic, disciplined process be developed for testing and evaluation in such a fluid and flexible environment?, and Is there adequate technical expertise within the acquisition community to fully exploit data gathered from previous stages to effectively combine information from various sources for test design and analysis?. Testing of Defense Systems in an Evolutionary Acquisition Environment provides the conclusions and recommendations of the CNSTAT following the workshop and its other investigations.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!