The Department of Defense (DoD) recently adopted evolutionary acquisition, a dynamic strategy for the development and acquisition of its defense systems. While the term has been used in various ways in DoD documents, the following statement is consistent with its description in the official DoD Instruction 5000.2:
Evolutionary defense systems are those that are planned, in advance, to be developed in several stages through a single procurement program. Each stage is planned to produce a viable system which, if shown to be superior to existing capabilities, could be fielded. The system requirements for each stage of development may be specified in advance of a given stage or may be decided at the outset of that stage’s development.
According to E.C. Aldridge, Jr. (2002), “[evolutionary acquisition] will allow us to reduce our cycle time and speed the delivery of advanced capability to our war fighters.” The term “evolutionary acquisition” is used somewhat inconsistently within DoD, and the resulting confusion has served to obscure the original goal and intent of the process. The implica-
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 9
Testing of Defense Systems in an Evolutionary Acquisition Environment 1 Introduction EVOLUTIONARY ACQUISITION: DEFINITION AND RATIONALE The Department of Defense (DoD) recently adopted evolutionary acquisition, a dynamic strategy for the development and acquisition of its defense systems. While the term has been used in various ways in DoD documents, the following statement is consistent with its description in the official DoD Instruction 5000.2: Evolutionary defense systems are those that are planned, in advance, to be developed in several stages through a single procurement program. Each stage is planned to produce a viable system which, if shown to be superior to existing capabilities, could be fielded. The system requirements for each stage of development may be specified in advance of a given stage or may be decided at the outset of that stage’s development. According to E.C. Aldridge, Jr. (2002), “[evolutionary acquisition] will allow us to reduce our cycle time and speed the delivery of advanced capability to our war fighters.” The term “evolutionary acquisition” is used somewhat inconsistently within DoD, and the resulting confusion has served to obscure the original goal and intent of the process. The implica-
OCR for page 9
Testing of Defense Systems in an Evolutionary Acquisition Environment tions of this issue are discussed in Chapter 4. In this report, we use only the terms “evolutionary acquisition” and “staged acquisition.” Reducing acquisition time and costs is a problem of great importance to DoD, as it often takes a decade or longer to complete the development and delivery of new major military systems. Even a small reduction in the costs of an ACAT I (acquisition category I) system provides significant savings in program costs. One reason given for delay in the development of complex defense systems is the phenomenon of “excessive requirements.” These can be overly optimistic initial expectations of what can be achieved, or they can be add-ons to the initial requirements, revisions of specifications, or changes in designs that occur during development and production. While most of these requirements are well intentioned, the net result is often significant slippage in development schedules (thereby delaying delivery of the capability to users in the battlefield) and substantial increases in procurement costs. Evolutionary acquisition has been advocated as a way to address these problems. The stages of evolutionary acquisition may correspond to new hardware and software capabilities, increased performance, or to system improvements traced to technological maturity and reliability growth advances. If the acquisition program actually encompasses a “system of systems,” then the stages can provide for an increase in the capabilities or performance levels of individual systems, or they can represent new systems that are incrementally added to the system of systems. With evolutionary acquisition, initial and intermediate versions of the system will be released to the field if the decision is made that the system at that stage is superior to those currently in use. For this process to work, however, the initial system architecture must be designed so as to allow for the incorporation of changes dictated by the capabilities or improvements at later stages. The evolutionary acquisition framework ideally allows for a fielded system to undergo further improvement without initiating a new procurement program. Thus, new technologies (particularly software) can be quickly deployed to enhance capabilities in the field, providing flexibility in responding to changing military missions, threats, and operating environments. By comparison, the traditional acquisition process, with its long lead times, often results in buying systems containing technologies that are outdated when finally fielded. The idea of modifying fielded defense systems to improve their performance is, of course, not new. The Services have been developing substantial upgrades to existing defense systems for many years. However, many of
OCR for page 9
Testing of Defense Systems in an Evolutionary Acquisition Environment these improvements were not planned for in advance in traditional single-stage development programs. As a result, the major block upgrades often have taken substantial amounts of time, redesign work, and cost. In evolutionary acquisition, in contrast, the planning for such major system improvements is supposed to be undertaken in advance to minimize the costs of redesign and retrofitting and to expedite the delivery to the field. GOALS OF THE REPORT The Office of the Under Secretary of Defense for Acquisition, Technology and Logistics (USD-AT&L) and the Director of Operational Testing and Evaluation (DOT&E) asked the Committee on National Statistics (CNSTAT) of the National Academies to examine the key issues and implications for defense testing from the introduction of evolutionary acquisition. The specific charge was as follows: “This study will involve the planning and conduct of a workshop to study test strategies for evolutionary acquisition. The study committee will review defense materials defining evolutionary acquisition and will interview test officials from the three major test service agencies to better understand the current approaches used to test systems procured using evolutionary acquisition. Possible alternatives will be examined to identify problems in implementation.” CNSTAT set up an oversight committee to plan and conduct the workshop and write a report based on the presentations and discussions during the workshop and subsequent deliberations among committee members. The members were selected for their expertise in defense acquisition, software system development, industrial system development, operations research, and experimental design. Several teleconferences and meetings at the Pentagon assisted in the planning of the workshop. The committee also examined information in relevant defense documents, the academic literature, reports of the Government Accountability Office (see, e.g., U.S. General Accounting Office, 2002; U.S. Government Accountability Office, 2004), and commercial industrial processes. The workshop took place on December 13-14, 2004. Participants represented a broad range of expertise from the DoD community, industry, and academia. The workshop agenda and participants are presented in Appendix D. The workshop was organized to learn about the evolutionary acquisition process itself, to examine the role of testing in an evolutionary environment, and to identify relevant best practices in industry and state-of-the-art methodology in the research literature.
OCR for page 9
Testing of Defense Systems in an Evolutionary Acquisition Environment Some of the specific questions were: What are the appropriate roles and objectives for testing in an evolutionary environment? Can a systematic, disciplined process be developed for testing and evaluation in such a fluid and flexible environment? How can information from the earlier stages of the evolutionary acquisition process be used effectively in developing test designs for subsequent stages? Are there methodologies, either in the academic literature (statistics, operations research, management science, etc.) or best practices in industry that can be adapted for use in the evolutionary acquisition environment in DoD? Are there advantages to data archiving and documenting results from past stages of development? Is there adequate technical expertise within the acquisition community to fully exploit data gathered from previous stages and to effectively combine information from various sources for test design and analysis? In the discussion of these questions, it became apparent that there are several broader, contextual issues that must also be addressed if the recommendations on test design are to be effective. Among these broader questions, the committee decided to focus on the following in the report: Is the meaning and intent of evolutionary acquisitions sufficiently clear within DoD, or is there a need for clarity and consistency in the terminology and enforcement of policies and procedures? Can the culture and organization of defense test and acquisition fully support the effective implementation of evolutionary acquisition? If not, what changes are needed in the DoD environment, the acquisition process, and incentives to ensure that the full benefits of testing in the evolutionary environment can be realized? Is the current level of cooperation among the program manager, contractors, and the developmental and operational testing communities adequate for supporting evolutionary acquisition? Some of these questions raise issues beyond this study’s original scope or would have required more resources and expertise than were available. Nevertheless, we have addressed them, since they are critical to carrying out our overall charge. We make a number of general recommendations. Many of these will require substantial changes in the way program managers, test officials, and contractors carry out their responsibilities—changes that will be difficult to implement and institutionalize. Although developing the details is beyond the charge and the expertise of the committee, we stress that this organizational redesign will be critical to achieving the potential benefits of the evolutionary acquisition process and therefore requires the serious attention of DoD’s top management and technical leadership.
OCR for page 9
Testing of Defense Systems in an Evolutionary Acquisition Environment ORGANIZATION OF THE REPORT The rest of the report is organized as follows. Following this introduction, Chapter 2 addresses the role of testing and evaluation, broadly speaking, in contributing to evolutionary system development. Chapter 3 examines efficient strategies for test design and analysis by combining information in the context of staged system development. Chapter 4 addresses changes in the process and culture and infrastructure changes that are needed to support a more effective implementation of evolutionary acquisition, and in particular the methods proposed in Chapters 2 and 3. Appendix A is an overview of the current single-stage milestone development process used in DoD. Appendix B provides more technical details in support of Chapter 3. Appendix C examines the special case of software development methods in the context of evolutionary acquisition. Appendix D presents the workshop agenda and the attendees. Appendix E presents biographical sketches of committee members and staff. Box 1-1 is a list of acronyms and abbreviations related to the topics of this report. Box 1-1 Acronyms and Abbreviations ACAT I acquisition category I ACTDs advanced concept technology demonstrations AoA analysis of alternatives APBs acquisition program baselines ATDs advanced technology demonstrations CNA Center for Naval Analyses CNSTAT Committee on National Statistics DoD U.S. Department of Defense DOT&E Office of the Director of Operational Testing and Evaluation FOC full operating capability FRP full rate production IOC initial operating conditions IOT&E initial operational test and evaluation LRIP low rate initial production OSD Office of the Secretary of Defense POS parametric operational situation RDT&E research, development, test, and evaluation USD-AT&L Under Secretary of Defense for Acquisition, Technology and Logistics