Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 1
Testing of Defense Systems in an Evolutionary Acquisition Environment Executive Summary Evolutionary acquisition is a Department of Defense (DoD) process for defense system development in which a system is developed in stages as part of a single acquisition program. The different stages can be additional hardware and software capabilities or performance gains due to advances in technological maturity and reliability growth. DoD has presented evolutionary acquisition as the preferred option for development of key complex defense systems, a leading example being the Army’s Future Combat System. While it is quite common to modify a defense system after fielding, the intention in evolutionary acquisition is that these improvements are planned for and accommodated by the choice of system architectures and overall system design, to the extent possible. A further underlying motivation is that some costs and development delays (e.g., due to redesign and retrofitting) that might arise from a single-stage development process could be reduced by (a) giving greater priority to the identification of failure modes early in system development, (b) introducing new technologies only when they are mature, and (c) limiting the introduction of too many new components or subsystems simultaneously. It is argued that this process will shorten the overall system development time, allow rapid insertion of new capability-enhancing technologies, and reduce life-cycle costs. In evolutionary acquisition, system capabilities are developed and acquired in stages. Hence there is a need for careful reexamination of current testing and evaluation policies and processes, which were designed for
OCR for page 2
Testing of Defense Systems in an Evolutionary Acquisition Environment single-stage developments. At the request of DoD, a committee of the National Academies planned and conducted a workshop to discuss the role of testing and evaluation in an evolutionary acquisition environment and to make appropriate conclusions and recommendations. The specific questions addressed include: What are the appropriate roles and objectives for testing in an evolutionary environment? Can a systematic, disciplined process be developed for testing and evaluation in such a fluid and flexible environment? How can information from the earlier stages of the evolutionary acquisition process be used effectively in developing test designs for subsequent stages? Are there methodologies, either in the academic literature (statistics, operations research, management science, etc.) or best practices in industry that can be adapted for use in the evolutionary acquisition environment in DoD? Are there advantages to data archiving and documenting results from past stages of development? Is there adequate technical expertise within the acquisition community to fully exploit data gathered from previous stages and to effectively combine information from various sources for test design and analysis? While discussing these questions, it became apparent that there are several broader, contextual issues that must also be addressed if the recommendations on test design are to be effective. Among these issues, the following were considered in the report: Is the meaning and intent of evolutionary acquisition sufficiently clear in DoD, or is there a need for clarity and consistency in the terminology and a need for enforcement of policies and procedures? Can the culture and organization of defense test and acquisition fully support the effective implementation of evolutionary acquisition? If not, what changes are needed in the DoD environment, the acquisition process, and incentives to ensure that the full benefits of testing in the evolutionary environment can be realized? Is the current level of cooperation among the program manager, contractors, and the developmental and operational testing communities adequate for supporting evolutionary acquisition? While these broader issues are somewhat beyond the study’s original scope, the committee concluded that they must be discussed, even if only briefly. The committee draws some conclusions and makes some recommendations on these broader issues. However, the committee could not recommend how to address these problems fully due to the limited scope of this study.
OCR for page 3
Testing of Defense Systems in an Evolutionary Acquisition Environment CONCLUSIONS AND RECOMMENDATIONS Operational testing and evaluation, as portrayed in the current milestone system, supports a decision to pass or fail a defense system before it goes to large-scale procurement. The 1998 National Research Council report Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements proposed a new paradigm in which testing should be viewed as a “continuous process of information gathering and decision making in which operational testing and evaluation plays an integral role.” This new paradigm, originally suggested in the context of the traditional single-stage testing, is even more important in the evolutionary acquisition environment. Conclusion 1: In evolutionary acquisition, the entire spectrum of testing activities should be viewed as a continuous process of gathering, analyzing, and combining information in order to make effective decisions. The primary goal of test programs should be to experiment, learn about the strengths and weaknesses of newly added capabilities or (sub)systems, and use the results to improve overall system performance. Furthermore, data from previous stages of development, including field data, should be used in design, development, and testing at future stages. Operational testing (testing for verification) of systems still has an important role to play in the evolutionary environment, although it may not be realistic to carry out operational testing comprehensively at each stage of the development process. Recommendation 1: The under secretary of defense (acquisition, technology and logistics) and the director of operational test and evaluation should revise DoD documents and processes (e.g., DoD Directive 5000.1 and DoD Instruction 5000.2) to explicitly recognize and accommodate a framework in which the primary goal of all acquisition testing and evaluation programs is to experiment, learn about the strengths and weaknesses of system components, and to incorporate these results into system enhancement initiatives. Under such a continuous learning process, testing activities must go beyond the traditional framework of focusing on estimating the performance of a system under typical test scenarios.
OCR for page 4
Testing of Defense Systems in an Evolutionary Acquisition Environment Conclusion 2: Testing early in the development stage should emphasize the detection of design inadequacies and failure modes. This will require testing in more extreme conditions than those typically required by either developmental or operational testing, such as highly accelerated stress environments. However, the current incentive structure in DoD may discourage testing outside the envelope or identifying limitations and failure modes early in the process. Conclusion 3: To have a reasonable likelihood of fully implementing the paradigm of testing to learn about and to improve systems prior to production and deployment, the roles of DoD and congressional oversight in the incentive system in defense acquisition and testing must be modified. In particular, incentives need to be put in place to support the process of learning and discovery of design inadequacies and failure modes early and throughout system development. In evolutionary acquisition, it will be practical to conduct full-scale operational tests only at stages with major upgrades or substantive new capabilities. At other stages, only developmental tests of components and subsystems and some limited tests on functionality, interoperability, etc., will be feasible. Thus, these tests should use operational realism to the extent needed to assess the performance of components and subsystems from an operational perspective. Recommendation 2: The under secretary of defense (acquisition, technology and logistics) and the director of operational test and evaluation should revise DoD testing procedures to explicitly require that developmental tests have an operational perspective (i.e., are representative of real-world usage conditions) in order to increase the likelihood of early identification of operational failure modes and system deficiencies, so that appropriate actions can be developed and deployed in a timely fashion. Conclusion 4: In the evolutionary acquisition environment, effective system development and optimization will require a high degree of coordination and communication among system developers, government testers, and system users. In particular, government testers should
OCR for page 5
Testing of Defense Systems in an Evolutionary Acquisition Environment have early access to all contractor data sources, including test plans, results of early stage testing and experimentation, and the results of all pertinent modeling and simulation products. Recommendation 3: The under secretary of defense (acquisition, technology and logistics) should develop and implement policies, procedures, and rules that require contractors to share all relevant data on system performance and the results of modeling and simulation developed under government contracts, including information on their validity, to assist in system evaluation and development. The traditional single-stage acquisition environment can encourage the adoption of risky, immature technology into an existing system, since it may take a decade or more before a new technology can be incorporated. Evolutionary acquisition provides opportunities to discipline this process by delaying the introduction of risky, immature technology to future stages, and by using advanced technology demonstrations or advanced concept technology demonstrations to further develop and evaluate immature technologies. This will eliminate the possibility of delay of the entire acquisition program due to the use of a single risky technology, or risk using technology that has not been demonstrated to be sufficiently effective or reliable. Recommendation 4: The under secretary of defense (acquisition, technology and logistics) should require that all technologies to be included in a formal acquisition program have demonstrated sufficient technological maturity before the acquisition program is approved or before the technology is inserted in a later stage of development. The decision about the sufficiency of technological maturity should be based on an independent assessment from the director of defense research and engineering or special reviews by the director of operational test and evaluation (or other designated individuals) of the technological maturity assessments made during the analysis of alternatives and during developmental testing and evaluation. Regardless of the introduction of evolutionary acquisition, the increasing complexity of defense systems implies that a single all-encompassing,
OCR for page 6
Testing of Defense Systems in an Evolutionary Acquisition Environment large-scale operational test, as currently practiced, will not be feasible in many cases. Given the sheer numbers of subsystems, components, materials, software, and resulting system interactions, it is unlikely that the operational evaluation of complex systems can be based primarily on an operational test representing the full spectrum of combat conditions. Conclusion 5: The DoD testing community should investigate alternative strategies for testing complex defense systems to gain, early in the development process, an understanding of their potential operational failure modes, limitations, and level of performance. There are increased opportunities for combining information to improve test design and analysis in an evolutionary acquisition environment, given data from developmental, operational, and field tests of the system from earlier, fielded versions. Chapter 3 and Appendix B discuss specific examples and methods for combining information for improved test design in a multistage development process that is inherent in evolutionary acquisition. However, the gains from the use of these methods can be assessed only by applying them to defense systems in development. Furthermore, combining information from the various sources, which rely on relevant linkages between previous and current sources of data, requires considerable care and subject-matter expertise. When done correctly, however, it has a tremendous payoff. Recommendation 5: The Service test agencies should undertake a pilot study, involving a few selected systems developed under the evolutionary acquisition paradigm, in order to identify specific opportunities for incorporating information from previous stages to improve system design and analysis. These case studies will be beneficial in demonstrating both the application of the various techniques and the benefits to be gained from combining data in staged development. In order to effectively implement evolutionary acquisition, the DoD testing community must have access to greater expertise in various areas. There is also a need for systematically archiving data from various sources and combining them effectively for improved test design.
OCR for page 7
Testing of Defense Systems in an Evolutionary Acquisition Environment Recommendation 6: (a) To support the implementation of evolutionary acquisition, DoD should acquire, either through hiring in-house or through consulting or contractual agreements, greater access to expertise in the following areas: (1) combining information from various sources for efficient multistage design, statistical modeling, and analysis; (2) software engineering; and (3) physics-based and operational-level modeling and simulation. (b) Test and field data archives should be established to facilitate access to data on test and field performance of systems in development and those fielded. These data archives should be used to support feedback loops to improve the system design, to improve testing methodology over time, and to help validate (and improve) modeling and simulation for operational evaluation. While evolutionary acquisition is formally defined in DoD Instruction 5000.2, the term has not been used in a consistent manner by DoD leadership or in various other DoD documents. The variety of terms, including evolutionary acquisition, incremental development, and spiral development, and their inconsistent usage have served to obscure the original goal and intent of the evolutionary acquisition process. Besides clear and consistent definitions, a consistent and definitive articulation of the goals and intention of the evolutionary acquisition process is needed, as well as supporting documentation detailing the specific changes that will need to be implemented throughout the acquisition community. Recommendation 7: The under secretary of defense (acquisition, technology and logistics) should eliminate inconsistencies in DoD Directive 5000.1 and DoD Instruction 5000.2 and clarify other significant memoranda and documents regarding evolutionary acquisition. All policies and procedures to be used in applying evolutionary acquisition principles to DoD acquisition programs should be strictly enforced. This clarification and enforcement should be applied to program management both in DoD and in all supporting contractor activities. Evolutionary acquisition is being folded into an acquisition environment that already has a counterproductive incentive system. The flexibilities
OCR for page 8
Testing of Defense Systems in an Evolutionary Acquisition Environment inherent in the evolutionary acquisition process present even greater opportunities for these counterproductive incentives to be expressed. Improving the incentive structure is perhaps the most onerous obstacle facing the DoD acquisition process if it is to meet the difficult challenges in acquiring and fielding complex, expensive systems in a timely, efficient manner. Recommendation 8: The deputy secretary of defense should charge a blue ribbon panel, including experts in organizational behavior, multiobjective decision making, and other relevant areas, to review the Defense Acquisition Performance Assessment panel’s proposed changes to DoD acquisition policies and procedures. This review should take place (if possible) before any changes to the policies are implemented in order to assess and improve their likelihood of being successfully implemented in the DoD and defense industry culture.
Representative terms from entire chapter: