Testing of Defense Systems in an Evolutionary Acquisition Environment (2006) / Chapter Skim
Currently Skimming:
2 Testing and Evaluation in an Evolutionary Acquisition Environment
2 Testing and Evaluation in an Evolutionary Acquisition Environment
Pages 14-27
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 14... ...
proposed a new paradigm in which testing should be viewed as a "continuous process of information gathering and decision making in which operational testing and evaluation plays an integral role." This new paradigm, suggested in the context of the traditional single-stage development, is even more important in the evolutionary acquisition environment. With staged development, if a system has gone into full-scale production in Stage I, then a goal in subsequent stages typically will be to identify ways to efficiently incorporate additional capabilities.
|
From page 15... ...
Operational testing and evaluation (or testing for verification) will still have a major role to play, since it is the only way to verify that the systems are in fact operationally effective and suitable.
|
From page 16... ...
TESTING OUTSIDE THE ENVELOPE In a learning environment, testing should go beyond estimating typical performance of a new or modified system to also emphasize the early discovery of failure modes,1 deficiencies, and weakness in the system design. This will require adding more operational realism to developmental tests as well as testing with demanding and possibly even accelerated stress environments.
|
From page 17... ...
The experience of the acquisition community is that system reliability in operational tests is generally substantially worse than that in developmental tests. For example, operational test mean times to first failure typically are a factor of roughly two to four times shorter than those for develo p m e n t a l tests.
|
From page 18... ...
Using this space, parametric simulation-based analyses should be used to identify feasible stressful situations for operational testing. As noted elsewhere in this re 3The parametric operational situation space was originally developed in a capability based planning study to structure a versatile rapid reaction force for the North Atlantic Treaty Organization's Supreme Allied Commander for Europe in 1992.
|
From page 19... ...
This will require testing in more extreme conditions than those typically required by either developmental or operational testing, such as highly acceler ated stress environments. The current incentive structure in DoD may discourage testing outside the envelope or identifying component or system limitations and failure modes early in the process.
|
From page 20... ...
and the director of operational test and evaluation should revise DoD testing procedures to explicitly re quire that developmental tests have an operational perspective (i.e., are representative of real-world usage conditions) in order to in crease the likelihood of early identification of operational failure modes and system deficiencies, so that appropriate actions can be developed and deployed in a timely fashion.
|
From page 21... ...
This is accomplished by creating acquisition processes that make use of continuous monitoring and testing." Testing in the design and development stage will aid critical decisions on the choice of materials, design layout, selecting parameter values for optimizing system performance, and exploring robustness to the impact of various uncontrollable "noise" factors in manufacturing and field-use conditions. In the commercial sector, a considerable part of the design and development takes place within the company, with direct control over these decisions.
|
From page 22... ...
addressed the instability of requirements and performance characteristics for major DoD acquisition programs. On the basis of interviews, it reported that senior
|
From page 23... ...
. Demonstrating technological maturity before including the technology in a formal acquisition program will eliminate the need to delay the entire acquisition program because of one risky technology area or risk using technology that may not be sufficiently effective or reliable.
|
From page 24... ...
of the technological maturity assessments made during the analysis of alternatives and during developmental testing and evaluation. COMPARISON WITH BASELINE SYSTEMS IN EVOLUTIONARY ACQUISITION In traditional acquisition programs, operational tests and evaluations often test both the system under development and the baseline system (i.e., the control)
|
From page 25... ...
OPERATIONAL TESTING OF VERY COMPLEX DEFENSE SYSTEMS There is another consideration that is independent of the issue of evolutionary acquisition, and that is the increasing complexity of major defense systems. The complexity includes: (1)
|
From page 26... ...
As discussed above, extensive operational testing will be reserved for stages that have the most substantive changes involving the most components and subsystems, and will be limited to a relatively small number of test scenarios, even across stages of development. For other stages, testing will be limited to component, subsystem, and software testing.
|
From page 27... ...
Conclusion 5: The DoD testing community should investigate alter nate strategies for testing complex defense systems to gain, early in the development process, an understanding of their potential operational failure modes, limitations, and level of performance.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.