Statistics, Testing, and Defense Acquisition New Approaches and Methodological Improvements (1998) / Chapter Skim
Currently Skimming:
4 Upgrading Statistical Methods for Testing and Evaluation
4 Upgrading Statistical Methods for Testing and Evaluation
Pages 50-60
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 50... ...
These changes are implementable in the short term and do not, generally speaking, require the institution of the new paradigm recommended here, although some of them would be more effective if implemented concurrently. Detailed recommendations related to test design, test evaluation, design and evaluation for reliability, availability, and maintainability, software test methodology, and use of modeling and simulation are in the chapters that follow; this chapter presents a less technical review of the inadequacy of current statistical practice in defense testing and the benefits to be gained from use of the best current methods and practices.
|
From page 51... ...
KEY ISSUES ILLUSTRATING THE USES OF STATISTICAL METHODS IN OPERATIONAL TESTING AND EVALUATION Test Planning and Design Test planning consists of collecting specific information about various characteristics of a system and the anticipated test scenarios and environments and recognizing the implications of this information for test design. Test planning is crucial to a test's success.
|
From page 52... ...
Establishing Standardized, Consistent Data Recording Procedures Operational tests are, in part, unscripted activities for which data collection is clearly complicated. To use test results from a given system for test design or evaluation of another (related)
|
From page 53... ...
The above components of test planning would be extremely helpful for the hypothetical major of Chapter 1 as a checklist when designing a complicated operational test and would assist the major in communicating with an experimental design expert. Templates for this purpose exist in the statistical literature and could be modified to be more specific to the operational testing of defense systems.
|
From page 54... ...
The feasibility of preliminary testing should be fully explored by service test agencies as part of operational test planning. Test Analysis and Reporting In order to fully use the information collected in an operational test, it is important that all reported test results typically averages and percentages be accompanied by an assessment of their uncertainty.
|
From page 55... ...
Estimates of uncertainty for performance in important individual scenarios should also be provided, as should information about variability due to model misspecification and its effect on simulation results. Combining Information from All Appropriate Sources for Test Design and Evaluation Sources of information available on the performance of a defense system under development, before operational testing, include the test results and field use of the system that is intended to be replaced, the performance of similar components on other systems currently in use, the results of developmental tests, data from possibly less controlled situations such as training exercises or contractor test results, and early operational assessments or the preliminary testing suggested above.
|
From page 56... ...
Recommendation 4.3: Information from tests and field use of related systems, developmental tests, early operational tests, and training and contractor testing should be examined for possible use in appropriate combination, when defensible, and with operational test results to achieve a more comprehensive assessment of system effectiveness and suitability. It is important to stress the use of the term appropriate.
|
From page 57... ...
Although some of these techniques and principles are finding their way into DoD's standard operational test design, current practice is still substantially distant from the state of the art. This has resulted in inefficient test designs, wasted resources, and less effective acquisition decision making.
|
From page 58... ...
Recommendation 4.5: Service test agencies should examine the use of alternative models to that of the exponential distribution for their appli cability to model failure-time distributions in the operational tests of defense systems. Software Testing Since ACAT I defense systems essentially now all have a software component and since software reliability is a common troublespot in recently developed defense systems, the proper testing of software in defense systems has a high priority.
|
From page 59... ...
Use of Modeling Modeling and simulation are now being widely considered and occasionally used by DoD to augment operational testing. Given the benefits of decreased cost, enhanced safety, and avoidance of environmental and other constraints, it is natural to explore the extent to which such use can contribute to operational testing.
|
From page 60... ...
60 STATISTICS, TESTING, AND DEFENSE ACQUISITION the model-test-model approach is to test the system again, but on new, possibly andomly selected scenarios and then, without refitting the model, measure the average distance between the model's predictions and the observed performance of the system. This approach could be given the name model-test-model-test, and it may be appropriate to use in augmenting some operational tests.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.