Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program (1993)

AIRCRAFT SPECIFIC

Overall Planning

• In general, JLF/Aircraft planning has been well organized and thorough.

• JLF/Aircraft established a formal process to designate test priorities; however, test priorities were actually driven by more pragmatic concerns (target availability and the need to ensure tri-service cooperation).

• The principal constraint on realism is the inability to simulate flight conditions on the ground. Airflow is used to simulate airspeed but the coverage area is small, and other environmental factors affecting fire are not simulated at all.

Test Planning

• JLF/Aircraft test designs are generally congruent with test objectives, efficient with respect to conserving targets, and realistic given their limited objectives.

• Some DTPS specified target requirements which exceeded the availability of those targets.

• Testers are highly sensitive to test efficiency from an engineering standpoint, i.e., designing tests to conserve targets and prevent testing effects.

• DTPS omit key information (e.g., data analysis plans) and are inconsistent in selection of threat velocities.

Implementation

• To the limited extent we could observe them, departures from test plans have generally been reasonable.

Analysis and Results

• Only one draft report has been completed—the F100 engine steady state fuel ingestion test. This report omitted key information, overstated the generalizability of results, and presented a highly questionable mode. Recommendations were congruent with results and sensitive to the likelihood of user acceptance.

GENERAL LFT ISSUES

Conflict over Objectives

• The JLF charter did not define live fire testing well enough to give test designers a clear direction.

• There have been several conflicting versions of the objectives of JLF and live fire testing in general. This appears to have in part resulted from the decision to task the JTCG’s to implement JLF.

• The conflict over objectives reflects underlying differences between the interests of proponents of full-up testing and those of modelers, resulting in largely incompatible approaches.

Availability of Targets

• The principal constraint faced by all JLF test officials is a lack of targets. This is in part a result of inadequate planning; there is no assigned responsibility to provide targets and related support to JLF. Consequently, test officials have had to spend a substantial portion of their time “selling” the program to skeptical service components.

• The systems and components that JLF does receive are frequently in poor condition, yet JLF provided no funds for restoration.

• JLF has been further hindered by competing governmental and non-governmental interests and negative attitudes toward destructive testing.

Shot Selection

• Controversy over shot selection is to some degree a conflict between sampling efficiency and the desire to avoid bias at all costs.

• Random sampling from combat distributions is a reasonable way to preclude intentional or inadvertent bias in shot selection. However, sampling from a uniform distribution avoids tester bias and biases in the combat data.

• The shot selection problem will not be resolved by technical solutions alone. An interim solution might be to designate that some proportion of shots be selected judgmentally and others randomly, but ultimately, it appears impossible to agree on how to select live fire shots without first deciding on test objectives.

Human Effects

• JLF plans do not provide an adequate treatment of human effects.

• The claims of some JLF officials that personnel vulnerability is well known are overstated.

• Given the current state of the art, it is unlikely that JLF will produce precise estimates of casualties.

Incentive Structure

• DoD’s incentive structure is not entirely conducive to realistic live fire testing.

COMPARISON PROGRAMS

Past Programs

• The state of the art of live fire testing has improved since prior live fire testing programs, but some potentially solvable problems raised earlier have not been solved. For example, little progress has been made in the empirical validation of V/L estimates

ADVANTAGES OF LIVE FIRE TESTING

• As the only method providing direct visual observation of the damage caused by a weapon/target interaction under realistic combat conditions, full-up live fire testing offers a unique advantage over all other methods of V/L assessment.

• The descriptions of directly observable damage that full-up testing provides are regarded as highly beneficial by users.

• Full-up testing has already demonstrated some value by producing several “surprises,” i.e., results that were not predicted, and might not have been detected by other methods of testing or analysis.

LIMITATIONS OF LIVE FIRE TESTING

High Cost

• The primary limitation of full-up, full-scale live fire testing is cost. On a per shot basis, it is considerably more expensive than inert or subscale testing, primarily due to the high cost and limited availability of targets. Testing and restoration costs are also higher, as are their associated time requirements. Nonetheless, live fire testing costs are a very small percentage of total program costs.

Limited Information

• Full-up testing potentially yields less information about damage mechanisms per shot than inert or subscale testing, primarily because catastrophic kills destroy the target and its components, along with much of the instrumentation used to record the damage. However, not all full-up shots result in catastrophic kills; such shots potentially yield more interpretable information than equivalent inert shots.

Limited Generalizability

• Full-up live fire test results typically are less easily generalized beyond the specific test conditions than inert or subscale testing. Full-up testing brings a larger number of variables into play that potentially affect outcomes, yet because full-up testing destroys targets, a smaller proportion of relevant test conditions can be examined.

Subscale Testing

• Subscale tests can support larger sample sizes than full-scale tests (whether full-up or inert), and are useful in bounding effects and providing input to models. Certain types of subscale testing are also useful for developing generic characterization of munitions effects.

• Subscale tests can provide only indirect evidence of synergistic effects on realistic targets, which must be inferred through an unproven analytical process (modeling). Therefore, subscale testing can supplement full-up, full-scale testing but not substitute for it.

Inert Testing

• Inert testing of full-scale targets is superior to full-up testing in characterizing mechanical damage to individual components and in conserving both components and targets.

• Catastrophic damage cannot be observed directly from shots on inert targets, and the standard method for inferring a K-kill underestimates its true likelihood. Like subscale tests, inert tests can provide only indirect evidence of effects on realistic (i.e., full-up) targets, inferred through models acknowledged to be weak on combustibles. Therefore, inert testing can supplement full-up, full-scale testing but not substitute for it.

Combat Data

• Analysis of combat data, if available, has several advantages over V/L testing: it provides greater realism, includes information above the level of vulnerability and lethality (e.g., aggregated survivability measures), and is considerably less expensive.

• Combat data provide less scientific control than testing, are limited to munitions and systems that have been employed in combat, and offer no direct view of the damage process or the conditions of firing. Like subscale and inert testing, combat data can supplement full-up, full-scale testing but not substitute for it.

TECHNICAL IMPROVEMENTS

We suggest that DoD

1. Improve the estimation of human effects. Begin by replacing noninstrumented plywood mannequins with the instrumented anthropomorphic type.

2. Improve the reliability and validity of quantitative V/ L estimates. For example, interrater agreement studies could determine the magnitude of the reliability problem, and provide insights into reducing it.

3. Expand efforts to improve statistical validity, and establish guidelines for the statistical interpretation of small-sample live fire test results.

4. Concentrate model improvements on currently weak areas vital to casualty estimation—fire and explosion and human effects.

5. Establish guidelines for how models can better support the design and interpretation of live fire tests.

6. Establish guidelines for how live fire test results can be used in the revision of models.

7. Allow outside analysis into the modeling and modeling revision process, and provide better documentation of the process for use by those analysts.

8. Accumulate comparisons of model predictions with live fire test results over multiple tests in order to assess improvements in models, and make results available to outside analysts; also redo predictions of earlier live fire shots after models have been revised in order to validate improvements.

9. Require that detailed test plans include shotlines, munitions, sample sizes, predictions, analysis plans, rationales for decisions, and other critical information to enable proper oversight. Keeping plans unclassified should not be a justification for omitting key information.

10. Develop, modify, or procure instrumentation to yield more information from catastrophic shots.

11. Improve methods for simulating in-flight conditions; specifically altitude, altitude history, maneuver load, and slosh.