National Academies Press: OpenBook

Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program (1993)

Chapter: 2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies

« Previous: 1 Review of Current Methodologies Used to Assess Aircraft Vulnerability and Identification of Applications of the Results
Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

2
Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies

Regardless of the specific application(s) of a vulnerability assessment of an aircraft, whether it is to aid in design or design validation, satisfy program requirements, support subsequent analytical assessments, predict test outcomes, satisfy the Live Fire Test law, or support acquisition decisions, the goal of the assessment is to obtain information on the vulnerability of that aircraft. This information can be described in terms of the attributes of the types, amounts, and applications of the information obtained; the accuracy of, or level of confidence in, the information; and the cost required to obtain the information. An effective methodology is one that provides a great deal of accurate information, of all types, with many applications, at very low cost, and with very few deficiencies. Both of the methodologies reviewed in Chapter 1 (analysis/modeling and live fire testing1) individually have certain advantages and disadvantages with respect to accomplishing the goal of obtaining information on the vulnerability of aircraft.

In general, the analysis/modeling methodology can provide considerable numerical information on the vulnerability of the total aircraft from all aspects for all threats at a reasonable cost, but the level of confidence placed in the information can vary from very low to high, depending upon the type of information obtained, the model used, the quality of input data, the analyst, and the evaluator.2 On the other hand, although the live fire testing methodology has the potential to obtain data that can be used to determine numerical information on the vulnerability of the aircraft to a particular weapon for many hits over the entire presented area of the aircraft from all aspects, in actual practice, testing provides information only on the aircraft’s vulnerability to hits in relatively few locations,3 and the expenditure of funds required to obtain this information is relatively large. However, the level of confidence in the test results usually is relatively high. This chapter examines the three information attributes of types (including amounts and applications), accuracy, and cost for both methodologies.

Analysis/Modeling

Type, Amount, and Applications of the Information. The analysis/modeling programs described in Chapter 1 for the

1

The reader is reminded of the difference between general live fire testing and the congressionally mandated Live Fire Testing.

2

The committee notes that there are strong differences in opinions concerning the level of confidence to be placed in the analytical results. Some consider them to be very accurate, whereas others believe that there are few, if any, analytical results that are accurate. However, there is consensus in the committee that the present analytical models are flawed in several ways and are weak in their ability to extrapolate beyond their existing data base.

3

These few locations are selected to maximize the amount of useful data obtained and to answer specific questions concerning the aircraft’s vulnerability.

Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

three types of weapons provide numerical values for the vulnerability of the individual critical components and the total aircraft from all aspects around the aircraft and for any selected ballistic projectile or guided missile. The numerical values can be in the form of vulnerable areas or probabilities of kill. This information can be used to aid in design and design validation, to satisfy program requirements, to support subsequent analytical assessments, to predict test outcomes, and to support acquisition decisions.


Accuracy of Analytical Models. There are three aspects of the accuracy of the analytical models that need to be examined: model verification (Are the internal workings of the model in good order—are the equations properly coded?), model validation (Does the model adequately represent the processes it portrays—do the equations adequately represent the actual physical situation?), and model accreditation (Is the model appropriate to the particular application—is the model capable of properly representing the aircraft and the weapon?).


Model Verification. In so far as model verification is concerned, there appears to be little reason to doubt the veracity of the logic and coding of the models described in Chapter 1. They have been widely employed by government laboratories in all three military Services and by a great many nongovernmental users. The committee accepts the fact that the internal workings of the models are in good order.


Model Validation. Credibility problems exist with model validation. Despite all the physical phenomena that the current aircraft vulnerability models attempt to depict (and many are depicted with high confidence), there are some phenomena that are known to exist, but have not been characterized and entered into the model structure, or are poorly modeled. Perhaps the most notable phenomenon not modeled is the random deviation or ricochet of the path of the penetrator or fragment from the assumed straight shotline as it passes through the aircraft components. Another phenomenon not modeled is the synergism that occurs when a portion of an aircraft is hit by a multitude of closely spaced fragments. In this multiple, closely spaced hit condition, the fragments can be far more damaging to the impacted structure than when they are not closely spaced. A third example of a phenomenon not explicitly modeled is spall. The backface spall generated by the impact of a projectile or fragment on a plate is not considered as additional fragments to be tracked through the aircraft by the shotline model.4

A poorly modeled phenomenon is the treatment of the remaining parts of an impacting fragment or penetrator after it penetrates a plate. The current model considers only the largest remaining piece in calculating subsequent effects. Another poorly modeled phenomenon is the increase in a component’s Pk/h that occurs when the component is hit by more than one projectile or fragment or is damaged by both blast and fragments. The assumption is nearly always made that the same Pk/h value for the first hit in the component is adequate for subsequent impacts on the component. The increase in a component’s vulnerability due to damage caused by prior hits is neglected. For example, a tail rotor drive shaft on a helicopter may not be killed when first hit by a tumbled 12.7-millimeter armor-piercing projectile with incendiaries (API), but a second hit in the vicinity of the first hit may cause the shaft to break due to the synergism in damage between projectiles. This increase in Pk/h due to previous hits is usually neglected.

Also, there are known physical phenomena that are modeled, but their numerical values are not well known because they have not been tested, or the test results are for conditions not satisfied in the combat incident. For example, the penetration equations that determine the velocity and mass decay of fragments and penetrators as they penetrate plates on the aircraft may not have the proper decay coefficients for the material of interest. Furthermore, these equations have been developed for specific geometric shapes of impactors, such as spheres and cubes, whereas fragments are usually irregular. Two of the most difficult kill modes to model in aircraft are fire and explosion, This is due to the randomness of fuel sloshing within a tank, fuel leakage into dry bays around the tanks, and fuel migration into distant portions of the aircraft.


Neglect of On-Board Ordnance. One of the most important findings of our study is that on-board ordnance is usually neglected as a contributor to vulnerability. Most of the simulation of internal ordnance aboard the aircraft has been treated by the aircraft vulnerability community as “clutter.” Clutter is inert material in a compartment that is considered only as a compartment filler in the calculation of overpressure in the compartment due to its volume. The overpressure from the explosive “clutter” when hit by projectiles and fragments can change the total pressure in the compartment dramatically. One of the basic premises of all development testing is that modeling must precede testing. Any aspect of hardware impact on vulnerability must first be modeled so that the testing can be used to verify the modeling rather than for the testing to be extensive enough to cover all statistical events. One of the basic requirements of the Live Fire Test program is to test full-scale vehicles with a full load of on-board ordnance. The continual neglect of one of

4

Spall that is generated within a component is implicitly accounted for in terms of its damage to the component since the empirically determined total damage is the sum of all of the damage mechanisms and processes within the component, including the internal spall.

Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

the basic vulnerability contributors will make it more difficult to convince anyone that computer modeling can be substituted for the full-up testing of combat-loaded aircraft.


Weapon Lethality Assessments Versus Aircraft Vulnerability Assessments. Some of the programs currently used to compute an aircraft’s vulnerability to the various types of weapons were developed by the Joint Technical Coordinating Group for Munitions Effectiveness. Because the munitions effectiveness community wanted conservative estimates for the prediction of a weapon’s lethality, its programs were developed so that our weapons were not to be credited with any kill capability that was not clearly justified; a sound policy from the weapon development point of view. However, as a result of this approach, the use of these programs to predict the vulnerability of a U.S. aircraft most likely leads to overly optimistic predictions. Potential vulnerabilities have been ignored unless clearly justified; an unsound policy where the survivability of U.S. aircrews is concerned. Examples of where the vulnerability of aircraft is underestimated are

  • the use of the Thor penetration equations, which consider only the largest remaining piece of a penetrator as it passes through the aircraft,

  • the lack of synergism due to both multiple simultaneous and sequential hits on a component;

  • the lack of direct consideration of spall, and

  • the lack of consideration of cascading damage, such as fuel tank damage leading to fuel ingestion.

The Joint Technical Coordinating Group on Aircraft Survivability (JTCG/AS) Pk/hWorkshop. Even if all other aspects of the model were perfect, some of those portions of the model that portray the results of the physical interactions or damage processes that occur when a threat weapon interacts with an aircraft are inaccurate or incomplete. In order to eliminate some of the model deficiencies, more vulnerability data are needed on component kill modes and Pk/h functions. This deficiency in the component vulnerability data base was the subject of a JTCG/AS Component Pk/h Workshop held from March 5–8, 1991, at Wright-Patterson Air Force Base (WPAFB).

The objectives of the JTCG/AS Workshop were to critically review the current state-of-the-art of component Pk/h prediction, to recommend a set of Pk/h or Pd/h values or functions for use in analyses, and to develop plans for improving and validating this set.5 Working panels were organized as follows:

Fuel System

Flight Controls/Hydraulics (Control and Power)

Crew Station

Engines and Accessories

Stores, Ammunition, and Flares

Electrical and Avionics

Structures, Landing Gear, and Armor

Helicopter Unique Components

The draft reports of the individual panels are currently being integrated into a final report. The findings of the eight working panels are given in Appendix D.6

The general conclusion from the workshop is that the component vulnerability data base is the weakest link in the vulnerability analysis chain. Clearly, much remains to be done with respect to validating those sections of the aircraft vulnerability models that deal with component and subsystem damage prediction. The level of confidence placed in the analytical models would be significantly increased if a concerted effort was made to determine the maximum error in Av that occurs in the model predictions using the current data base, and to what extent this error might be reduced with the availability of new test data.


Model Accreditation. Model accreditation is accomplished by the JTCG/AS. This organization, consisting of representatives from all three Services, has established procedures for verifying, validating, and accrediting models. Despite the deficiencies identified by the aircraft vulnerability assessment community in its analytical models, the community has been exemplary in the exchange of data and ideas among the three Services and industry, in the development of handbooks and design guides for reducing vulnerability, and in the establishment of validated data bases from test data. Although there is still much to learn, more perhaps is known. There will always be a need to update models with new input data as aircraft materials and designs change and as the threat weapons change.


The Cost of Analysis. It is extremely difficult to determine the costs associated with a typical analytical assessment since the cost depends heavily on the type of aircraft being assessed and the particular application for the assessment. However, some rough estimates for the costs to conduct an analysis that would be appropriate for a major milestone decision are given below. The numbers were obtained in a personal communication from a representative from the Ballistic Research Laboratory.

5

Component vulnerability is sometimes quantified by Pd/h, the probability the component is damaged given a hit.

6

These findings are taken from a preliminary copy of a briefing being prepared to be given to the JTCG/AS Central Office and to the Office of the Secretary of Defense. This briefing material was provided by Gerald Bennett, ASD/XRM, WPAFB.

Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

Cost of the Model. The cost of creating a COVART or HEVART model for a helicopter is roughly between $75,000 and $200,000, with the low end for an upgrade to an existing system and the high end for an all new, high detail description of a U.S. system. These costs also include the generation of the shotline data files needed to run the Computation of Vulnerable Area and Repair Time Model (COVART) or High Explosive Vulnerable Area and Repair Time (HEVART) model. The cost of developing appropriate target models for the Stochastic Qualitative Analysis of System Hierarchies (SQuASH) might be two to three times that required for the conventional vulnerable area models.7 As the design community increases its use of three-dimensional modeling and grid generation, and as computer usage becomes less expensive, the cost of modeling will come down significantly. The major cost will be the preparation of input data.


Cost of Computer Runs. A typical batch of computer runs for a helicopter includes two flight modes (hover and forward flight) and from 6 to 26 attack aspects. The cost of this batch of runs using COVART or HEVART ranges from $45,000 to $100,000. The analysis and preparation of input data account for most of this cost.


Cost of Obtaining Supporting Experimental Data for Pk/hFunctions. The execution of an analytical model without any live fire test data on kill modes and Pk/h functions as input data on component vulnerability could be accomplished, but the level of confidence in the results would be very low. Consequently, the cost of obtaining the necessary data on component vulnerability must be included in the cost of the analysis. If all of these data are available from prior tests, the cost of gathering them is relatively low. If all of the necessary data are not available, supporting tests must be carried out to obtain the missing data. These tests can be as simple as the firing of fragments at pieces of plate to measure penetration or V50 velocities, or they can be as complicated as firing several fragments at running engines to determine the severity of damage, or firing at major portions of the aircraft structure that contains fuel tanks to simulate the fire/explosion and hydraulic ram phenomena. The typical costs associated with two types of live fire tests are given below.

For a helicopter engine and a 23-millimeter HEI threat, basic preliminary testing with components and/or a static engine would cost approximately $30,000. Comprehensive testing with a full-up running engine would cost approximately $250,000. For a tail boom structure, basic shoot-and-look damage characterization tests would cost approximately $25,000. A comprehensive evaluation of the test results, including postdamage controlled structural experiments would cost approximately $150,000.

Live Fire Testing

A live fire test is one in which live ammunition, either explosive or non-explosive, is fired at a target. Live fire testing can be used to assist in the design and design validation of an aircraft, to provide empirical information on component vulnerability in support of the analytical models, to satisfy the Live Fire Test law, and to support acquisition decisions. Live fire testing is part of either developmental testing (DT) or Live Fire Testing. The difference between the two types of tests is that DT is part of the normal design and development process, whereas Live Fire Testing is that testing intended to satisfy the Live Fire Test law. “As currently defined, development test and evaluation (DT&E) is that test and evaluation (T&E) conducted throughout the acquisition process to assist in the engineering design and development process and to verify attainment of technical performance specifications and objectives and supportability. DT&E includes T&E of components, computer software, subsystems, and hardware/software integration. It encompasses the use of modeling, simulations, and test beds, as well as advance development, prototype, and full-scale engineering development models of the system. Technical performance specifications must be validated through DT&E in order for the developer (program manager) to certify that the weapon system is ready for the final phase of Initial Operational Test and Evaluation (IOT&E)” (OSD, 1987).

Of particular interest here is the specific test and evaluation methodology required to satisfy the Live Fire Test (LFT) legislation. The law requires realistic survivability (vulnerability) testing. Although such a test program will in fact involve early component and subsystem or sub-scale Live Fire Testing, the hallmark of this approach is a substantial test program that involves a significant number of shots against a combat-configured, full-scale version of the weapon system (i.e., a full-scale LFT). The early Live Fire Tests on components and subsystems provide information on any vulnerabilities of the individual components and subsystems. Once the information from these tests has been evaluated, the tests on the full-scale aircraft are to be conducted as mandated by the law, unless a waiver has been granted. The full-scale test program will often include a number of shots that are randomly chosen and a number of shots that are selected to address specific issues. Real (or realistic surrogates of) threats likely to be encountered in combat must be used in the tests. These threats can be non-explosive ballistic projectiles, ballistic projectiles with contact-fuzed and proximity-fuzed high-explosive (HE) warheads, and guided missiles with contact-fuzed and proximity-fuzed HE warheads. The three information attributes of

7

Note that SQuASH has not yet been applied to aircraft vulnerability assessment.

Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

types (including amounts and applications), accuracy, and cost for both sub-scale testing and full-scale testing are examined below.


Type, Amount, and Applications of the Information. In general, the Live Fire Tests on both sub-scale and full-scale targets produce information on what actually happened for a particular set of test conditions (e.g., the specific target, weapon, and shotline) and typically for a relatively small number of shots under these conditions.8 Some typical examples of tests on sub-scale targets are tests to determine the penetration capability of fragments through plates of composite materials, tests on helicopter rotor blades, tail booms, and gear boxes using small arms projectiles and small-caliber HE AAA rounds, tests on fuel tank simulators using small caliber HE AAA to determine the efficacy of a particular fuel tank protection scheme, and stand-alone tests on running engines using several impacting fragments. The information from these tests ranges from measured fragment velocities, temperatures, and overpressures to the ability of the tested article to continue to function after the hit. Tests on the full-scale aircraft will most likely be conducted with the aircraft on the ground or suspended; it can neither crash nor be forced to land as a result of the shot. Thus, the ability of the aircraft to sustain the essential functions for flight after the hit is not observed directly. Furthermore, rather than produce a sufficient amount of numerical data from the full-scale test that can be used directly to determine the kill probability of the aircraft for the shot, each full-scale test provides a list of damaged components along with descriptions of the details of the extent and severity of the damage and the associated damage events for each shot.

With respect to the applications of the information obtained from the tests, the committee notes that the Live Fire Tests are conducted primarily to (1) satisfy the LFT law and its intent (i.e., to determine any inherent vulnerabilities in the design sufficiently early in the program to allow the vulnerabilities to be corrected), and (2) provide information in support of acquisition decisions. They are not specifically intended to provide information that can be used to improve the analytical models for predicting aircraft vulnerability. Nevertheless, previous experience with full-scale Live Fire Tests on ground vehicles has shown that important types of damage and kill modes have been observed that were not included in the analytical models for these vehicles. Thus, the information provided by these LFTs can be used in the other applications. Besides satisfying the letter of the law, the LFT&E program provides (in principle) an opportunity to

  • obtain information on the vulnerability of the aircraft;

  • find vulnerabilities that were not anticipated by the analyses; and

  • gather data on the synergism among the different damage processes and kill modes.

All three aid in the design and design validation, support the analytical models, and support acquisition decisions.


Accuracy of and Level of Confidence in the Information. The level of confidence that is placed in the subscale and full-scale Live Fire Test results depends primarily upon the level of realism of the tests. Certainly, there is a possibility for realism in Live Fire Testing. Some of the vulnerability aspects that are represented in Live Fire Testing are the effects of blast, including structural deformation and component damage, and component damage due to multiple fragment hits, including bending, breaking, and perforating. Other aspects of vulnerability that are represented include the occurrence of spall and the penetration through components. The penetration damage and velocity decay in the test are the result of real penetrators going through real materials, with no assumptions about breakup, ricochet, etc.9 Synergisms among blast, fire, spall, and fragments are realistically represented to the extent that the other aspects of the test conditions (e.g., air speed and altitude) are adequately represented.

Although full-scale tests have the potential to provide the most realism, there are some problems, particularly with the weapons used, the flight conditions, crew vulnerability, and on-board munitions. The weapons selected for testing will most likely be those that are not overmatching (i.e., they will not have a high probability of destroying the aircraft). Consequently, most of the weapons will be either non-explosive or small-caliber explosive rounds. When larger explosive weapons are used, particularly the large-caliber projectiles and guided missiles, the damage to the aircraft can be severe and widespread, making it very difficult to repair the aircraft and return it to a condition that would be satisfactory for further testing. Associated with the explosive weapon is the location of the point of detonation. Contact-fuzed HE warheads must impact the aircraft in order to damage it, but proximity-fuzed weapons can detonate at distances ranging from the aircraft skin to several hundred feet away. If the warhead is detonated too close to the aircraft, it could destroy it.

With respect to the flight conditions, airflow can be simulated to some extent, although it seems unlikely that an entire

8

Typically, the smaller the target, the more greater is the number of test shots, and the more detailed is the information.

9

Although penetration is realistically represented in the test, the information obtained from the test most likely will not provide penetration data per se for the analytical models because there will be few if any data on the parameters associated with the individual penetrations, such as impact and exit velocities, masses, and angles.

Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

transport aircraft would be placed in a uniform high-speed airstream. Static loading conditions can be simulated, at least over a part of the aircraft, but there is some concern about the effects of wind gusts and about transient loads produced by maneuvering and loss of control. Furthermore, altitude and temperature are not simulated. For externally detonating warheads, a static test may result in a different set of blast and fragment impact conditions from those of a dynamic test. In some warhead/target encounters (e.g., in head-on pass or when overtaking at very high velocity), the high-velocity fragments may impact on one part of the target aircraft’s structure while the slower blast may affect a different area. In static tests, both affect the same portion of the target.

Clearly, real people cannot be used in the tests. Anthropomorphic dummies, pressure gauges, and gas-sampling equipment can be used to obtain data on vulnerability issues related to personnel vulnerability. On the other hand, the responses of the crew to shock, incapacitation, temporary loss of control, etc., are not directly measured and must be inferred. These responses are critical to both crew and aircraft survivability.10


Vulnerability of the Aircraft. Although it is true that certain catastrophic kills, such as an explosion within an aircraft fuel tank, would be observable in a test, it is also true that other types of kills would not be directly observable. For example, would the aircraft actually crash after damage to one of the control surfaces? Even if the actual kill was directly observed, there are the problems of associating the results with the other kill categories and levels, and of extrapolating the results to other threats and tactical conditions. For example, suppose the proximity-fuzed detonation of an 85-millimeter HE warhead near the left differential stabilator of the aircraft removed 90% of the stabilator. Is this a kill, and if it is for the 85-millimeter weapon, would a 57-millimeter warhead detonation in the same location cause the same kill?


Unanticipated Vulnerabilities. The analytic models are presently structured to provide aircraft kill probabilities based upon assumed kill modes, and an aircraft with reduced vulnerability is designed to prevent these kill modes from occurring. However, if a kill mode is unanticipated in both the analysis and the design, the model will underestimate the aircraft’s actual vulnerability, and the aircraft will contain this vulnerability. These unanticipated kill modes may be local effects that occur in the vicinity of the original impact, or they may be caused by cascading damage from the impact location to a distant part of the aircraft. For a local effect example, suppose the unanticipated kill mode was an electrical wire bundle that caught fire when hit by a bullet or fragment. This is a local kill mode that could be discovered by component or subsystem testing. On the other hand, suppose the unanticipated kill mode was a fuel ingestion kill of an engine mounted externally on the rear portion of the fuselage. Hydraulic ram pressure in the fuel in the wing fuel tank in front of the engine (due to a penetrator) caused fuel to spew out the top of the tank. This fuel was ingested by the engine, which then died. A test of only the wing fuel tank might not reveal this kill mode because of the absence of the running engine.

Although a full-scale Live Fire Test might reveal a local unanticipated kill mode, such as the burning wire bundle, full-scale testing is not necessarily an efficient methodology for obtaining this information. Furthermore, one can not say with great confidence that if no unexpected vulnerabilities occurred in the full-scale Live Fire Tests, then there are none to be discovered later in combat. Kill modes involving cascading damage may not always occur in a test, and they may be particularly difficult to observe in a full-scale test if they do occur.11 Some kill modes due to cascading effects are well known, such as the kill of an engine due to the ingestion of fuel from a damaged fuel tank next to an air inlet, and are relatively easy to observe. Others, such as the migration of toxic fumes or flames from one portion of the aircraft to another, are not as well known, may not always occur, and may be difficult to detect if they do occur.

It is not feasible to test all possible combat situations for unanticipated vulnerabilities using full-scale aircraft because of the large number of parameters that affect the target’s vulnerability. These parameters include all of the weapons likely to be encountered in combat, the tactical situations of interest, and all of the possible impact locations on the aircraft (i.e., the shotlines). Furthermore, the damaged aircraft should be returned to its original condition after each test if appropriate and possible. Consequently, the test plan may contain a number of random shots, a number of random shots from directions expected in combat, a number of selected shots, and an ordering that schedules potentially catastrophic situations at the end of the program.12


Likelihood of Discovering a Particular Vulnerability. Unless a particular vulnerability is relatively insensitive to the parameters associated with a large subset of the conditions and has a relatively high probability of occurring, it may remain undiscovered in a test program. For example, suppose that a particular vulnerability event associated with a

10

The committee notes that these departures from realism in the live fire tests are more severe in analysis/modeling.

11

This is the reason that Live Fire Test programs require some random hits and place emphasis on incidents that occur relatively frequently rather than on all unanticipated kill modes.

12

The number of shotlines in the vulnerability analysis of the F-22 was approximately 300,000 per threat.

Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

FIGURE 2-1 Relationship among the probability of occurrence on each test, the number of tests, and the probability of at least one observation over the number of tests.

high-explosive weapon had a 0.2 probability of occurrence on any random shot for all test conditions of interest. If the test plan consisted of 10 random shots around the aircraft, the probability the event would occur at least once in the 10 tests is 1−(1−0.2)10=0.89 and hence the probability it would not occur is 0.11. Thus, this vulnerability would most likely be observed.13 On the other hand, if the event could only occur on 2 of the 10 random shots with a 0.2 probability (and a probability of zero on the other 8 shots), the probability the event would occur at least once is

and the probability that it would not occur is 0.64. Thus, this particular vulnerability would most likely not be observed.

Figure 2-1 shows the relationship among the probability of occurrence; the probability of observation of 0.25, 0.50, 0.75, or 0.95; and the number of tests. This figure can be used to determine the number of tests required to obtain a probability of observation of 0.25, 0.50, 0.75, or 0.95 for a given probability of occurrence. For example, if the probability of occurrence in each test is 0.2 and the desired probability of observance at least once is 0.75 or higher, at least seven tests must be conducted.

Suppose there were 5 independent vulnerability events possible on each of the 10 random shots, and each event had a 0.2 probability of occurring on each of the 10 shots. The probability that any one of the events would occur at least once during the test program is 0.89. The probability that all of the events would occur at least once (not necessarily on the same shot) during the program is

Hence, there is a 0.44 probability that one or more of the five kill modes will not be observed. Thus, the test program is essentially equally likely either to reveal all or to miss one or more of the aircraft’s vulnerabilities. If each vulnerability

13

Other known vulnerabilities may also occur during the tests that could obscure the particular vulnerability, and other unknown vulnerabilities could also occur.

Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

could occur on only 2 of the 10 test shots, the probability that one or more vulnerabilities would not occur in the 10 tests is essentially one.14


Synergism Among the Damage Processes and Kill Modes. Live Fire Tests are also a valuable source of vulnerability information on the synergism among the damage processes and kill modes, albeit a limited source. Suppose there are two kill modes I and II that exhibit synergism. Mode I has a probability of occurrence of 0.1, regardless of the occurrence of mode II. Mode II has a probability of occurrence of 0.1 if the first mode does not occur and a 0.5 probability of occurrence if the first mode does occur. On one shot, the probability that both modes occur is 0.05. The probability that only mode I occurs is 0.05, the probability that only mode II occurs is 0.09, and the probability that neither mode occurs is 0.81. If these modes were independent, and both had a probability of occurrence of 0.1, mode I only occurs with a probability of 0.09, mode II only occurs with a probability of 0.09, modes I and II together occur with a probability of 0.01, and neither mode occurs with a probability of 0.81. Thus, the probability that neither mode occurs is the same in both the synergistic case and the independent case. In the synergistic case, mode II is more likely to occur (0.14 vs. 0.10) and the probability that both modes occur together is higher than in the independent case (0.05 vs. 0.01).


Validation of the Analytical Model. The analytical models are presently structured to provide kill probabilities and vulnerabilities based upon a selected kill category. The Live Fire Tests do not directly provide the numerical data required to validate the model’s predictions; they only provide information on the components that were damaged or killed, the occurrence or non-occurrence of the kill modes, and any cascading damage. Thus, some compromises must be made that require additional analyses in order to relate the empirical test results to actual combat conditions and to determine if the test results correspond to an actual kill of the aircraft. If enough test shots could be made under the identical set of conditions, a statistical inference could be made regarding the probability of an aircraft kill given the test conditions. Unfortunately, the number of identical shots that can be conducted is usually small, and hence the confidence level in the sample mean probability of kill is low. Furthermore, because the models are expected value models, and given the randomness of the empirical results from the limited number of tests, the likelihood that the test results and the model predictions are in general agreement with respect to the components affected and their vulnerability is low. For example, suppose the model predicts a wing fuel tank will explode when hit with a probability of 0.5. If each of three identical tests result in an explosion, does this mean the model is inaccurate? The answer is no.15 Thus, the Live Fire Tests can not be used in themselves to directly validate the models, and they are not conducted for that purpose. However, they do provide valuable information on damage processes that should be modeled.


Neglect of On-Board Munitions.. The Live Fire Test law requires the aircraft to be full-up when tested. This means that any munitions normally carried by the aircraft must be on-board at the time of the test. This will probably be done in only a very few tests; it is not the intent of the Live Fire Test law to destroy aircraft needlessly. The intent of the law is to obtain information on the vulnerability of the aircraft sufficiently early to allow any design deficiency to be corrected. Consequently, it does not seem reasonable to intentionally create a situation in which the aircraft could be destroyed when essentially the same information on the vulnerability of the aircraft to the on-board munitions can be obtained by using realistic off-line tests of sub-scale models of the portion of the aircraft in the vicinity of the munitions. This is particularly true for aircraft with internally stored munitions. Consequently, inert surrogates most likely will be used for munitions on-board the full-scale aircraft in order to prevent a catastrophic kill. The increase (or decrease16) in aircraft vulnerability due to the presence of the munitions can be determined by relating the observed projectile and fragment impacts on these surrogates to the vulnerability data on munition reactions to impacts obtained from the offline sub-scale tests.17 It is not necessary to load the munitions on the full-scale aircraft in order to determine the likelihood of an adverse munition reaction to a hit or a fire. If a design deficiency with respect to the on-board munitions is discovered and a less vulnerable design can be developed, it can be incorporated without the loss of the test aircraft.


The Cost of Live Fire Testing. No complete Live-Fire Testing program for any aircraft has been carried out as yet. Consequently, the following information is offered only as an indication of the costs involved in full-scale Live Fire Testing. If no waiver has been given from full-scale testing,

14

These examples are given to put LFT in the same context with the analysis/modeling methodology. It is not the purpose of LFT to find all vulnerabilities.

15

Nevertheless, a prudent engineer would thoroughly examine the results of the experiment and the analysis in an attempt to determine whether the sequence of three explosions was not indicative of a higher probability of occurrence.

16

Some munitions may provide a shielding effect, such as the bombs carried below a wing. Projectiles and fragments that do not penetrate a nonreacting bomb are prevented from impacting and damaging the wing.

17

The off-line tests must provide sufficient information to enable a valid evaluation of the reactions of the munitions and the effects of those reactions on the aircraft, including lower-order effects that are not catastrophic.

Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

TABLE 2-1 Live Fire Test Options for the C-17A (IDA, 1989)

Option

Test Item

Test Item Cost ($ million)

Testing Cost1 ($ million)

Total Cost ($ million)

1

Full-up aircraft

154

4

158

2

Fuselage & one wing

107

4

111

3

Fuselage section & one wing

60

4

64

4

One wing

36

4

40

5

One wing leading edge

?

?

152

1This does not include the cost of repairing the damage for the next shot. Also, some members of the committee believe that $4 million is too low.

2An Air Force estimate.

at least one production or preproduction aircraft is required.18 If the test is carefully designed, and if the target is reparable up to the last shot, one aircraft should be sufficient. The cost of the test aircraft is most likely the major cost of the total program. A question arises as to the proper cost of the aircraft to use when determining the program cost. Should the actual construction cost of the test aircraft, which might be one of the first five or six aircraft built, be used? Should the cost also include the research and development costs, or should the average flyaway cost be used? The particular cost used will have a major impact on the perceived benefit of the Live Fire Test program. If one aircraft is tested out of a total aircraft buy of 400, and the average aircraft cost over the buy is used, the cost of the Live Fire Tests will be less than 0.3% of the total program cost.


Costs for the C-17A. The Institute for Defense Analyses (IDA) has presented a number of test program options, issues, and costs for Live Fire Testing the C-17A in a draft report (IDA, 1989). The report makes no recommendations on which (if any) of the options should be selected. The costs of five options are given in Table 2-1.

At least one production aircraft is required for option 1. If the test is carefully designed, and if the target is reparable up to the last shot, one aircraft should be sufficient. The aircraft cost of $154 million is based on a FY1990 recurring flyaway cost of $181 million with engineering, tooling, and avionics costs removed.19 On the other hand, the FY1997 and 1998 flyaway costs are $78 million. Thus, the cost of the test article depends upon the method of bookkeeping used. The most elaborate test program suggested by IDA is estimated to cost no more than $4 million and involves 100 small arms projectiles, 50 small AAA HE rounds, and 20 man-portable infrared missile shots.


Costs for the RAH-66A COMANCHE Helicopter. The cost of Live Fire Testing the COMANCHE helicopter has been estimated by a representative from the Ballistic Research Laboratory and coordinated with the Program Manager (PM). Based on projected production system fly-away costs for COMANCHE, a full-up low-rate initial production (LRIP) target that is representative of an operational, combat-configured system will cost in excess of $7.5 million. Given the alternative (and planned) use of an engineering test prototype aircraft built up to meet specific Live Fire Test requirements (a minimal configuration), the cost would be less. However, it still is a major percentage of the LFT&E program expense. At least two sets of target components/ subsystems, plus repair provisions, would also be needed. Steps can be taken to attempt to minimize the risks to the hardware, but these may not always work. The operational reutilization of the test articles is unlikely.20

The number of shots that can be conducted on one target is highly variable. However, given ideal repair capability and no catastrophes, 25 or more small-caliber API shots

18

The use of static test articles or prototypes for the LFTs should be considered.

19

In the committee’s evaluation of the cost of full-up, full-scale aircraft estimates are based on flyaway cost without avionics. This would no longer be a “full-up” configuration. In fact, recent testing on aircraft shows avionics provide excellent shielding. In some aircraft, avionics components are themselves vulnerable.

20

It should be noted that all of the test vehicles used in the U.S. Army’s ABRAMS Live Fire Tests were repaired and returned to service.

Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

TABLE 2-2A Relative Advantages and Disadvantages of the Two Methodologies: Type, Amount, and Applications of the Information

Analysis/Modeling

Live Fire Testing

Sub-scale Testing

Full-scale Testing

Advantages

Advantages

Advantages

  • The information is available in the form of vulnerable areas and probabilities of kill given a hit.

  • The models can provide much information on the vulnerability of the aircraft over a wide range of conditions in which the systems will be used, over a wide range of potential threat weapons, and over a wide range of system design alternatives.

  • The information can be used to aid in the early design of the aircraft, to satisfy program requirements, to support subsequent campaign/war game assessments, to predict test outcomes, and to support acquisition decisions.

  • The type of information from sub-scale testing is in the form of semi-realistic physical outcomes, such as penetration or no penetration, fire or no fire, etc.

  • The information from sub-scale tests can be used to aid the design (when the tests are conducted sufficiently early in the program), to validate the design, to provide data for component Pk/h data bases and new damage processes in support of subsequent assessments, and to support acquisition decisions.

  • The type of information from full-scale testing is in the form of realistic physical outcomes, including cascading and synergistic damage, such as which components are damaged, if a fire occurred how far it spread, etc.

  • The information from full-scale tests can be used to aid the design (when the tests are conducted sufficiently early in the program), to provide information on new damage processes in support of subsequent assessments, to satisfy the requirements of the Live Fire Test Law, and to support acquisition decisions.

  • The Live Fire Tests that were performed on full-scale armored vehicles provided data on the types of damage that resulted in delayed catastrophic damage. These data were useful in training the crew in emergency exit procedures. Similar data on ejection procedures might come from Live-Fire Tests of aircraft.

  • Damaged aircraft offer an opportunity for battle damage crews to practice repairing realistic combat damage.

Disadvantages

Disadvantages

Disadvantages

  • Model results reflect the assumptions on vulnerability and therefore do not provide any new information on vulnerabilities resulting from new materials and damage processes.

  • The amount of information is relatively small, is confined to the tested components and subsystems, and is limited to the specific test conditions.

  • The information from sub-scale tests may not be directly relatable to an aircraft kill.

  • The amount of information is relatively small, is confined to the tested regions, and is limited to the specific test conditions. An unexpected catastrophic reaction in one of the early tests can prevent the gathering of additional information.

  • The information from full-scale tests may not be directly related to an aircraft kill.

  • Because of production-like aircraft must be tested in the full-scale test, it may not be practical to correct design deficiencies discovered in the tests.

  • Full-scale tests will not be used to produce the necessary broad and detailed data base for supporting the models because of the lack of control over the test outcomes and the costs involved.

Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

TABLE 2-2B Relative Advantages and Disadvantages of the Two Methodologies: Accuracy of, and Level of Confidence in, the Information

Analysis/Modeling

Live Fire Testing

Sub-scale Testing

Full-scale Testing

Advantages

Advantages

Advantages

  • Models can provide estimates of the major vulnerabilities, particularly when they are supported by sub-scale live fire tests.

  • Sub-scale tests can provide vulnerability data on new materials, new components, new construction processes, new threats, etc.

  • Short of actual combat, full-scale live fire testing is as close to real data on a combat-configured aircraft as is possible to achieve. It accounts for the weapons effects, synergism among damage processes, and cascading damage.

Disadvantages

Disadvantages

Disadvantages

  • The models will never be able to totally represent all of the possible vulnerability events that can occur in combat, such as synergism and cascading damage. Some damage processes are modeled poorly, and some are not modeled at all. Thus, the estimates of vulnerability do not have a high level of confidence, particularly for new aircraft.

  • Current models tend to underestimate vulnerability because of their conservative assumptions regarding weapon lethality (e.g., they neglect spall, fuel ingestion, and multiple hit effects).

  • There are serious data gaps on componant and subsystem vulnerabilities, such as those identified by the 1991 JTCG/AS Pk/h Workshop.

  • Sub-scale tests can vary from simple component tests to realistic tests on complete subsystems. However, because these tests cannot simulate any interaction between the tested article and the portions of the aircraft not included, they are not fully representative of the actual combat situation.

  • Those weapons likely to be encountered in combat that can destroy the aircraft are not used in such a manner. They are accepted as overmatching threats.

  • Altitude effects and the conditions of flying are not included in full-scale vulnerability testing.

  • Because the air crew is not on board during an LFT, the possible effects of the test on the crew are difficult to determine.

plus four or more small-caliber HEI projectile shots might be possible. The cost per shot is highly variable, ranging from $500 to $10,00021, and is a function of the weapon used, the target configuration, the scope of the test issues, the extent of the instrumentation and data capture, the workup costs, etc.

Advantages and Disadvantages of Analysis/Modeling and Live Fire Testing

Both analysis/modeling and Live Fire Testing have advantages and disadvantages or deficiencies relative to one another. Furthermore, there are relative advantages and disadvantages of both sub-scale testing and full-scale testing. Table 2-2 presents a comparison of the methodologies for each of the three information attributes of type, amount, and applications; accuracy or level of confidence; and cost.

Conclusion

The committee concludes that the combination of analytical models, supported by live fire tests on components and subsystems, and the sub-scale and full-scale Live Fire Tests are mutually compatible in the vulnerability analysis, evaluation, and design of aircraft. They complement each other, and the whole is superior to the sum of the parts. More work is needed to unify these approaches in order to obtain the maximum benefit. The aircraft vulnerability

21

Some committee members believe that the cost of the tests would range from $5,000 to $50,000.

Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

TABLE 2-2C Relative Advantages and Disadvantages of the Two Methodologies: Cost

Analysis/Modeling

Live Fire Testing

Sub-scale Testing

Full-scale Testing

Advantages

Advantages

Advantages

  • The cost of analysis is significantly less than the cost of Live Fire Testing.

  • Sub-scale tests are considerably less expensive than full-scale tests.

  • Live Fire Tests are a small percentage of the total program costs. If only one aircraft is tested out of a total aircraft buy of 400, the cost of the Live Fire Test program is less than 0.3% of the total program cost.

  • The use of static test articles or prototypes can significantly reduce the cost of the tests.

Disadvantages

Disadvantages

Disadvantages

  • Although the actual cost of conducting an analysis is relatively low, there are considerable sub-scale test costs required to build up the essential vulnerability data base.

  • Although sub-scale tests are less costly than full-scale tests, a comprehensive sub-scale test program in support of a particular aircraft acquisition program is not inexpensive, and the program manager may not be inclined to spend the required funds.

  • The cost of sub-scale testing may preclude testing for basic phenomenological data on damage processes and kill modes.

  • The cost of full-scale testing is primarily influenced by the cost of the test aircraft. Thus, the cost of testing a full-scale aircraft is considerably more than the cost of analyses or sub-scale testing when the aircraft was procured only for that purpose and cannot be repaired and put into service.

community, based on its plans for the Live Fire Tests, seems to appreciate the need to integrate these approaches, having witnessed the success in using the data from the Live Fire Tests on the Abrams tank and the Bradley Fighting Vehicle to improve both the analytical methodology and the vehicle designs.

References

• Institute for Defense Analyses (IDA), 1989. C-17A Live Fire Test Options Report, Paper P-2228.

• Office of the Secretary of Defense, 1987. Report of the Secretary of Defense on Test and Evaluation in the Department of Defense.

Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 26
Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 27
Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 28
Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 29
Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 30
Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 31
Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 32
Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 33
Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 34
Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 35
Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 36
Suggested Citation:"2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 37
Next: 3 The OSD Live Fire Test and Evaluation Program »
Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!