2
Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies

Regardless of the specific application(s) of a vulnerability assessment of an aircraft, whether it is to aid in design or design validation, satisfy program requirements, support subsequent analytical assessments, predict test outcomes, satisfy the Live Fire Test law, or support acquisition decisions, the goal of the assessment is to obtain information on the vulnerability of that aircraft. This information can be described in terms of the attributes of the types, amounts, and applications of the information obtained; the accuracy of, or level of confidence in, the information; and the cost required to obtain the information. An effective methodology is one that provides a great deal of accurate information, of all types, with many applications, at very low cost, and with very few deficiencies. Both of the methodologies reviewed in Chapter 1 (analysis/modeling and live fire testing1) individually have certain advantages and disadvantages with respect to accomplishing the goal of obtaining information on the vulnerability of aircraft.

In general, the analysis/modeling methodology can provide considerable numerical information on the vulnerability of the total aircraft from all aspects for all threats at a reasonable cost, but the level of confidence placed in the information can vary from very low to high, depending upon the type of information obtained, the model used, the quality of input data, the analyst, and the evaluator.2 On the other hand, although the live fire testing methodology has the potential to obtain data that can be used to determine numerical information on the vulnerability of the aircraft to a particular weapon for many hits over the entire presented area of the aircraft from all aspects, in actual practice, testing provides information only on the aircraft’s vulnerability to hits in relatively few locations,3 and the expenditure of funds required to obtain this information is relatively large. However, the level of confidence in the test results usually is relatively high. This chapter examines the three information attributes of types (including amounts and applications), accuracy, and cost for both methodologies.

Analysis/Modeling

Type, Amount, and Applications of the Information. The analysis/modeling programs described in Chapter 1 for the

1

The reader is reminded of the difference between general live fire testing and the congressionally mandated Live Fire Testing.

2

The committee notes that there are strong differences in opinions concerning the level of confidence to be placed in the analytical results. Some consider them to be very accurate, whereas others believe that there are few, if any, analytical results that are accurate. However, there is consensus in the committee that the present analytical models are flawed in several ways and are weak in their ability to extrapolate beyond their existing data base.

3

These few locations are selected to maximize the amount of useful data obtained and to answer specific questions concerning the aircraft’s vulnerability.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 26
typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original Evaluation of the Cost, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution. Effectiveness, and Deficiencies of These Methodologies 2 Regardless of the specific application(s) of a vulnerability numerical information on the vulnerability of the aircraft to a assessment of an aircraft, whether it is to aid in design or particular weapon for many hits over the entire presented area design validation, satisfy program requirements, support of the aircraft from all aspects, in actual practice, testing subsequent analytical assessments, predict test outcomes, provides information only on the aircraft’s vulnerability to satisfy the Live Fire Test law, or support acquisition decisions, hits in relatively few locations,3 and the expenditure of funds the goal of the assessment is to obtain information on the required to obtain this information is relatively large. vulnerability of that aircraft. This information can be However, the level of confidence in the test results usually is described in terms of the attributes of the types, amounts, and relatively high. This chapter examines the three information applications of the information obtained; the accuracy of, or attributes of types (including amounts and applications), level of confidence in, the information; and the cost required accuracy, and cost for both methodologies. to obtain the information. An effective methodology is one that provides a great deal of accurate information, of all types, with many applications, at very low cost, and with very few Analysis/Modeling deficiencies. Both of the methodologies reviewed in Chapter 1 (analysis/modeling and live fire testing1) individually have Type, Amount, and Applications of the Information. The certain advantages and disadvantages with respect to analysis/modeling programs described in Chapter 1 for the accomplishing the goal of obtaining information on the vulnerability of aircraft. In general, the analysis/modeling methodology can provide considerable numerical information on the vulnerability of the total aircraft from all aspects for all threats at a reasonable cost, but the level of confidence placed in the information can vary from very low to high, depending upon 2 The committee notes that there are strong differences in opinions the type of information obtained, the model used, the quality concerning the level of confidence to be placed in the analytical results. of input data, the analyst, and the evaluator.2 On the other Some consider them to be very accurate, whereas others believe that there are few, if any, analytical results that are accurate. However, there is hand, although the live fire testing methodology has the consensus in the committee that the present analytical models are flawed potential to obtain data that can be used to determine in several ways and are weak in their ability to extrapolate beyond their existing data base. 3 These few locations are selected to maximize the amount of useful data obtained and to answer specific questions concerning the aircraft’s 1 The reader is reminded of the difference between general live fire vulnerability. testing and the congressionally mandated Live Fire Testing. 26

OCR for page 26
EVALUATION OF COST, EFFECTIVENESS, AND DEFICIENCIES 27 three types of weapons provide numerical values for the A poorly modeled phenomenon is the treatment of the typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original vulnerability of the individual critical components and the remaining parts of an impacting fragment or penetrator after it total aircraft from all aspects around the aircraft and for any penetrates a plate. The current model considers only the selected ballistic projectile or guided missile. The numerical largest remaining piece in calculating subsequent effects. values can be in the form of vulnerable areas or probabilities Another poorly modeled phenomenon is the increase in a and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution. of kill. This information can be used to aid in design and component’s Pk/h that occurs when the component is hit by design validation, to satisfy program requirements, to support more than one projectile or fragment or is damaged by both subsequent analytical assessments, to predict test outcomes, blast and fragments. The assumption is nearly always made and to support acquisition decisions. that the same Pk/h value for the first hit in the component is adequate for subsequent impacts on the component. The Accuracy of Analytical Models. There are three aspects of the increase in a component’s vulnerability due to damage caused accuracy of the analytical models that need to be examined: by prior hits is neglected. For example, a tail rotor drive shaft model verification (Are the internal workings of the model in on a helicopter may not be killed when first hit by a tumbled good order—are the equations properly coded?), model 12.7-millimeter armor-piercing projectile with incendiaries validation (Does the model adequately represent the processes (API), but a second hit in the vicinity of the first hit may cause it portrays—do the equations adequately represent the actual the shaft to break due to the synergism in damage between physical situation?), and model accreditation (Is the model projectiles. This increase in Pk/h due to previous hits is usually appropriate to the particular application—is the model capable neglected. of properly representing the aircraft and the weapon?). Also, there are known physical phenomena that are modeled, but their numerical values are not well known Model Verification. In so far as model verification is concerned, because they have not been tested, or the test results are for there appears to be little reason to doubt the veracity of the conditions not satisfied in the combat incident. For example, logic and coding of the models described in Chapter 1. They the penetration equations that determine the velocity and have been widely employed by government laboratories in all mass decay of fragments and penetrators as they penetrate three military Services and by a great many nongovernmental plates on the aircraft may not have the proper decay users. The committee accepts the fact that the internal workings coefficients for the material of interest. Furthermore, these of the models are in good order. equations have been developed for specific geometric shapes of impactors, such as spheres and cubes, whereas fragments are Model Validation. Credibility problems exist with model usually irregular. Two of the most difficult kill modes to model validation. Despite all the physical phenomena that the current in aircraft are fire and explosion, This is due to the randomness aircraft vulnerability models attempt to depict (and many are of fuel sloshing within a tank, fuel leakage into dry bays depicted with high confidence), there are some phenomena around the tanks, and fuel migration into distant portions of that are known to exist, but have not been characterized and the aircraft. entered into the model structure, or are poorly modeled. Perhaps the most notable phenomenon not modeled is the Neglect of On-Board Ordnance. One of the most important random deviation or ricochet of the path of the penetrator or findings of our study is that on-board ordnance is usually fragment from the assumed straight shotline as it passes through neglected as a contributor to vulnerability. Most of the the aircraft components. Another phenomenon not modeled is simulation of internal ordnance aboard the aircraft has been the synergism that occurs when a portion of an aircraft is hit by treated by the aircraft vulnerability community as “clutter.” a multitude of closely spaced fragments. In this multiple, Clutter is inert material in a compartment that is considered closely spaced hit condition, the fragments can be far more only as a compartment filler in the calculation of overpressure damaging to the impacted structure than when they are not in the compartment due to its volume. The overpressure from closely spaced. A third example of a phenomenon not explicitly the explosive “clutter” when hit by projectiles and fragments modeled is spall. The backface spall generated by the impact can change the total pressure in the compartment dramatically. of a projectile or fragment on a plate is not considered as One of the basic premises of all development testing is that additional fragments to be tracked through the aircraft by the modeling must precede testing. Any aspect of hardware impact shotline model.4 on vulnerability must first be modeled so that the testing can be used to verify the modeling rather than for the testing to be extensive enough to cover all statistical events. One of the 4 Spall that is generated within a component is implicitly accounted basic requirements of the Live Fire Test program is to test full- for in terms of its damage to the component since the empirically scale vehicles with a full load of on-board ordnance. The determined total damage is the sum of all of the damage mechanisms and continual neglect of one of processes within the component, including the internal spall.

OCR for page 26
28 VULNERABILITY ASSESSMENT OF AIRCRAFT the basic vulnerability contributors will make it more Fuel System typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original difficult to convince anyone that computer modeling can be Flight Controls/Hydraulics (Control and Power) substituted for the full-up testing of combat-loaded aircraft. Crew Station Engines and Accessories Weapon Lethality Assessments Versus Aircraft Vulnerabil ity Stores, Ammunition, and Flares and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution. Assessments. Some of the programs currently used to compute Electrical and Avionics an aircraft’s vulnerability to the various types of weapons were Structures, Landing Gear, and Armor developed by the Joint Technical Coordinating Group for Helicopter Unique Components Munitions Effectiveness. Because the munitions effectiveness community wanted conservative estimates for the prediction The draft reports of the individual panels are currently being of a weapon’s lethality, its programs were developed so that integrated into a final report. The findings of the eight our weapons were not to be credited with any kill capability working panels are given in Appendix D.6 that was not clearly justified; a sound policy from the weapon The general conclusion from the workshop is that the development point of view. However, as a result of this approach, component vulnerability data base is the weakest link in the the use of these programs to predict the vulnerability of a U.S. vulnerability analysis chain. Clearly, much remains to be aircraft most likely leads to overly optimistic predictions. done with respect to validating those sections of the aircraft Potential vulnerabilities have been ignored unless clearly vulnerability models that deal with component and justified; an unsound policy where the survivability of U.S. subsystem damage prediction. The level of confidence placed aircrews is concerned. Examples of where the vulnerability of in the analytical models would be significantly increased if a aircraft is underestimated are concerted effort was made to determine the maximum error in AV that occurs in the model predictions using the current data • the use of the Thor penetration equations, which consider base, and to what extent this error might be reduced with the only the largest remaining piece of a penetrator as it passes availability of new test data. through the aircraft, • the lack of synergism due to both multiple simultaneous Model Accreditation. Model accreditation is accomplished and sequential hits on a component; by the JTCG/AS. This organization, consisting of • the lack of direct consideration of spall, and representatives from all three Services, has established • the lack of consideration of cascading damage, such as procedures for verifying, validating, and accrediting models. fuel tank damage leading to fuel ingestion. Despite the deficiencies identified by the aircraft vulnerability assessment community in its analytical models, the community The Joint Technical Coordinating Group on Aircraft has been exemplary in the exchange of data and ideas among Survivability (JTCG/AS) Pk/h Workshop. Even if all other the three Services and industry, in the development of aspects of the model were perfect, some of those portions of handbooks and design guides for reducing vulnerability, and the model that portray the results of the physical interactions in the establishment of validated data bases from test data. or damage processes that occur when a threat weapon interacts Although there is still much to learn, more perhaps is known. with an aircraft are inaccurate or incomplete. In order to There will always be a need to update models with new input eliminate some of the model deficiencies, more vulnerability data as aircraft materials and designs change and as the threat data are needed on component kill modes and Pk/h functions. weapons change. This deficiency in the component vulnerability data base was the subject of a JTCG/AS Component Pk/h Workshop held from The Cost of Analysis. It is extremely difficult to determine the March 5–8, 1991, at Wright-Patterson Air Force Base costs associated with a typical analytical assessment since the (WPAFB). cost depends heavily on the type of aircraft being assessed and The objectives of the JTCG/AS Workshop were to critically the particular application for the assessment. However, some review the current state-of-the-art of component P k/h rough estimates for the costs to conduct an analysis that would prediction, to recommend a set of P k/h or P d/h values or be appropriate for a major milestone decision are given below. functions for use in analyses, and to develop plans for The numbers were obtained in a personal communication from improving and validating this set. 5 Working panels were a representative from the Ballistic Research Laboratory. organized as follows: 6 These findings are taken from a preliminary copy of a briefing being prepared to be given to the JTCG/AS Central Office and to the Office of 5 Component vulnerability is sometimes quantified by P d/h , the the Secretary of Defense. This briefing material was provided by Gerald probability the component is damaged given a hit. Bennett, ASD/XRM, WPAFB.

OCR for page 26
EVALUATION OF COST, EFFECTIVENESS, AND DEFICIENCIES 29 Cost of the Model. The cost of creating a COVART or HEVART Live Fire Testing typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original model for a helicopter is roughly between $75,000 and $200,000, with the low end for an upgrade to an existing A live fire test is one in which live ammunition, either system and the high end for an all new, high detail description explosive or non-explosive, is fired at a target. Live fire of a U.S. system. These costs also include the generation of the and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution. testing can be used to assist in the design and design shotline data files needed to run the Computation of validation of an aircraft, to provide empirical information on Vulnerable Area and Repair Time Model (COVART) or High component vulnerability in support of the analytical models, Explosive Vulnerable Area and Repair Time (HEVART) to satisfy the Live Fire Test law, and to support acquisition model. The cost of developing appropriate target models for decisions. Live fire testing is part of either developmental the Stochastic Qualitative Analysis of System Hierarchies testing (DT) or Live Fire Testing. The difference between the (SQuASH) might be two to three times that required for the two types of tests is that DT is part of the normal design and conventional vulnerable area models. 7 As the design development process, whereas Live Fire Testing is that community increases its use of three-dimensional modeling testing intended to satisfy the Live Fire Test law. “As and grid generation, and as computer usage becomes less currently defined, development test and evaluation (DT&E) expensive, the cost of modeling will come down significantly. is that test and evaluation (T&E) conducted throughout the The major cost will be the preparation of input data. acquisition process to assist in the engineering design and development process and to verify attainment of technical Cost of Computer Runs. A typical batch of computer runs for a performance specifications and objectives and helicopter includes two flight modes (hover and forward flight) supportability. DT&E includes T&E of components, and from 6 to 26 attack aspects. The cost of this batch of runs computer software, subsystems, and hardware/software using COVART or HEVART ranges from $45,000 to $100,000. integration. It encompasses the use of modeling, The analysis and preparation of input data account for most of simulations, and test beds, as well as advance development, this cost. prototype, and full-scale engineering development models of the system. Technical performance specifications must be Cost of Obtaining Supporting Experimental Data for Pk/h validated through DT&E in order for the developer (program Functions. The execution of an analytical model without any manager) to certify that the weapon system is ready for the live fire test data on kill modes and Pk/h functions as input data on final phase of Initial Operational Test and Evaluation component vulnerability could be accomplished, but the level of (IOT&E)” (OSD, 1987). confidence in the results would be very low. Consequently, the Of particular interest here is the specific test and evaluation cost of obtaining the necessary data on component vulnerability methodology required to satisfy the Live Fire Test (LFT) must be included in the cost of the analysis. If all of these data are legislation. The law requires realistic survivability available from prior tests, the cost of gathering them is relatively (vulnerability) testing. Although such a test program will in low. If all of the necessary data are not available, supporting tests fact involve early component and subsystem or sub-scale Live must be carried out to obtain the missing data. These tests can be Fire Testing, the hallmark of this approach is a substantial test as simple as the firing of fragments at pieces of plate to measure program that involves a significant number of shots against a penetration or V50 velocities, or they can be as complicated as combat-configured, full-scale version of the weapon system firing several fragments at running engines to determine the (i.e., a full-scale LFT). The early Live Fire Tests on severity of damage, or firing at major portions of the aircraft components and subsystems provide information on any structure that contains fuel tanks to simulate the fire/ explosion vulnerabilities of the individual components and subsystems. and hydraulic ram phenomena. The typical costs associated with Once the information from these tests has been evaluated, the two types of live fire tests are given below. tests on the full-scale aircraft are to be conducted as mandated For a helicopter engine and a 23-millimeter HEI threat, by the law, unless a waiver has been granted. The full-scale test basic preliminary testing with components and/or a static program will often include a number of shots that are engine would cost approximately $30,000. Comprehensive randomly chosen and a number of shots that are selected to testing with a full-up running engine would cost address specific issues. Real (or realistic surrogates of) threats approximately $250,000. For a tail boom structure, basic likely to be encountered in combat must be used in the tests. shoot-and-look damage characterization tests would cost These threats can be non-explosive ballistic projectiles, approximately $25,000. A comprehensive evaluation of the ballistic projectiles with contact-fuzed and proximity-fuzed test results, including postdamage controlled structural high-explosive (HE) warheads, and guided missiles with experiments would cost approximately $150,000. contact-fuzed and proximity-fuzed HE warheads. The three information attributes of 7 Note that SQuASH has not yet been applied to aircraft vulnerability assessment.

OCR for page 26
30 VULNERABILITY ASSESSMENT OF AIRCRAFT types (including amounts and applications), accuracy, and • obtain information on the vulnerability of the aircraft; typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original cost for both sub-scale testing and full-scale testing are • find vulnerabilities that were not anticipated by the examined below. analyses; and • gather data on the synergism among the different damage Type, Amount, and Applications of the Information. In processes and kill modes. and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution. general, the Live Fire Tests on both sub-scale and full-scale All three aid in the design and design validation, support the targets produce information on what actually happened for a analytical models, and support acquisition decisions. particular set of test conditions (e.g., the specific target, weapon, and shotline) and typically for a relatively small Accuracy of and Level of Confidence in the Informa tion. The number of shots under these conditions. 8 Some typical level of confidence that is placed in the subscale and full-scale examples of tests on sub-scale targets are tests to determine Live Fire Test results depends primarily upon the level of the penetration capability of fragments through plates of realism of the tests. Certainly, there is a possibility for realism composite materials, tests on helicopter rotor blades, tail in Live Fire Testing. Some of the vulnerability aspects that are booms, and gear boxes using small arms projectiles and small- represented in Live Fire Testing are the effects of blast, caliber HE AAA rounds, tests on fuel tank simulators using including structural deformation and component damage, and small caliber HE AAA to determine the efficacy of a particular component damage due to multiple fragment hits, including fuel tank protection scheme, and stand-alone tests on running bending, breaking, and perforating. Other aspects of engines using several impacting fragments. The information vulnerability that are represented include the occurrence of from these tests ranges from measured fragment velocities, spall and the penetration through components. The penetration temperatures, and overpressures to the ability of the tested damage and velocity decay in the test are the result of real article to continue to function after the hit. Tests on the full- penetrators going through real materials, with no assumptions scale aircraft will most likely be conducted with the aircraft about breakup, ricochet, etc.9 Synergisms among blast, fire, on the ground or suspended; it can neither crash nor be forced spall, and fragments are realistically represented to the extent to land as a result of the shot. Thus, the ability of the aircraft that the other aspects of the test conditions (e.g., air speed and to sustain the essential functions for flight after the hit is not altitude) are adequately represented. observed directly. Furthermore, rather than produce a Although full-scale tests have the potential to provide the sufficient amount of numerical data from the full-scale test most realism, there are some problems, particularly with the that can be used directly to determine the kill probability of weapons used, the flight conditions, crew vulnerability, and the aircraft for the shot, each full-scale test provides a list of on-board munitions. The weapons selected for testing will damaged components along with descriptions of the details most likely be those that are not overmatching (i.e., they will of the extent and severity of the damage and the associated not have a high probability of destroying the aircraft). damage events for each shot. Consequently, most of the weapons will be either non- With respect to the applications of the information explosive or small-caliber explosive rounds. When larger obtained from the tests, the committee notes that the Live Fire explosive weapons are used, particularly the large-caliber Tests are conducted primarily to (1) satisfy the LFT law and its projectiles and guided missiles, the damage to the aircraft can intent (i.e., to determine any inherent vulnerabilities in the be severe and widespread, making it very difficult to repair the design sufficiently early in the program to allow the aircraft and return it to a condition that would be satisfactory vulnerabilities to be corrected), and (2) provide information in for further testing. Associated with the explosive weapon is support of acquisition decisions. They are not specifically the location of the point of detonation. Contact-fuzed HE intended to provide information that can be used to improve warheads must impact the aircraft in order to damage it, but the analytical models for predicting aircraft vulnerability. proximity-fuzed weapons can detonate at distances ranging Nevertheless, previous experience with full-scale Live Fire from the aircraft skin to several hundred feet away. If the Tests on ground vehicles has shown that important types of warhead is detonated too close to the aircraft, it could destroy it. damage and kill modes have been observed that were not With respect to the flight conditions, airflow can be included in the analytical models for these vehicles. Thus, the simulated to some extent, although it seems unlikely that an information provided by these LFTs can be used in the other entire applications. Besides satisfying the letter of the law, the LFT&E program provides (in principle) an opportunity to 9 Although penetration is realistically represented in the test, the information obtained from the test most likely will not provide penetration data per se for the analytical models because there will be few if any data 8 Typically, the smaller the target, the more greater is the number of on the parameters associated with the individual penetrations, such as test shots, and the more detailed is the information. impact and exit velocities, masses, and angles.

OCR for page 26
EVALUATION OF COST, EFFECTIVENESS, AND DEFICIENCIES 31 transport aircraft would be placed in a uniform high-speed electrical wire bundle that caught fire when hit by a bullet or typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original airstream. Static loading conditions can be simulated, at least fragment. This is a local kill mode that could be discovered over a part of the aircraft, but there is some concern about the by component or subsystem testing. On the other hand, effects of wind gusts and about transient loads produced by suppose the unanticipated kill mode was a fuel ingestion kill maneuvering and loss of control. Furthermore, altitude and of an engine mounted externally on the rear portion of the and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution. temperature are not simulated. For externally detonating fuselage. Hydraulic ram pressure in the fuel in the wing fuel warheads, a static test may result in a different set of blast and tank in front of the engine (due to a penetrator) caused fuel to fragment impact conditions from those of a dynamic test. In spew out the top of the tank. This fuel was ingested by the some warhead/target encounters (e.g., in head-on pass or when engine, which then died. A test of only the wing fuel tank overtaking at very high velocity), the high-velocity fragments might not reveal this kill mode because of the absence of the may impact on one part of the target aircraft’s structure while running engine. the slower blast may affect a different area. In static tests, both Although a full-scale Live Fire Test might reveal a local affect the same portion of the target. unanticipated kill mode, such as the burning wire bundle, full- Clearly, real people cannot be used in the tests. scale testing is not necessarily an efficient methodology for Anthropomorphic dummies, pressure gauges, and gas- obtaining this information. Furthermore, one can not say with sampling equipment can be used to obtain data on great confidence that if no unexpected vulnerabilities vulnerability issues related to personnel vulnerability. On the occurred in the full-scale Live Fire Tests, then there are none to other hand, the responses of the crew to shock, incapacitation, be discovered later in combat. Kill modes involving temporary loss of control, etc., are not directly measured and cascading damage may not always occur in a test, and they must be inferred. These responses are critical to both crew and may be particularly difficult to observe in a full-scale test if aircraft survivability.10 they do occur.11 Some kill modes due to cascading effects are well known, such as the kill of an engine due to the ingestion Vulnerability of the Aircraft. Although it is true that certain of fuel from a damaged fuel tank next to an air inlet, and are catastrophic kills, such as an explosion within an aircraft fuel relatively easy to observe. Others, such as the migration of tank, would be observable in a test, it is also true that other toxic fumes or flames from one portion of the aircraft to types of kills would not be directly observable. For example, another, are not as well known, may not always occur, and may would the aircraft actually crash after damage to one of the be difficult to detect if they do occur. control surfaces? Even if the actual kill was directly observed, It is not feasible to test all possible combat situations for there are the problems of associating the results with the other unanticipated vulnerabilities using full-scale aircraft because kill categories and levels, and of extrapolating the results to of the large number of parameters that affect the target’s other threats and tactical conditions. For example, suppose vulnerability. These parameters include all of the weapons the proximity-fuzed detonation of an 85-millimeter HE likely to be encountered in combat, the tactical situations of warhead near the left differential stabilator of the aircraft interest, and all of the possible impact locations on the aircraft removed 90% of the stabilator. Is this a kill, and if it is for the (i.e., the shotlines). Furthermore, the damaged aircraft should 85-millimeter weapon, would a 57-millimeter warhead be returned to its original condition after each test if detonation in the same location cause the same kill? appropriate and possible. Consequently, the test plan may contain a number of random shots, a number of random shots Unanticipated Vulnerabilities. The analytic models are from directions expected in combat, a number of selected presently structured to provide aircraft kill probabilities based shots, and an ordering that schedules potentially catastrophic upon assumed kill modes, and an aircraft with reduced situations at the end of the program.12 vulnerability is designed to prevent these kill modes from occurring. However, if a kill mode is unanticipated in both Likelihood of Discovering a Particular Vulnerability. Unless the analysis and the design, the model will underestimate the a particular vulnerability is relatively insensitive to the aircraft’s actual vulnerability, and the aircraft will contain parameters associated with a large subset of the conditions this vulnerability. These unanticipated kill modes may be and has a relatively high probability of occurring, it may remain local effects that occur in the vicinity of the original impact, undiscovered in a test program. For example, suppose that a or they may be caused by cascading damage from the impact particular vulnerability event associated with a location to a distant part of the aircraft. For a local effect example, suppose the unanticipated kill mode was an 11 This is the reason that Live Fire Test programs require some random hits and place emphasis on incidents that occur relatively frequently rather than on all unanticipated kill modes. 10 The committee notes that these departures from realism in the live 12 The number of shotlines in the vulnerability analysis of the F-22 fire tests are more severe in analysis/modeling. was approximately 300,000 per threat.

OCR for page 26
32 VULNERABILITY ASSESSMENT OF AIRCRAFT typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution. FIGURE 2-1 Relationship among the probability of occurrence on each test, the number of tests, and the probability of at least one observation over the number of tests. high-explosive weapon had a 0.2 probability of occurrence on or 0.95; and the number of tests. This figure can be used to any random shot for all test conditions of interest. If the test determine the number of tests required to obtain a probability plan consisted of 10 random shots around the aircraft, the of observation of 0.25, 0.50, 0.75, or 0.95 for a given probability the event would occur at least once in the 10 tests probability of occurrence. For example, if the probability of is 1–(1–0.2)10=0.89 and hence the probability it would not occurrence in each test is 0.2 and the desired probability of occur is 0.11. Thus, this vulnerability would most likely be observance at least once is 0.75 or higher, at least seven tests observed.13 On the other hand, if the event could only occur on must be conducted. 2 of the 10 random shots with a 0.2 probability (and a Suppose there were 5 independent vulnerability events probability of zero on the other 8 shots), the probability the possible on each of the 10 random shots, and each event had a event would occur at least once is 0.2 probability of occurring on each of the 10 shots. The probability that any one of the events would occur at least 1–(1–0.2)2=0.36 once during the test program is 0.89. The probability that all of the events would occur at least once (not necessarily on the and the probability that it would not occur is 0.64. Thus, this same shot) during the program is particular vulnerability would most likely not be observed. Figure 2-1 shows the relationship among the probability of 0.892=0.56 occurrence; the probability of observation of 0.25, 0.50, 0.75, Hence, there is a 0.44 probability that one or more of the five kill modes will not be observed. Thus, the test program is 13 Other known vulnerabilities may also occur during the tests that essentially equally likely either to reveal all or to miss one or could obscure the particular vulnerability, and other unknown more of the aircraft’s vulnerabilities. If each vulnerability vulnerabilities could also occur.

OCR for page 26
EVALUATION OF COST, EFFECTIVENESS, AND DEFICIENCIES 33 could occur on only 2 of the 10 test shots, the probability that a wing fuel tank will explode when hit with a probability of typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original one or more vulnerabilities would not occur in the 10 tests is 0.5. If each of three identical tests result in an explosion, does this mean the model is inaccurate? The answer is no.15 Thus, essentially one.14 the Live Fire Tests can not be used in themselves to directly Synergism Among the Damage Processes and Kill Modes. Live validate the models, and they are not conducted for that and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution. Fire Tests are also a valuable source of vulnerability information purpose. However, they do provide valuable information on on the synergism among the damage processes and kill modes, damage processes that should be modeled. albeit a limited source. Suppose there are two kill modes I and II that exhibit synergism. Mode I has a probability of occurrence Neglect of On-Board Munitions.. The Live Fire Test law requires of 0.1, regardless of the occurrence of mode II. Mode II has a the aircraft to be full-up when tested. This means that any probability of occurrence of 0.1 if the first mode does not munitions normally carried by the aircraft must be on-board at occur and a 0.5 probability of occurrence if the first mode does the time of the test. This will probably be done in only a very occur. On one shot, the probability that both modes occur is few tests; it is not the intent of the Live Fire Test law to destroy 0.05. The probability that only mode I occurs is 0.05, the aircraft needlessly. The intent of the law is to obtain information probability that only mode II occurs is 0.09, and the on the vulnerability of the aircraft sufficiently early to allow probability that neither mode occurs is 0.81. If these modes any design deficiency to be corrected. Consequently, it does were independent, and both had a probability of occurrence not seem reasonable to intentionally create a situation in which of 0.1, mode I only occurs with a probability of 0.09, mode II the aircraft could be destroyed when essentially the same only occurs with a probability of 0.09, modes I and II together information on the vulnerability of the aircraft to the on-board occur with a probability of 0.01, and neither mode occurs munitions can be obtained by using realistic off-line tests of with a probability of 0.81. Thus, the probability that neither sub-scale models of the portion of the aircraft in the vicinity of mode occurs is the same in both the synergistic case and the the munitions. This is particularly true for aircraft with independent case. In the synergistic case, mode II is more internally stored munitions. Consequently, inert surrogates most likely to occur (0.14 vs. 0.10) and the probability that both likely will be used for munitions on-board the full-scale aircraft modes occur together is higher than in the independent case in order to prevent a catastrophic kill. The increase (or decrease16) in aircraft vulnerability due to the presence of the (0.05 vs. 0.01). munitions can be determined by relating the observed Validation of the Analytical Model. The analytical models are projectile and fragment impacts on these surrogates to the presently structured to provide kill probabilities and vulnerability data on munition reactions to impacts obtained from the offline sub-scale tests.17 It is not necessary to load the vulnerabilities based upon a selected kill category. The Live Fire Tests do not directly provide the numerical data required munitions on the full-scale aircraft in order to determine the to validate the model’s predictions; they only provide likelihood of an adverse munition reaction to a hit or a fire. If information on the components that were damaged or killed, a design deficiency with respect to the on-board munitions is the occurrence or non-occurrence of the kill modes, and any discovered and a less vulnerable design can be developed, it cascading damage. Thus, some compromises must be made can be incorporated without the loss of the test aircraft. that require additional analyses in order to relate the empirical test results to actual combat conditions and to determine if the The Cost of Live Fire Testing. No complete Live-Fire Testing test results correspond to an actual kill of the aircraft. If enough program for any aircraft has been carried out as yet. test shots could be made under the identical set of conditions, Consequently, the following information is offered only as an a statistical inference could be made regarding the probability indication of the costs involved in full-scale Live Fire Testing. of an aircraft kill given the test conditions. Unfortunately, the If no waiver has been given from full-scale testing, number of identical shots that can be conducted is usually small, and hence the confidence level in the sample mean probability of kill is low. Furthermore, because the models are 15 Nevertheless, a prudent engineer would thoroughly examine the results of the experiment and the analysis in an attempt to determine expected value models, and given the randomness of the whether the sequence of three explosions was not indicative of a higher empirical results from the limited number of tests, the likelihood probability of occurrence. that the test results and the model predictions are in general 16 Some munitions may provide a shielding effect, such as the bombs carried below a wing. Projectiles and fragments that do not penetrate a agreement with respect to the components affected and their nonreacting bomb are prevented from impacting and damaging the vulnerability is low. For example, suppose the model predicts wing. 17 The off-line tests must provide sufficient information to enable a valid evaluation of the reactions of the munitions and the effects of 14 These examples are given to put LFT in the same context with the those reactions on the aircraft, including lower-order effects that are analysis/modeling methodology. It is not the purpose of LFT to find all not catastrophic. vulnerabilities.

OCR for page 26
34 VULNERABILITY ASSESSMENT OF AIRCRAFT TABLE 2-1 Live Fire Test Options for the C-17A (IDA, 1989) typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original Test Item Testing Cost1 Total Cost Cost ($ million) Test Item Option ($ million) ($ million) and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution. 4 1 Full-up 154 158 aircraft 107 2 4 Fuselage & 111 one wing 3 64 4 Fuselage 60 section & one 36 4 One wing 4 40 152 ? ? One wing 5 leading edge 1 This does not include the cost of repairing the damage for the next shot. Also, some members of the committee believe that $4 million is too low. 2 An Air Force estimate. at least one production or preproduction aircraft is required.18 flyaway costs are $78 million. Thus, the cost of the test article If the test is carefully designed, and if the target is reparable up depends upon the method of bookkeeping used. The most to the last shot, one aircraft should be sufficient. The cost of elaborate test program suggested by IDA is estimated to cost the test aircraft is most likely the major cost of the total no more than $4 million and involves 100 small arms program. A question arises as to the proper cost of the aircraft projectiles, 50 small AAA HE rounds, and 20 man-portable to use when determining the program cost. Should the actual infrared missile shots. construction cost of the test aircraft, which might be one of the first five or six aircraft built, be used? Should the cost also Costs for the RAH-66A COMANCHE Helicopter. The cost of include the research and development costs, or should the Live Fire Testing the COMANCHE helicopter has been average flyaway cost be used? The particular cost used will estimated by a representative from the Ballistic Research have a major impact on the perceived benefit of the Live Fire Laboratory and coordinated with the Program Manager (PM). Test program. If one aircraft is tested out of a total aircraft buy Based on projected production system fly-away costs for of 400, and the average aircraft cost over the buy is used, the COMANCHE, a full-up low-rate initial production (LRIP) cost of the Live Fire Tests will be less than 0.3% of the total target that is representative of an operational, combat- program cost. configured system will cost in excess of $7.5 million. Given the alternative (and planned) use of an engineering test Costs for the C-17A. The Institute for Defense Analyses (IDA) prototype aircraft built up to meet specific Live Fire Test has presented a number of test program options, issues, and requirements (a minimal configuration), the cost would be less. costs for Live Fire Testing the C-17A in a draft report (IDA, However, it still is a major percentage of the LFT&E program 1989). The report makes no recommendations on which (if expense. At least two sets of target components/ subsystems, any) of the options should be selected. The costs of five options plus repair provisions, would also be needed. Steps can be are given in Table 2-1. taken to attempt to minimize the risks to the hardware, but At least one production aircraft is required for option 1. If these may not always work. The operational reutilization of the test articles is unlikely.20 the test is carefully designed, and if the target is reparable up to the last shot, one aircraft should be sufficient. The aircraft The number of shots that can be conducted on one target is cost of $154 million is based on a FY1990 recurring flyaway highly variable. However, given ideal repair capability and no cost of $181 million with engineering, tooling, and avionics catastrophes, 25 or more small-caliber API shots costs removed.19 On the other hand, the FY1997 and 1998 would no longer be a “full-up” configuration. In fact, recent testing on 18 The use of static test articles or prototypes for the LFTs should be aircraft shows avionics provide excellent shielding. In some aircraft, considered. avionics components are themselves vulnerable. 19 20 In the committee’s evaluation of the cost of full-up, full-scale It should be noted that all of the test vehicles used in the U.S. Army’s aircraft estimates are based on flyaway cost without avionics. This ABRAMS Live Fire Tests were repaired and returned to service.

OCR for page 26
EVALUATION OF COST, EFFECTIVENESS, AND DEFICIENCIES 35 TABLE 2-2A Relative Advantages and Disadvantages of the Tvvo Methodologies; Type, Amount, and Applications typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original of the Information Analysis/Modeling Live Fire Testing and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution. Sub-scale Testing Full-scale Testing Advantages Advantages Advantages • The information is available in the • The type of information from sub- • The type of information from full- form of vulnerable areas and scale testing is in the form of semi- scale tearing is. in the form of realistic probabilities of kill given a hit. realistic physical outcomes, such as physical outcomes, including cascading penetration or no penetration, rife, or no and synergistic damage, such as which • The models can provide much fire, etc. components are damaged, if a fire informaiinn on the vulnerability of the occurred how far it spread, etc. aircraft Over a wide range of conditions • The information from sub-scale tests in which the systems will be used, over can be used to aid the design (when the • The information from full-scale tests a wide range of potential threat tests are conducted sufficiently early in can be used lo aid ihc design (when the weapon, arid Over a wide range of the program), to validate the design, to tests arc conducted sufficiently early in system design alternatives. provide data for component Pk/h data the program), to provide information on bases and new damage processes in new damage processes in support of • The information can be used to aid in support of subsequent assessments, and subsequent assessments, to satisfy the the early design of the aircraft, to to support acquisition decisions. requirements of the Live Fire Test. Law, satisfy program requirements, to support and to support acquisition decisions. subsequent campaign/war game assessments, to predict test outcomes, • The Live Fire Tests that were and to support acquisition decisions. performed on full-scale armored vehicles provided data on the rypes of damage that resulted in delayed catastrophic damage. These data were useful in training the crew in emergency exit procedures, Similar data on ejection procedures might come from Live-Fire Tests of aircraft. • Damaged aircraft offer an opportunity for bailie damage crews to practice repairing realistic combat damage. Disadvantages Disadvantages Disadvantages • Model results reflect the assumptions * The amounl of information is • The amount of information is on vulnerability and therefore do not relatively small, is confined to the tested relatively small, is confined to the tested provide any new information on components and subsystems, and is regions, and is limited to the specific vulnerabilities resulting from new limited to the specific test conditions. test conditions. An unexpected materials and damage processes. catastrophic reaction in one of the early • The information from sub-scale tests tests can prevent the gathering of may not be directly relatable to an additional information. aircraft kill. • The information from fiill-scaie tests may not be directly related to an aircraft kill. • Because of production-Iike aircraft must be rested in the full-scale test, it may not be practical to correct design deficiencies discovered in the tests. * Full-scale rests will not be used to produce the necessary broad and detailed data base for supporting the models because of the lack of control over the test outcomes and the costs involved.

OCR for page 26
36 VULNERABILITY ASSESSMENT OF AIRCRAFT TABLE 2-2B Relative Advantages and Disadvantages of the Two Methodologies; Accuracy of, and Level of typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original Confidence in, the Information Analysis/Modeling Live Fire Testing and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution. Sub-scaie Testing Full-scale Testing Advantages Advantages Advantages • Models can provide eslimatcs of the • Sub-scale tests can provide • Short of actual combat full-scale live major vulnerabilities, particularly when vulnerability data un new materials, fire testing is as close to real data on a new components, new construction combat-configured aircraft as is possible rtiey are supported by sub-scale live fire processes, new threats, etc. to achieve. It accounts for the weapons testa. effects, synergism among damage processes, and cascading damage. Disadvantages Disadvantages Disadvantages • The models will never be able to • Sub-scaie tests can vary from simple * Those weapons likely to be totally represent all of the possible component tests to realistic tests on encountered in combat that can destroy vulnerability evtnts that can occur in complete subsystems. However, because the aircraft are not used in such a these lasts cannot simulate any manner. They are accepted as combat, such as syncrgism and interaction between the tested article overmatching, threats. cascading damage. Some damage and the portions of the aircraft not processes are modeled poorly, and some • Altitude effects and the conditions of included, they aie not fully are not modeled at all. Thus, the flying are not included in full-scale representative of the actual combat estimates of vulnerability do not have a vulnerability testing. situation. high level of confidence, particularly for • Because the air crew is not on board new aircraft. during an LFT, the possible effects of • Current models tend to underestimate the lest on the crew are difficult to vulnerability because of their determine. conservative assumptions regarding weapon lethality (e.g., they neglect spall, fuel ingestion, and multiple hit effects). • There are serious data gaps on component and subsystem vulnerabilities, such as those identified by the 1991 JTCG/AS Pk/h Workshop. plus four or more small-caliber HEI projectile shots might be disadvantages of both sub-scale testing and full-scale testing. possible. The cost per shot is highly variable, ranging from Table 2-2 presents a comparison of the methodologies for each $500 to $10,00021, and is a function of the weapon used, the of the three information attributes of type, amount, and target configuration, the scope of the test issues, the extent of applications; accuracy or level of confidence; and cost. the instrumentation and data capture, the workup costs, etc. Conclusion Advantages and Disadvantages of Analysis/ Modeling and Live Fire Testing The committee concludes that the combination of analytical models, supported by live fire tests on components and Both analysis/modeling and Live Fire Testing have subsystems, and the sub-scale and full-scale Live Fire Tests are advantages and disadvantages or deficiencies relative to one mutually compatible in the vulnerability analysis, another. Furthermore, there are relative advantages and evaluation, and design of aircraft. They complement each other, and the whole is superior to the sum of the parts. More work is needed to unify these approaches in order to obtain the 21 Some committee members believe that the cost of the tests would maximum benefit. The aircraft vulnerability range from $5,000 to $50,000.

OCR for page 26
EVALUATION OF COST, EFFECTIVENESS, AND DEFICIENCIES 37 TABLE 2-2C Relative Advantages and Disadvantages of the Two Methodologies: Cost typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original A nalysislModeling Live Fire Testing Sub-scale Tesiing Full-scale Testing and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution. Advantages Advantages Advantages • The cost of analysis is significantly • Sub-scale tests are considerably less • Live Fire Tests are a small less than the cost of Live Fire Testing. expensive than full-scale tests. percentage of the total program costs. If only one aircraft is tested out of a total aircraft buy of 400, the cost of the Live Fire Test program is less than 0.3% of the total program cost. • The use of static test articles or prototypes can significantly reduce the tost of the tests. Disadvantages Disadvantages Disadvantages • Although the actual cost of • The cost of full-scale testing is • Although sub-scale tests are less primarily influenced by the cost of the conducting an analysis is relatively low, costly than full-scale tests, a test aircraft. Thus, the cost of testing a comprehensive sub-scale test program in there are considerable sub-scale test full-scale aircraft is considerably more support of a particular aircraft costs required to build up the essential than the cost of analyses or sub-scale acquisition program is not inexpensive, vulnerability data base. testing when the aircraft was procured and the program manager may not be only for that purpose and cannot be inclined to spend the required funds. repaired and put into service. • The cost of sub-scale testing may preclude testing for basic phenomenological data on damage processes and kill modes. community, based on its plans for the Live Fire Tests, seems to References appreciate the need to integrate these approaches, having • Institute for Defense Analyses (IDA), 1989. C-17A Live Fire Test witnessed the success in using the data from the Live Fire Tests Options Report, Paper P-2228. on the Abrams tank and the Bradley Fighting Vehicle to • Office of the Secretary of Defense, 1987. Report of the Secretary of improve both the analytical methodology and the vehicle Defense on Test and Evaluation in the Department of Defense. designs.