Click for next page ( 32


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 31
3 Practicality' Affordability, and Cost-Benefit This chapter presents Me committee's views on the practicality of full-up, full-scale testing for the F-22. It also discusses affordability and cost-benefit aspects of conducting and not conducting these tests. Finally, the committee's opinion on the desirability of the F-22 waiver is expressed in the context of its conclusions on practicality, affordability, and cost-benefit. PRACTICALITY The committee defines an activity as "practical" if meaningful results can be achieved with existing technology and reasonably available resources. The issue facing the committee is whether it is practical to conduct fi~-up, full-scale, live fire tests of the F-22. Several considerations were taken into account in addressing this issue; they serve to organize the discussion of practicality that follows. Relative Importance of Vulnerability Reduction to F-22 Survivability The live fire test law's definition of "realistic survivability testing" states that "the primary emphasis [is] on testing vulnerability with respect to potential user casualties arid taking into equal consideration the susceptibility to attack and combat performance of the system" (10 U.S.C. 2366~. Judgments about the practicality of full-up, fi~-scale tests for the F-22 must therefore take into account its susceptibility and combat performance. The probability of any system surviving hostile action is usually expressed as the difference between unity (a perfect return rate if hostile action is ineffective) and the product of the probabilities representing the effectiveness of hostile action. In other words, the probability of survival is defined as ~-PD(P~(P~), where PD is the probability of being detected; PH,D is the probability of being hit by the enemy's weapon, once detected; and PIC,H is the probability of the friendly system being rendered permanently and completely ineffective, once hit. It is clear that the lower the value of any one of these three 31

OCR for page 31
32 Live Fire Testing of the F-22 probabilities PD' PHID, Or PIC,H with the other two held constant, the greater the friendly system's chance of survival. The last of these probabilities, PIC/H, is the metric influenced by reducing vulnerability and dealt with in planning and executing live fire tests. As indicated earlier in this report, the F-22 design has been optimized for its primary mission of offensive courter air. The aircraft includes both offensive and defensive capabilities that are expected to result in destruction of enemy aircraft beyond visual range before Me enemy aircraft can locate the F-22 arid fire a missile. Only a very small proportion of encounters are expected to result in closure to ranges at which enemy missiles or guns could be fired at the F-22. The F-22 has been designed with a high decree of maneuverability to deal with the occasional close engagement. Accordingly, F-22 survivability depends more on low values of susceptibility (i.e., the product of PD and PHID) and less on low values of vulnerability (PK/H) Any option to reduce F-22 vulnerability further must be evaluated for its effect on combat performance (e.g., Tow susceptibility), with priority generally going to preserving the performance. Of course, the priority on Tow susceptibility does not Rae out design changes to address a critical wInerability discovered in the live fire test program and attendant analysis and other testing. The issue is not whether a live fire test program is required all agree that it is. The issue is how far to go with the live fire test program. For the F-22, this judgment must be influenced in part by the relatively low weight the Air Force has given to vulnerability in the overall survivability equation. Chapter 2 cited indications that the F-22 Will also have an air-to-surface mission in the future. As currently envisioned, that mission we involve delivery of munitions from a relatively high altitude. An a~r-to-surface mission could affect the terms in the survivability equation. Specifically, the vulnerability term might assume greater significance because of increased exposure of the F-22 to surface- to-air defenses. Future missions for the F-22 will require Mat the Air Force reassess the relative importance of vulnerability and susceptibility arid adjust the wInerability assessment program accordingly. c:7__ ~ ~ High-performance fighters result from a highly integrated and carefully balanced optimization of a substantial variety of systems. Since survivability is influenced by all three terms (PD' PM/D, and PK/H)' the aircraft designer should not make changes to any one of them without considering its effects on the other two. For example, if it were possible to reduce PK1H by armor plating an avionics bay, but in so doing the armor increased the aircraft's radar reflectivity, raising its PD, and if the weight of the armor significantly reduced the aircraft's maneuverability, raising its PH,D, then the improvement in PK,H could be ill advised. This kind of one-dimensional improvement activity involves the dangers of suboptimization.

OCR for page 31
Practicality, Affordability, and Cost-Benefit 33 No consideration of vulnerability reduction can be definitive without being specific about the threat, which can influence all three terms in the survivability equation. The next section discusses threat realism, and Chapter 4 describes the threat environment projected for the F-22 and how that environment is reflected in the vulnerability assessment program. Realism in Aircraft Testing The committee believes that live fire testing can be conducted at four levels of aircraft readiness for the intended mission. These levels, which reflect various degrees of realism, are discussed below. Even the highest level does not adequately simulate actualflight conditions. Level i'. The first and lowest level is live fire testing of hardware simulations or mock-ups of production systems. For example, the hydraulic elements of a flight control system might be assembled in the proper filll-scale geometry and provided with flight-condition operating pressures, but mounted on a working representation of an airframe. The committee believes that such tests are encompassed by the waiver Drovision of the live fire test law resee Ann~nr1i~r R Section 2366(c)(2)]. ~Lo err ^ ~ [eve! 2. The second level involves fi~-scale aircraft components, subsystems, or subassemblies representative of production items. Depending on the purpose of the test, all hardware that would be present in the production configuration may or may not be present in the component, subsystem, or subassembly that is tested. The committee considers these to be the tests referred to in the waiver provision of the live fire test law "Section 2366(c)~2~. Level 3. The third level of live fire testing involves a complete, production aircraft not loaded with live ordnance or fuel. Selected systems may or may not be operating. The corrunittee considers this to be inert (not Let-up), full-scale testing. A waiver would be needed if only this type of full-scale system testing were planned. [eve! 4. The fourth and highest level utilizes a complete production aircraft with all systems, fuel, and live ordnance installed and in operation typical of combat. A fi~-up, fiull-scale test can be considered a verification (based on a random sample) of the results of the lower level tests. Tests on the aircraft would be conducted in ground-test facilities where test conditions can be closely controlled and flight conditions can be approximately simulated. Win currently available facilities, it is impossible to generate the air flows and actual design stress levels encountered in flight. Parts of the aircraft can be bathed in high-speed subsonic air, and some lower states of stress can be simulated. Tests under these conditions would, in the opinion of the co~runittee, come the closest to meeting

OCR for page 31
34 practically the congressional intent for fi~-up, full-scale testing. It is desirable, of course, to make any test as realistic as rea- sonable. In live fire tests of aircraft, realism can be thought of as having three independent aspects, as discussed below: the threat, the configuration of the friendly system, and the opera- tional status of the friendly system. If cost were not a consideration, complete realism, or something close to it, might be achieved in all three aspects. But since costs are in fact a determinative con- straint, it is important to consider the cost of approaching realism in any one aspect in the light of realism achievable in Me other two. Live Fire Testing of the F-22 . . . ...,. . , Ad. I,~t , , . . ,[~.. flu tu.~ .: ~.~ Ql .~ o:l -I . . . : 'ci?~PmRii.~:~.~:~'n~i:iA2h~': :: : . ' ~ arms Blat theme law apes pus in co~ecti i i se of . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . '., :,, "w,2,or.'.d, ."subasse"~b,'lms." ' 'it' ' ' ' s"" ' "' ' e' :::, : . a. w~.$, 8; . ~lage . e~.. ~e :.i ..: sub~se~iies : be of e : :. .............. . . . . . . : .: . . ~ompos ~t t ade - I: by: , '~,','.'i''=,'~,]lit,> ^ Is'" . cond ct~. ~i s ~ es ~ ~e ....co mi .ee...uses....te~. s like 'r.l ge ~] s,": . : the "m-yor ~=s" ~ ' ~ s :: ............. . ................ . ..... subseries " ese : e ssi erel:: . . bledhi : . ~ ~ :. ..~sembliesmeds0~bel~ge.~ugh~ :: ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... . . . . . . . . . . . . . . . . . . . . . . . . . ..... . .. . . . . . . . . . ^ - ......... ............... . ,. .. : : . sutt~cle.~t .y . .represe~itat~ve : ~ a pace U;~1:~3n .. , ......... . . . . : :: : -. ~ confi=r~or~ to meet He i~e fire te~ ... . . = .. .. . .. .. .... .. Threat realism involves a wide range of variables (e.g., if the threat is explosive, variables include the distance from its target upon detonation, the kinematics of the intercept, and the characteristics of the blast generated). Because testing the virtually infinite number of possibilities is obviously out of the question, definition of the threat involves the assignment of probabilities to all of these characteristics of weapon-target relationships. The indeterminable features of threat definition are at the root of the statistical nature of the analysis needed to interpret live fire test results. In complex ways, such considerations must enter into the determination of Pa,, and influence measured or computed values of Pan. NO Achieving realism in the aircraft s configuration is straightforward, but it can be expensive. The possibility of discovering an unexpected interaction between systems argues that everything to be catTied on a mission be in place during live fire testing. This is the fulI-up aspect, which includes feel, ammunition, and hydraulic fluid. In addition to correct size, shape, and hardness, the fi~-scale aspect requires that all components be installed (e.g., wire bundles and batteries). Although the aircraft's configuration when hit by hostile fire cannot be predicted, it will usually be possible to predict the most vulnerable condition (e.g., live ordnance and filets. There is little that cannot be determined about the friendly aircraft's configuration for live fire testing.

OCR for page 31
Practicality, Affordability, and Cost-Benefit 35 Achieving realism In the aircraft's operating conditions, however, is another matter. Some operational circumstances can be replicated in live fire tests at reasonable cost, but others cannot. For example, hydraulic systems can be pressurized and engines can be running in ground tests. On the other hand, the combined effects of high-speed airstreams and maneuvers may make the results of ground tests a poor indicator of what wait haDnen under dynamic Fight conditions (e.g., a high-g turning pull-up). The variability introduced by indeterminable aspects of the threat and operating conditions leads to one of the most important considerations in achieving realism, namely, the concept of statistical significance. Statistical significance quantifies the confidence that one can place in the test results obtained. This confidence is a function of the variability of the data and the size of the sample set, which is the number of trials that can be conducted in the case of live fire testing.2 It is important to be able to support adequate sample sizes to achieve reasonable statistical significance across all levels of testing. Vulnerability assessment models and simulations provide a way of achieving relatively large sample sizes at relatively Tow cost. However, current models and simulations have their own sets of problems, which are discussed later (see Chapter 51. Taking all of the above into account, the committee believes there is no feasible way known to test the vulnerability of a first-line fighter like the F-22 in a fully realistic way, that is, as defined by law. It might be possible to conduct a one-or-few-trials flight test with drone aircraft. But the variability associated with threats and operational conditions would require tens of trials for meaningful results. Although large numbers of trials in the fi~-up, full-scale configuration could conceivably be feasible for obsolete fighters, Hey are not practical for an expensive new operational system such as He F-22. The difficulty (in some cases, inability) of conducting realistic tests must be considered when planning live fire test programs for the F-22 arid evaluating their results. _~] ^ ~_^,_ ~_} I,,,_ ^--~-~ 2 It is important to recognize, in particular, that the results of a single test do not provide statistically significant information about low probability events, such as "unknown unknowns." These events are not likely to occur in a single test (or even a few tests). For example, if an event has a 10 percent probability of occurrence, and one wants 95 percent confidence that it will be observed in the test sequence, then the number of test trials needed is about 28 (NRC, 1993~. With half the number of tests, the confidence of observing the event drops to 75 percent. With one- fourth the number of tests (i.e., 7), the confidence drops to SO percent.

OCR for page 31
36 Live Fire Testing of the F-22 Destructive Versus Nondestructive Testing Destructive testings can be of great value in any developmental effort. Such testing can be conducted at any of the levels discussed previously. However, by itself, destructive testing of a complete aircraft at the full-scale level yields extremely Tow confidence factors for probabilistic outcomes due to the small number of trials possible. At Tower levels of testing (e.g., the level that involves components), greater numbers of destructive trials become possible. On the other hand, some aspects of realism (e.g., the synergistic effects associated with a full- up, full-scare test program) may be lost at these levels. Even a full-up, fulI-scaTe destructive test would not in itself provide meaningfully representative information. Nondestructive tests can also be conducted at all levels with varying degrees of realism. These tests can provide a sufficient number of trials for reasonable confidence at reasonable costs. For example, nondestructive and repeatable fi~-scale tests with nonflammable, nontoxic materials may in certain circumstances provide useful data. The committee believes that there is a major difference between destructive and nondestructive testing of the F-22. When nondestructive tests can be designed (e.g., with high-power microwaves4) it appears feasible to test the filll-scale aircraft in sufficient trials to make a mearungfu} assessment of vulnerability. The committee supports such testing. The committee's support does not extend to full-up testing of the F-22 in situations that might detonate any live ordnance or fuel on board. Provisions to bypass the destructive features would, of course, make the tests less than Let-up. Expert Opinion Finally, the opinion of most experts with whom the committee discussed the ~ .. ... ~ .~ . ~ ~ ~ , . - ~ 1 . _ _ ~ matter, and the committee7s opinion, is that tull-scale testing or a complete a~rcran is much less likely to provide useful information than are appropriate component, subsystem, and subassembly tests. This opinion is based on Free factors: 3 Destructive testing is commonly understood to be the opposite of nondestructive testing, which is an approach to testing that does not involve damage or destruction of the test sample. By its very nature, live fire testing causes damage to the material or component being tested. In fact, the extent of damage is one of the key results of a live fire test. The committee adopts the common understanding discussed here in its use of the terms "destructive" and "nondestructive' testing. 4 The committee notes that high-power microwaves can cause some damage to components, so even these kinds of tests may not be considered truly nondestructive.

OCR for page 31
Practicality, Affordability, and Cost-Benefit 37 A full-scale test with a threat weapon sufficiently large to affect the whole aircraft may not be repeatable over a reasonable range of conditions urdess many aircraft samples are used. The committee judges that the benefits of such an approach would not be worm the costs for an aircraft like the F-22. If a munition with a local effect is used for the test, then a component, subsystem, or subassembly can be used as the test specimen. A buildup test sequence could be used that (a) starts at the component level, where the greatest number of trials are possible; and (b) moves to the subsystem and then the subassembly or large assembly level, as appropriate. This approach would permit sufficient trials for developing and confirming (or disproving) various hypotheses about what darnage might or might not occur at the different levels. The collective wisdom (including hard data) that is established by this process provides reasonable confidence in the results. There is a need to understand damage and failure mechanisms at the component, subsystem, and subassembly levels. The committee was briefed by at least one individual (O'Bryon, 1994) wh stated his strong belief that testing at the full-scale level is necessary, using production-authentic hardware and systems, but without live ordnance aboard. He is correct that the current F-22 live fire program does not go this far. Specific arguments raised in the above individuaT's presentation for accomplishing fi~-scale (if not full-up) testing of the F-22 were essentially (a) there are secondary effects (ricochet, debris, spelling, etc.) that do not reveal themselves in smaller scale tests; (b) synergistic effects twhere damage to one subsystem may cause damage elsewhere, sometimes called cascading effects (see Chapter 4~] should be determined by full-scale tests; (c) system degradation should be measured as the result of such tests; (~) battle-damage repair insights should be encouraged; (e) fire starting mechanisms should be observed. and fire O A, _ _ ~ SUppreSSlOn Should be evaluated; Ed (~) '~own ~o~s" can occur during full-scare tests. These arguments appeared to represent not only this individual's views but also the views of others who advocate fi~-up, fi~-scale testing. The committee carefully weighed the array of arguments on bow sides of the fi~-scaTe testing issue. The committee was persuaded that, for a system like the F-22, considering its mission and system characteristics, a step-by-step approach to vulnerability assessment was best. A test plan that dictates a methodical buildup of tests from the component level, to the subsystem level, to the subassembly level, to the large assembly level, and, if required, to the full-scale level made more sense to the committee than assuming at the outset that fi~-scale tests were necessary in every case. Additionally, while each of the enumerated benefits of

OCR for page 31
38 Live Fire Testing of the F-22 full-scale testing is important, in the opinion of the committee arid others who briefed the committee, most can be achieved without resorting to full-scale testing. The philosophical issue involved is that tests must proceed to a level that, when taken collectively with the results of all the lower level tests, produces a reasonable likelihood of revealing all necessary information. To some, that dictates full-scale testing, others disagree. It is true that fi~-scale tests have a potential for disclosing surprises that no one could predict through modeling, analysis, or component-to-subassembly testing. However, for the F-22, especially in light of its very high cost, a test without a reasonable expectation of additional valuable data to be derived is unwarranted. The committee did not discern a reasonable expectation of deriving valuable data (e.g., ''unknown unknowns") from a full-scale test of the F-22. If it had, the committee would have had no qualms about recommending such testing. The committee concluded, in light of its own expert judgment and the prevailing opinion of most others with whom the committee met, that full-scale testing of an aircraft like the F-22 is not justified. However, the committee does agree that testing of F-22 subassemblies in a complete or nearly complete production configuration is justified in appropriate circumstances, as discussed in Chapter 4. AFFORDABILITY The committee defines an "affordable" activity as one within budgetary constraints or attainable budgets. Like all acquisition programs, the prioritization of alternatives within the F-22 program is based on an assessment of their marginal utility. Unaffordable budget items remain unended until either more funding is authorized or the priorities of activities within the program are reassessed (e.g., previously unaffordable items displace funded items based on analyses Mat reassess their relative benefits). The committee was not able to consider Filly or challenge the prioritization of items within the F-22 program budget. However, based on the committee's judgment of benefits versus costs, even if an aircraft could be provided, the full- up, fi~-scale tests would not be recommended. As a result, the overall program budget and prioritization of expenditures within that budget, as established by the F-22 SPO, was accepted by He committee. Clearly, in the SPO director's estimation, full-up, full-scale testing exceeds the available budget (Raggio, 1994~.

OCR for page 31
Practicality, Affordability, anc! Cost-Benefit Affordability of Full-Up, Full-Scale Testing 39 According to data provided by the SPO as of February ~ 995, the cost of the currently planned F-22 live fire tests is slightly over $38 million (then-year dollars). This amount funds multiple component, subsystem, and subassembly tests. The SPO projects that a full-up, full-scale test of the F-22 would cost an additional $250 million (then-year dollars) above the currently planned program. The major component of this amount is purchase of a full-up production aircraft.5 The underlying assumption is that this test asset would be devoted fully to the tests and could not reasonably be refurbished to have military utility after completion of the tests. The committee requested information to support the contention that a production aircraft would be required in lieu of some less realistic alternative with somewhat less test fidelity (e.g., the prototype test aircraft). The SPO responded that use of three alternatives had been examined: (~) the prototype air vehicle, (2) the static ground test article, and (3) an engineering and manufacturing development flight test aircraft (Graves, 1995~. None ofthese was deemed feasible in the judgment of the SPO. The SPO's estimate that the cost of a full-up, full- scale test would be on the order of $250 million results directly from this judgment. The committee has no basis for refuting the SPO's judgment. There is an argument that $250 million is only a small percentage (less than 0.5 percent) of total F-22 program costs, and therefore cost should not be a determining factor. The committee understands this view. Nonetheless, the committee's judgment is that the benefits of fi~-up, full-scale tests are not commensurate with the costs. Even if $250 million were provided for additional vulnerability assessment of the F-22, the committee would not support using the fiends for full-up, full-scale testing. Both Air Force and Navy experts on aircraft vulnerability assessment indicated that, if they were provided a new production aircraft for live fire testing, they would prefer to disassemble it arid perform live fire testing on a less than filll-up, full-scale configuration. They stated that Weir preference to test at the component, subsystem, and subassembly levels was derived from their experience that they would learn more about vuinerahiiitv and could mace orbiter ~onfirl~nr~. ire the results, for the resources expended.6 ~, ~ ~~~~~ rat s The fill-up aircraft represents well over 90 percent of the $250 million cost estimate (Graves, 1995~. 6 The opinions in this paragraph were expressed during discussions with members of the committee on February 21, 1995, at the Naval Air Warfare Center, China Lake, California.

OCR for page 31
40 Live Fire Testing of the F-22 Investment Methodology for F-22 Vulnerability Tests The committee was briefed by the SPO on an investment methodology that examined incremental live fire testing of the F-22 (Griffis and La~z:~e, 1 995~. The committee was not persuaded by the SPO's analysis. The committee believes that the SPO's investment mode} offers a limited context from which to reject, analytically, additional increments of live fire testing. Analytic approaches based on incorrect premises have a way of producing logically derived results that may well be wrong. Additional comments on cost-benefit methodology appear below. COST-BENEFIT METHODOLOGY The committee received briefings on the current state of the art of cost- benefit analysis of live fire testing (Griffins and Lauzze, 1995; Klopcic, 19951. Attempts are under way within DoD to develop a methodology for determining the return on investment of successive levels of live fire tests by predicting the costs of fixture aircraft attrition. If the cost of an additional live fire test is known (it is), and one is reasonably sure of the incremental reduced cost of attrition that a test would bring about (one is not), the return on investment would be clear and the additional test could be judged on that basis. The methodologies presented are immature at this time. They appear to address suboptimal measures of benefit. Despite these shortcomings, the committee believes that a validated, useful methodology for determining, quantitatively, the cost-benefit relationships of live fire testing would be a valuable too! for vulnerability assessment. This kind of too! would help prioritize the various levels of live fire testing together with other competing program activities. The committee acknowledges that validating a usefill cost-benefit framework is, in fact, extremely difficult because of the need to establish a priori benefit valuations. There is a danger that a framework could produce misleading results if the benefit measures chosen are so narrow as to preclude interesting alternatives. The committee suggests that this risk be minimized by conducting a series of excursions that assess proposed test programs in the light of alternative measures as well as widely varying but conceivable test results. In addition, it might be useful to consider military scenarios that would elevate the importance of reduced vuInerability7 arid, thereby, possibly enhance the benefit-to-cost ratio of given levels of testing. 7 Here, the concept of reduced vulnerability could extend to fewer losses of flight crews (i.e., capture or death) and smaller likelihood that U.S. systems would be recovered and exploited by the enemy.

OCR for page 31
Practicality, Affordability, and Cost-Benefit 41 Additional live fire testing of the F-22 at any level should be examined in terms of opportunities for information to be gained, test costs, and conceivable consequences of not performing the tests. Such testing should not be rejected because of cost-benefit scores in a single, and perhaps overly simple, construct of how the future may develop. For example, assume that (a) the United States will be faced with a series of wars like Desert Storm extending through the first half of the twenty-first century, and (b) the F-22 wait confront significantly improved defensive systems, wall be heavily used, and will be at risk in each sortie it flies. In the face of these assumptions, the total number of combat sorties flown is the key measure. Reduced vulnerability and increased capability to repair battle-damaged aircraft may become determinative. Under these conditions, additional investment in live fire testing could be attractive, particularly if it might reveal ways to produce a significant increase in the number of combat sorties flown (e.g., through increased tolerance to damage sustained in combat arid the development of techniques for repairing damaged aircraft in the field). CONCLUSIONS Having weighed the arguments on both sides of the full-scale testing issue, the committee was persuaded that, for a system like the F-22, a well conceived, incremental build-up of tests that proceed from the component level to the subassembly or large assembly levels made the most sense. The committee did not discern a reasonable expectation of deriving additional valuable data (e.g., "unknown unknowns") from a filll-scale test of the F-22. The committee concludes that completely realistic, destructive, full-up, full- scale testing of the F-22 is not practical and entails high costs relative to the resulting benefits. The judgment of most members of the live fire test community with whom the committee met is that incremental testing to the point of relatively large subassemblies is the way to proceed and Mat full-up, full-scale tests of combat-configured aircraft are of marginal utility. The committee agrees with this judgment. The combination of the lack of realism in test conditions, the difficulty of obtaining a sufficient number of trials, and expert opinion all support this conclusion. Full-up, full-scale testing in a configuration that could destroy the entire aircraft if a detonation occurs is therefore not warranted for the F-22. On the other hand, nondestructive testing (e.g. with high-power microwaves) is practical, as is destructive testing of components, subsystems, and subassemblies. The committee s The F-22's new composite materials and new systems could require battle-damage-repair techniques that are much different from those used on current fighters.

OCR for page 31
42 Live Fire Testing of the F-22 endorses the use of production-representative articles for these tests. Also, it is practical to test parts of an aircraft under simulated loads arid exposure to high- velocity airflows. Regarding the affordability of the tests, the committee has concluded that affordability is not the matter of foremost relevance. Cost-benefit is most relevant. Affordability only becomes relevant if the benefits relative to the costs of whatever tests are being considered are commensurate with the benefits relative to the costs of other alternatives. With respect to full-up, full-scare tests for the F-22, the committee judges the benefits to not be worth the costs. Based on its conclusions concerning the impracticality and low benefits for the costs of fi~-up, full-scale, live fire testing for the F-22, the committee is unanimous in its opinion that a waiver is the appropriate course of action for the F-22. It must be pointed out, however, that while the committee was asked to examine practicality, affordability, and cost-benefit, the law states that a waiver may be granted by the Secretary of Defense based on a certification "that live-fire testing . . . would be unreasonably expensive and impractical "emphasis added]." The committee's interpretation is that both conditions must be true before a waiver can be granted. Regarding the term ''unreasonably expensive," the committee believes that the Tow benefits relative to costs (i.e., high costs relative to benefits) means that the tests are unreasonably expensive. If a waiver is granted, there needs to be some measure of sufficiency for the test program that is conducted. The sufficiency of live fire tests currently planned for the F-22 is addressed in the next chapter. Finally, the committee recognizes that its judgments regarding the costs arid benefits of full-up, full-scale testing were reached in the absence of a mature methodology for assessing benefits relative to costs. The committee is leery of reliance on cost-benefit methodologies Mat use an overly simple construct of the F-22's fixture to make judgments about how far to go with live fire testing. A broader ar~alytical framework could elevate the importance of reduced F-22 vulnerability over the long haul and might enhance the benefit-to-cost comparisons of given levels of testing.

OCR for page 31
Practicality, Affordability, and Cost-Benefit REFERENCES 43 Graves, I.T. 1995. National Research Council Questions on Live Fire Test. Memorandum from Deputy Director, F-22 System Program Office, to National Research Council, Mike Clarke, February 14. Griffis, H., and R. Mae. ~ 995. Cost Benefit Analysis Methodology. Presentation to the Committee on the Study of Live Fire Survivability Testing of the F-22 Aircraft, Dayton, Ohio, January 20. Klopcic, I.T. 1995. Knowledge-Based Benefit/Cost Methodology for Live Fire Test Evaluation. Presentation to the Committee on the Study of Live Fire Survivability Testing of the F-22 Aircraft, National Academy of Sciences, Washington, D.C., February 16. NRC (National Research Council). ~ 993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Air Force Studies Board, NRC. Washington, D.C.: National Academy Press. O'Bryon, I.F. 1994. Discussion of Live Fire Testing Philosophy and the History Associated with First Report. Presentation to the Committee on He Study of Live Fire Survivability Testing of the F-22 Aircraft, National Academy of Sciences, Washington, D.C., December 21. Raggio, R.F. 1994. Overview of the F-22 Program. Presentation by to the Committee on the Study of Live Fire Survivability Testing of the F-22 Aircraft, National Academy of Sciences, Washington, D.C., December 21.