|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
|
| ||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 31
3
Practicality' Affordability, and Cost-Benefit
This chapter presents Me committee's views on the practicality of full-up,
full-scale testing for the F-22. It also discusses affordability and cost-benefit
aspects of conducting and not conducting these tests. Finally, the committee's
opinion on the desirability of the F-22 waiver is expressed in the context of its
conclusions on practicality, affordability, and cost-benefit.
PRACTICALITY
The committee defines an activity as "practical" if meaningful results can be
achieved with existing technology and reasonably available resources. The issue
facing the committee is whether it is practical to conduct fi~-up, full-scale, live
fire tests of the F-22. Several considerations were taken into account in addressing
this issue; they serve to organize the discussion of practicality that follows.
Relative Importance of Vulnerability Reduction to F-22 Survivability
The live fire test law's definition of "realistic survivability testing" states that
"the primary emphasis [is] on testing vulnerability with respect to potential user
casualties arid taking into equal consideration the susceptibility to attack and
combat performance of the system" (10 U.S.C. 2366~. Judgments about the
practicality of full-up, fi~-scale tests for the F-22 must therefore take into account
its susceptibility and combat performance.
The probability of any system surviving hostile action is usually expressed
as the difference between unity (a perfect return rate if hostile action is
ineffective) and the product of the probabilities representing the effectiveness of
hostile action. In other words, the probability of survival is defined as
~-PD(P~(P~), where PD is the probability of being detected; PH,D is the
probability of being hit by the enemy's weapon, once detected; and PIC,H is the
probability of the friendly system being rendered permanently and completely
ineffective, once hit. It is clear that the lower the value of any one of these three
31
OCR for page 32
32
Live Fire Testing of the F-22
probabilities PD' PHID, Or PIC,H with the other two held constant, the greater the
friendly system's chance of survival.
The last of these probabilities, PIC/H, is the metric influenced by reducing
vulnerability and dealt with in planning and executing live fire tests.
As indicated earlier in this report, the F-22 design has been optimized for its
primary mission of offensive courter air. The aircraft includes both offensive and
defensive capabilities that are expected to result in destruction of enemy aircraft
beyond visual range before Me enemy aircraft can locate the F-22 arid fire a
missile. Only a very small proportion of encounters are expected to result in
closure to ranges at which enemy missiles or guns could be fired at the F-22. The
F-22 has been designed with a high decree of maneuverability to deal with the
occasional close engagement.
Accordingly, F-22 survivability depends more on low values of susceptibility
(i.e., the product of PD and PHID) and less on low values of vulnerability (PK/H)
Any option to reduce F-22 vulnerability further must be evaluated for its effect on
combat performance (e.g., Tow susceptibility), with priority generally going to
preserving the performance.
Of course, the priority on Tow susceptibility does not Rae out design changes
to address a critical wInerability discovered in the live fire test program and
attendant analysis and other testing. The issue is not whether a live fire test
program is required all agree that it is. The issue is how far to go with the live
fire test program. For the F-22, this judgment must be influenced in part by the
relatively low weight the Air Force has given to vulnerability in the overall
survivability equation.
Chapter 2 cited indications that the F-22 Will also have an air-to-surface
mission in the future. As currently envisioned, that mission we involve delivery
of munitions from a relatively high altitude. An a~r-to-surface mission could affect
the terms in the survivability equation. Specifically, the vulnerability term might
assume greater significance because of increased exposure of the F-22 to surface-
to-air defenses. Future missions for the F-22 will require Mat the Air Force
reassess the relative importance of vulnerability and susceptibility arid adjust the
wInerability assessment program accordingly.
c:7__ ~
~ High-performance fighters result from a highly integrated and carefully balanced
optimization of a substantial variety of systems. Since survivability is influenced by all three terms
(PD' PM/D, and PK/H)' the aircraft designer should not make changes to any one of them without
considering its effects on the other two. For example, if it were possible to reduce PK1H by armor
plating an avionics bay, but in so doing the armor increased the aircraft's radar reflectivity, raising
its PD, and if the weight of the armor significantly reduced the aircraft's maneuverability, raising
its PH,D, then the improvement in PK,H could be ill advised. This kind of one-dimensional
improvement activity involves the dangers of suboptimization.
OCR for page 33
Practicality, Affordability, and Cost-Benefit
33
No consideration of vulnerability reduction can be definitive without being
specific about the threat, which can influence all three terms in the survivability
equation. The next section discusses threat realism, and Chapter 4 describes the
threat environment projected for the F-22 and how that environment is reflected
in the vulnerability assessment program.
Realism in Aircraft Testing
The committee believes that live fire testing can be conducted at four levels
of aircraft readiness for the intended mission. These levels, which reflect various
degrees of realism, are discussed below. Even the highest level does not adequately
simulate actualflight conditions.
Level i'. The first and lowest level is live fire testing of hardware simulations
or mock-ups of production systems. For example, the hydraulic elements of a
flight control system might be assembled in the proper filll-scale geometry and
provided with flight-condition operating pressures, but mounted on a working
representation of an airframe.
The committee believes that such tests are
encompassed by the waiver Drovision of the live fire test law resee Ann~nr1i~r R
Section 2366(c)(2)].
~Lo err ^ ~
[eve! 2. The second level involves fi~-scale aircraft components,
subsystems, or subassemblies representative of production items. Depending on the
purpose of the test, all hardware that would be present in the production
configuration may or may not be present in the component, subsystem, or
subassembly that is tested. The committee considers these to be the tests referred
to in the waiver provision of the live fire test law "Section 2366(c)~2~.
Level 3. The third level of live fire testing involves a complete, production
aircraft not loaded with live ordnance or fuel. Selected systems may or may not
be operating. The corrunittee considers this to be inert (not Let-up), full-scale
testing. A waiver would be needed if only this type of full-scale system testing
were planned.
[eve! 4. The fourth and highest level utilizes a complete production aircraft
with all systems, fuel, and live ordnance installed and in operation typical of
combat. A fi~-up, fiull-scale test can be considered a verification (based on a
random sample) of the results of the lower level tests. Tests on the aircraft would
be conducted in ground-test facilities where test conditions can be closely
controlled and flight conditions can be approximately simulated. Win currently
available facilities, it is impossible to generate the air flows and actual design
stress levels encountered in flight. Parts of the aircraft can be bathed in high-speed
subsonic air, and some lower states of stress can be simulated. Tests under these
conditions would, in the opinion of the co~runittee, come the closest to meeting
OCR for page 34
34
practically the congressional intent
for fi~-up, full-scale testing.
It is desirable, of course, to
make any test as realistic as rea-
sonable. In live fire tests of
aircraft, realism can be thought of
as having three independent
aspects, as discussed below: the
threat, the configuration of the
friendly system, and the opera-
tional status of the friendly system.
If cost were not a consideration,
complete realism, or something
close to it, might be achieved in
all three aspects. But since costs
are in fact a determinative con-
straint, it is important to consider
the cost of approaching realism in
any one aspect in the light of
realism achievable in Me other
two.
Live Fire Testing of the F-22
. . . ...,. . , Ad. I,~t , , . . ,[~.. flu tu.~ .: ~.~ Ql .~ o:l -I . . .
· : 'ci?~PmRii.~:~.~:~'n~i:iA2h~': :: : . ' ~
arms Blat theme law apes
pus in co~ecti i i se of
. . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
'., :,, "w,2,or.'.d, ."subasse"~b,'lms." ' 'it' ' ' ' s"" ' "' ' e' :::,
: . a. w~.$, 8;
. ~lage . e~.. ~e :.i
..: sub~se~iies : be of e : :.
.............. . . . . . . : .: . .
· ~ompos ~t t ade - I: by:
, '~,','.'i''=,'~,]lit,> ^ Is'"
. cond ct~. ~i s ~ es ~ ~e
....co mi .ee...uses....te~. s like 'r.l ge ~] s,":
. : the "m-yor ~=s" ~ ' ~ s ::
· ............. . ................ .
..... subseries " ese : e ssi erel::
. . bledhi :
. ~ ~
:. ..~sembliesmeds0~bel~ge.~ugh~ ::
... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... . . . . . . . . . . . . . . . . . . . . . . . . . ..... . .. . . . . . . . . .
· ^ - ......... ............... . ,. .. · : :
. sutt~cle.~t .y . .represe~itat~ve : ~ a pace U;~1:~3n
.. , ......... . . . . : :: : -. ~
confi=r~or~ to meet He i~e fire te~
... . . = .. .. . .. .. .... ..
Threat realism involves a
wide range of variables (e.g., if the
threat is explosive, variables include the distance from its target upon detonation,
the kinematics of the intercept, and the characteristics of the blast generated).
Because testing the virtually infinite number of possibilities is obviously out of the
question, definition of the threat involves the assignment of probabilities to all of
these characteristics of weapon-target relationships. The indeterminable features
of threat definition are at the root of the statistical nature of the analysis needed
to interpret live fire test results. In complex ways, such considerations must enter
into the determination of Pa,, and influence measured or computed values of Pan.
NO
Achieving realism in the aircraft s configuration is straightforward, but it
can be expensive. The possibility of discovering an unexpected interaction between
systems argues that everything to be catTied on a mission be in place during live
fire testing. This is the fulI-up aspect, which includes feel, ammunition, and
hydraulic fluid. In addition to correct size, shape, and hardness, the fi~-scale
aspect requires that all components be installed (e.g., wire bundles and batteries).
Although the aircraft's configuration when hit by hostile fire cannot be predicted,
it will usually be possible to predict the most vulnerable condition (e.g., live
ordnance and filets. There is little that cannot be determined about the friendly
aircraft's configuration for live fire testing.
OCR for page 35
Practicality, Affordability, and Cost-Benefit
35
Achieving realism In the aircraft's operating conditions, however, is another
matter. Some operational circumstances can be replicated in live fire tests at
reasonable cost, but others cannot. For example, hydraulic systems can be
pressurized and engines can be running in ground tests. On the other hand, the
combined effects of high-speed airstreams and maneuvers may make the results
of ground tests a poor indicator of what wait haDnen under dynamic Fight
conditions (e.g., a high-g turning pull-up).
The variability introduced by indeterminable aspects of the threat and
operating conditions leads to one of the most important considerations in achieving
realism, namely, the concept of statistical significance. Statistical significance
quantifies the confidence that one can place in the test results obtained. This
confidence is a function of the variability of the data and the size of the sample
set, which is the number of trials that can be conducted in the case of live fire
testing.2 It is important to be able to support adequate sample sizes to achieve
reasonable statistical significance across all levels of testing. Vulnerability
assessment models and simulations provide a way of achieving relatively large
sample sizes at relatively Tow cost. However, current models and simulations have
their own sets of problems, which are discussed later (see Chapter 51.
Taking all of the above into account, the committee believes there is no
feasible way known to test the vulnerability of a first-line fighter like the F-22 in
a fully realistic way, that is, as defined by law. It might be possible to conduct a
one-or-few-trials flight test with drone aircraft. But the variability associated with
threats and operational conditions would require tens of trials for meaningful
results. Although large numbers of trials in the fi~-up, full-scale configuration
could conceivably be feasible for obsolete fighters, Hey are not practical for an
expensive new operational system such as He F-22. The difficulty (in some cases,
inability) of conducting realistic tests must be considered when planning live fire
test programs for the F-22 arid evaluating their results.
· _~] ^ ~_^,_ ~_} I,,,_ ^--~-~
2 It is important to recognize, in particular, that the results of a single test do not provide
statistically significant information about low probability events, such as "unknown unknowns."
These events are not likely to occur in a single test (or even a few tests). For example, if an event
has a 10 percent probability of occurrence, and one wants 95 percent confidence that it will be
observed in the test sequence, then the number of test trials needed is about 28 (NRC, 1993~. With
half the number of tests, the confidence of observing the event drops to 75 percent. With one-
fourth the number of tests (i.e., 7), the confidence drops to SO percent.
OCR for page 36
36
Live Fire Testing of the F-22
Destructive Versus Nondestructive Testing
Destructive testings can be of great value in any developmental effort. Such
testing can be conducted at any of the levels discussed previously. However, by
itself, destructive testing of a complete aircraft at the full-scale level yields
extremely Tow confidence factors for probabilistic outcomes due to the small
number of trials possible. At Tower levels of testing (e.g., the level that involves
components), greater numbers of destructive trials become possible. On the other
hand, some aspects of realism (e.g., the synergistic effects associated with a full-
up, full-scare test program) may be lost at these levels. Even a full-up, fulI-scaTe
destructive test would not in itself provide meaningfully representative
information.
Nondestructive tests can also be conducted at all levels with varying degrees
of realism. These tests can provide a sufficient number of trials for reasonable
confidence at reasonable costs. For example, nondestructive and repeatable
fi~-scale tests with nonflammable, nontoxic materials may in certain
circumstances provide useful data.
The committee believes that there is a major difference between destructive
and nondestructive testing of the F-22. When nondestructive tests can be designed
(e.g., with high-power microwaves4) it appears feasible to test the filll-scale
aircraft in sufficient trials to make a mearungfu} assessment of vulnerability. The
committee supports such testing.
The committee's support does not extend to full-up testing of the F-22 in
situations that might detonate any live ordnance or fuel on board. Provisions to
bypass the destructive features would, of course, make the tests less than Let-up.
Expert Opinion
Finally, the opinion of most experts with whom the committee discussed the
~ .. ... ~ · · · .~ . ~ ~ ~ , . - ~ 1 . _ · _ ~
matter, and the committee7s opinion, is that tull-scale testing or a complete a~rcran
is much less likely to provide useful information than are appropriate component,
subsystem, and subassembly tests. This opinion is based on Free factors:
3 Destructive testing is commonly understood to be the opposite of nondestructive testing,
which is an approach to testing that does not involve damage or destruction of the test sample. By
its very nature, live fire testing causes damage to the material or component being tested. In fact,
the extent of damage is one of the key results of a live fire test. The committee adopts the common
understanding discussed here in its use of the terms "destructive" and "nondestructive' testing.
4 The committee notes that high-power microwaves can cause some damage to components,
so even these kinds of tests may not be considered truly nondestructive.
OCR for page 37
Practicality, Affordability, and Cost-Benefit
37
A full-scale test with a threat weapon sufficiently large to affect the
whole aircraft may not be repeatable over a reasonable range of
conditions urdess many aircraft samples are used. The committee
judges that the benefits of such an approach would not be worm the
costs for an aircraft like the F-22.
If a munition with a local effect is used for the test, then a component,
subsystem, or subassembly can be used as the test specimen. A buildup
test sequence could be used that (a) starts at the component level,
where the greatest number of trials are possible; and (b) moves to the
subsystem and then the subassembly or large assembly level, as
appropriate. This approach would permit sufficient trials for
developing and confirming (or disproving) various hypotheses about
what darnage might or might not occur at the different levels. The
collective wisdom (including hard data) that is established by this
process provides reasonable confidence in the results.
There is a need to understand damage and failure mechanisms at the
component, subsystem, and subassembly levels.
The committee was briefed by at least one individual (O'Bryon, 1994) wh
stated his strong belief that testing at the full-scale level is necessary, using
production-authentic hardware and systems, but without live ordnance aboard. He
is correct that the current F-22 live fire program does not go this far.
Specific arguments raised in the above individuaT's presentation for
accomplishing fi~-scale (if not full-up) testing of the F-22 were essentially
(a) there are secondary effects (ricochet, debris, spelling, etc.) that do not reveal
themselves in smaller scale tests; (b) synergistic effects twhere damage to one
subsystem may cause damage elsewhere, sometimes called cascading effects (see
Chapter 4~] should be determined by full-scale tests; (c) system degradation should
be measured as the result of such tests; (~) battle-damage repair insights should
be encouraged; (e) fire starting mechanisms should be observed. and fire
O A, _ _ ~
SUppreSSlOn Should be evaluated; Ed (~) '~own ~o~s" can occur during
full-scare tests. These arguments appeared to represent not only this individual's
views but also the views of others who advocate fi~-up, fi~-scale testing.
The committee carefully weighed the array of arguments on bow sides of the
fi~-scaTe testing issue. The committee was persuaded that, for a system like the
F-22, considering its mission and system characteristics, a step-by-step approach
to vulnerability assessment was best. A test plan that dictates a methodical buildup
of tests from the component level, to the subsystem level, to the subassembly
level, to the large assembly level, and, if required, to the full-scale level made
more sense to the committee than assuming at the outset that fi~-scale tests were
necessary in every case. Additionally, while each of the enumerated benefits of
OCR for page 38
38
Live Fire Testing of the F-22
full-scale testing is important, in the opinion of the committee arid others who
briefed the committee, most can be achieved without resorting to full-scale testing.
The philosophical issue involved is that tests must proceed to a level that,
when taken collectively with the results of all the lower level tests, produces a
reasonable likelihood of revealing all necessary information. To some, that dictates
full-scale testing, others disagree. It is true that fi~-scale tests have a potential for
disclosing surprises that no one could predict through modeling, analysis, or
component-to-subassembly testing. However, for the F-22, especially in light of
its very high cost, a test without a reasonable expectation of additional valuable
data to be derived is unwarranted. The committee did not discern a reasonable
expectation of deriving valuable data (e.g., ''unknown unknowns") from a
full-scale test of the F-22. If it had, the committee would have had no qualms
about recommending such testing.
The committee concluded, in light of its own expert judgment and the
prevailing opinion of most others with whom the committee met, that full-scale
testing of an aircraft like the F-22 is not justified. However, the committee does
agree that testing of F-22 subassemblies in a complete or nearly complete
production configuration is justified in appropriate circumstances, as discussed in
Chapter 4.
AFFORDABILITY
The committee defines an "affordable" activity as one within budgetary
constraints or attainable budgets. Like all acquisition programs, the prioritization
of alternatives within the F-22 program is based on an assessment of their
marginal utility. Unaffordable budget items remain unended until either more
funding is authorized or the priorities of activities within the program are
reassessed (e.g., previously unaffordable items displace funded items based on
analyses Mat reassess their relative benefits).
The committee was not able to consider Filly or challenge the prioritization
of items within the F-22 program budget. However, based on the committee's
judgment of benefits versus costs, even if an aircraft could be provided, the full-
up, fi~-scale tests would not be recommended. As a result, the overall program
budget and prioritization of expenditures within that budget, as established by the
F-22 SPO, was accepted by He committee. Clearly, in the SPO director's
estimation, full-up, full-scale testing exceeds the available budget (Raggio, 1994~.
OCR for page 39
Practicality, Affordability, anc! Cost-Benefit
Affordability of Full-Up, Full-Scale Testing
39
According to data provided by the SPO as of February ~ 995, the cost of the
currently planned F-22 live fire tests is slightly over $38 million (then-year
dollars). This amount funds multiple component, subsystem, and subassembly
tests. The SPO projects that a full-up, full-scale test of the F-22 would cost an
additional $250 million (then-year dollars) above the currently planned program.
The major component of this amount is purchase of a full-up production aircraft.5
The underlying assumption is that this test asset would be devoted fully to the
tests and could not reasonably be refurbished to have military utility after
completion of the tests.
The committee requested information to support the contention that a
production aircraft would be required in lieu of some less realistic alternative with
somewhat less test fidelity (e.g., the prototype test aircraft). The SPO responded
that use of three alternatives had been examined: (~) the prototype air vehicle, (2)
the static ground test article, and (3) an engineering and manufacturing
development flight test aircraft (Graves, 1995~. None ofthese was deemed feasible
in the judgment of the SPO. The SPO's estimate that the cost of a full-up, full-
scale test would be on the order of $250 million results directly from this
judgment. The committee has no basis for refuting the SPO's judgment.
There is an argument that $250 million is only a small percentage (less than
0.5 percent) of total F-22 program costs, and therefore cost should not be a
determining factor. The committee understands this view. Nonetheless, the
committee's judgment is that the benefits of fi~-up, full-scale tests are not
commensurate with the costs. Even if $250 million were provided for additional
vulnerability assessment of the F-22, the committee would not support using the
fiends for full-up, full-scale testing.
Both Air Force and Navy experts on aircraft vulnerability assessment
indicated that, if they were provided a new production aircraft for live fire testing,
they would prefer to disassemble it arid perform live fire testing on a less than
filll-up, full-scale configuration. They stated that Weir preference to test at the
component, subsystem, and subassembly levels was derived from their experience
that they would learn more about vuinerahiiitv and could mace orbiter ~onfirl~nr~.
ire the results, for the resources expended.6
~, ~ ~~~~~ rat
s The fill-up aircraft represents well over 90 percent of the $250 million cost estimate
(Graves, 1995~.
6 The opinions in this paragraph were expressed during discussions with members of the
committee on February 21, 1995, at the Naval Air Warfare Center, China Lake, California.
OCR for page 40
40
Live Fire Testing of the F-22
Investment Methodology for F-22 Vulnerability Tests
The committee was briefed by the SPO on an investment methodology that
examined incremental live fire testing of the F-22 (Griffis and La~z:~e, 1 995~. The
committee was not persuaded by the SPO's analysis. The committee believes that
the SPO's investment mode} offers a limited context from which to reject,
analytically, additional increments of live fire testing. Analytic approaches based
on incorrect premises have a way of producing logically derived results that may
well be wrong. Additional comments on cost-benefit methodology appear below.
COST-BENEFIT METHODOLOGY
The committee received briefings on the current state of the art of cost-
benefit analysis of live fire testing (Griffins and Lauzze, 1995; Klopcic, 19951.
Attempts are under way within DoD to develop a methodology for determining
the return on investment of successive levels of live fire tests by predicting the
costs of fixture aircraft attrition. If the cost of an additional live fire test is known
(it is), and one is reasonably sure of the incremental reduced cost of attrition that
a test would bring about (one is not), the return on investment would be clear and
the additional test could be judged on that basis.
The methodologies presented are immature at this time. They appear to
address suboptimal measures of benefit. Despite these shortcomings, the committee
believes that a validated, useful methodology for determining, quantitatively, the
cost-benefit relationships of live fire testing would be a valuable too! for
vulnerability assessment. This kind of too! would help prioritize the various levels
of live fire testing together with other competing program activities.
The committee acknowledges that validating a usefill cost-benefit framework
is, in fact, extremely difficult because of the need to establish a priori benefit
valuations. There is a danger that a framework could produce misleading results
if the benefit measures chosen are so narrow as to preclude interesting alternatives.
The committee suggests that this risk be minimized by conducting a series of
excursions that assess proposed test programs in the light of alternative measures
as well as widely varying but conceivable test results. In addition, it might be
useful to consider military scenarios that would elevate the importance of reduced
vuInerability7 arid, thereby, possibly enhance the benefit-to-cost ratio of given
levels of testing.
7 Here, the concept of reduced vulnerability could extend to fewer losses of flight crews (i.e.,
capture or death) and smaller likelihood that U.S. systems would be recovered and exploited by
the enemy.
OCR for page 41
Practicality, Affordability, and Cost-Benefit
41
Additional live fire testing of the F-22 at any level should be examined in
terms of opportunities for information to be gained, test costs, and conceivable
consequences of not performing the tests. Such testing should not be rejected
because of cost-benefit scores in a single, and perhaps overly simple, construct of
how the future may develop.
For example, assume that (a) the United States will be
faced with a series
of wars like Desert Storm extending through the first half of the twenty-first
century, and (b) the F-22 wait confront significantly improved defensive systems,
wall be heavily used, and will be at risk in each sortie it flies. In the face of these
assumptions, the total number of combat sorties flown is the key measure.
Reduced vulnerability and increased capability to repair battle-damaged aircraft
may become determinative. Under these conditions, additional investment in live
fire testing could be attractive, particularly if it might reveal ways to produce a
significant increase in the number of combat sorties flown (e.g., through increased
tolerance to damage sustained in combat arid the development of techniques for
repairing damaged aircraft in the field).
CONCLUSIONS
Having weighed the arguments on both sides of the full-scale testing issue,
the committee was persuaded that, for a system like the F-22, a well conceived,
incremental build-up of tests that proceed from the component level to the
subassembly or large assembly levels made the most sense. The committee did not
discern a reasonable expectation of deriving additional valuable data (e.g.,
"unknown unknowns") from a filll-scale test of the F-22.
The committee concludes that completely realistic, destructive, full-up, full-
scale testing of the F-22 is not practical and entails high costs relative to the
resulting benefits. The judgment of most members of the live fire test community
with whom the committee met is that incremental testing to the point of relatively
large subassemblies is the way to proceed and Mat full-up, full-scale tests of
combat-configured aircraft are of marginal utility. The committee agrees with this
judgment. The combination of the lack of realism in test conditions, the difficulty
of obtaining a sufficient number of trials, and expert opinion all support this
conclusion.
Full-up, full-scale testing in a configuration that could destroy the entire
aircraft if a detonation occurs is therefore not warranted for the F-22. On the other
hand, nondestructive testing (e.g. with high-power microwaves) is practical, as is
destructive testing of components, subsystems, and subassemblies. The committee
s The F-22's new composite materials and new systems could require battle-damage-repair
techniques that are much different from those used on current fighters.
OCR for page 42
42
Live Fire Testing of the F-22
endorses the use of production-representative articles for these tests. Also, it is
practical to test parts of an aircraft under simulated loads arid exposure to high-
velocity airflows.
Regarding the affordability of the tests, the committee has concluded that
affordability is not the matter of foremost relevance. Cost-benefit is most relevant.
Affordability only becomes relevant if the benefits relative to the costs of whatever
tests are being considered are commensurate with the benefits relative to the costs
of other alternatives. With respect to full-up, full-scare tests for the F-22, the
committee judges the benefits to not be worth the costs.
Based on its conclusions concerning the impracticality and low benefits for
the costs of fi~-up, full-scale, live fire testing for the F-22, the committee is
unanimous in its opinion that a waiver is the appropriate course of action for the
F-22. It must be pointed out, however, that while the committee was asked to
examine practicality, affordability, and cost-benefit, the law states that a waiver
may be granted by the Secretary of Defense based on a certification "that live-fire
testing . . . would be unreasonably expensive and impractical "emphasis added]."
The committee's interpretation is that both conditions must be true before a waiver
can be granted. Regarding the term ''unreasonably expensive," the committee
believes that the Tow benefits relative to costs (i.e., high costs relative to benefits)
means that the tests are unreasonably expensive.
If a waiver is granted, there needs to be some measure of sufficiency for the
test program that is conducted. The sufficiency of live fire tests currently planned
for the F-22 is addressed in the next chapter.
Finally, the committee recognizes that its judgments regarding the costs arid
benefits of full-up, full-scale testing were reached in the absence of a mature
methodology for assessing benefits relative to costs. The committee is leery of
reliance on cost-benefit methodologies Mat use an overly simple construct of the
F-22's fixture to make judgments about how far to go with live fire testing. A
broader ar~alytical framework could elevate the importance of reduced F-22
vulnerability over the long haul and might enhance the benefit-to-cost comparisons
of given levels of testing.
OCR for page 43
Practicality, Affordability, and Cost-Benefit
REFERENCES
43
Graves, I.T. 1995. National Research Council Questions on Live Fire Test.
Memorandum from Deputy Director, F-22 System Program Office, to
National Research Council, Mike Clarke, February 14.
Griffis, H., and R. Mae. ~ 995. Cost Benefit Analysis Methodology. Presentation
to the Committee on the Study of Live Fire Survivability Testing of the F-22
Aircraft, Dayton, Ohio, January 20.
Klopcic, I.T. 1995. Knowledge-Based Benefit/Cost Methodology for Live Fire
Test Evaluation. Presentation to the Committee on the Study of Live Fire
Survivability Testing of the F-22 Aircraft, National Academy of Sciences,
Washington, D.C., February 16.
NRC (National Research Council). ~ 993. Vulnerability Assessment of Aircraft: A
Review of the Department of Defense Live Fire Test and Evaluation
Program. Air Force Studies Board, NRC. Washington, D.C.: National
Academy Press.
O'Bryon, I.F. 1994. Discussion of Live Fire Testing Philosophy and the History
Associated with First Report. Presentation to the Committee on He Study of
Live Fire Survivability Testing of the F-22 Aircraft, National Academy of
Sciences, Washington, D.C., December 21.
Raggio, R.F. 1994. Overview of the F-22 Program. Presentation by to the
Committee on the Study of Live Fire Survivability Testing of the F-22
Aircraft, National Academy of Sciences, Washington, D.C., December 21.
Representative terms from entire chapter:
fire testing
