Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program (1993)

Aircraft vulnerability refers to the inability of the aircraft to withstand the damage caused by one or more hits by the damage mechanisms associated with air defense weapons, such as warhead fragments and blast. Vulnerability assessment or evaluation is a part of every U.S. military aircraft acquisition program. The assessment can be accomplished by using two methodologies: (1) analyses or computer models that simulate the various reactions of the aircraft and its components to the hits, and (2) live fire testing.

The Live Fire Test Law. As a result of the controversy over the vulnerability testing of the U.S. Army’s Bradley Fighting Vehicle, Congress passed a law in fiscal year (FY) 1987, known as the Live Fire Test (LFT) law, which mandates realistic survivability and lethality testing of covered systems or programs. The law was modified in 1988. In the current version of the law, realistic survivability testing is defined as “testing for vulnerability of the system in combat by firing munitions likely to be encountered in combat (or munitions with a capability similar to such munitions) at the system, configured for combat, with the primary emphasis on testing vulnerability with respect to potential user casualties and taking into equal consideration the susceptibility to attack and combat performance of the system.” (Note that survivability is used when vulnerability is intended.)

According to the LFT Guidelines in the law, the live fire tests are to be carried out sufficiently early in the development phase of the system to allow any design deficiency demonstrated by the testing to be corrected in the design of the system before proceeding beyond low-rate initial production (LRIP). The system acquisition program cannot proceed beyond LRIP until the testing is completed. The FY 1988–1989 Department of Defense Authorization Act Conference Report states that Congress intended that the Secretary of Defense implement the LFT law “in a manner which encourages the conduct of full-up

vulnerability and lethality tests under realistic combat conditions, first at the sub-scale level as they are developed, and later at the full-scale level mandated in the legislation.”

A waiver from the requirement for realistic survivability testing is allowed if the Secretary of Defense notifies Congress, prior to entry into full-scale engineering development, that live fire testing of the system would be unreasonably expensive and impractical. The notification of the waiver must be accompanied by a report describing an alternate program for evaluating the survivability of the system and assessing possible alternatives to realistic survivability testing of the system.

When the Live Fire Test law was passed, the position of Director, Live Fire Testing, was established under the Office of the Deputy Director, Defense Research and Engineering (Test and Evaluation) [DDDR&E(T&E)] and was given the responsibility for implementing the LFT legislation. The tests specifically associated with the congressionally mandated Live Fire Test and Evaluation (LFT&E) program are referred to herein as Live Fire Tests, whereas other tests using live ammunition that are not specifically part of the LFT&E program are referred to as live fire tests.

Study Tasks. The tasks assigned to this committee were (1) to review and evaluate the current vulnerability assessment methodologies for aircraft, including both analysis/modeling and live fire testing; (2) to review and evaluate the current direction of the congressionally mandated Live Fire Test programs within OSD and the Services; and (3) to recommend changes to these methodologies and programs, if appropriate. The committee was instructed to consider all aspects of the vulnerability assessment methodologies, programs, and the Live Fire Test legislation.

The committee met four times between July 1991 and April 1992 and received presentations from personnel from the Office of the Secretary of Defense (OSD) LFT&E Office, the DDDR&E(T&E)/Tactical Weapons Programs Office, the U.S. Army’s Test and Evaluation Management Agency and Comanche Program Office, the U.S. Navy’s Survivability Branch, Naval Air Systems Command, the U.S. Air Force’s Test and Evaluation Office and F-22 and C-17 System Program Offices, and the live fire test organizations of the three Services, as well as vulnerability assessment experts from four U.S. aircraft companies and the Institute for Defense Analyses. Several committee members also interviewed Mr. Joseph Cirincione, the congressional staff member who drafted the LFT legislation in 1987.

The study was divided into six parts: (1) review of the analysis/modeling and live fire testing assessment methodologies and identification of the applications of the results of the assessments; (2) evaluation of the cost, effectiveness, and deficiencies of the two methodologies; (3) review and evaluation of the OSD LFT&E program; (4) review and evaluation of the LFT&E programs of the three Services; (5) examination of the future of vulnerability assessment of aircraft; and (6) conclusions and recommendations.

Methodology Review and Applications of the Results. The study reviewed both of the methodologies used by the Army, Navy, and Air Force to determine the vulnerability of airborne systems to guns and guided missiles, and identified the applications of the results. The weapons considered include the nonexplosive armor-piercing (AP) penetrator or fragment, the contact-fuzed high-explosive (HE) warhead, and the proximity-fuzed externally detonating HE warhead. The target aircraft is either full-scale (the complete system) or sub-scale (a partial system consisting of one or more components and/or subsystems) and inert (no combustibles) or full-up (with combustibles).

The committee identified six applications for the results from analysis/modeling and live fire testing. They are (1) to aid in the design and design validation, (2) to satisfy the

vulnerability assessment requirements contained in DoD MIL-STD 2069,¹ (3) to develop data bases in support of subsequent analytical assessments, (4) to predict test outcomes, (5) to satisfy the requirements of the Live Fire Test law, and (6) to support acquisition decisions.

Cost, Effectiveness, and Deficiencies of the Methodologies. The committee notes that the primary objectives and applications of the two methodologies are, in general, different. Analysis and modeling are primarily used to aid in design and to quantify vulnerability, whereas live fire testing is conducted to gain insight into the major types of physical damage, including cascading and synergistic effects; to develop component vulnerability data bases; to aid in design; and to validate the design and the model, when appropriate.

The committee selected the information or results provided by the methodologies as the basis for comparison with respect to cost, effectiveness, and deficiencies. In particular, the information attributes of (1) type, amount, and applications; (2) accuracy, or level of confidence; and (3) cost were evaluated for both analysis/modeling and testing. The testing methodology was divided into two categories, tests on sub-scale targets and tests on full-scale targets.

In general, the results from analyses and models consist of numerical values of vulnerability for the components and the aircraft for all weapons and all threat directions. These results can be used in all six applications listed above. The overall level of confidence in the analytical results is relatively low because of the inadequate modeling and supporting data base of some damage processes and vulnerabilities, and the omission of others. The cost of analysis/modeling is also relatively low.

The live fire tests on both sub-scale and full-scale targets produce information on what actually happened for a particular set of test conditions but only for a relatively small number of shots under these conditions. The applications of the test results are to aid in design and design validation, to develop vulnerability data bases, to satisfy the Live Fire Test law, and to support acquisition decisions. The level of confidence in the results is relatively high, and so is the cost, particularly for the full-scale aircraft tests. However, using other full-scale test articles, such as prototypes, for the full-scale vulnerability testing can significantly reduce the cost. Any Live Fire Tests on prototypes must be carefully audited for their applicability to production articles since the differences between the two may be large.

One of the most important findings of this study is that on-board ordnance has been neglected in both analyses and live fire testing as a contributor to vulnerability. One of the basic requirements of the Live Fire Test program is to test full-scale vehicles with the full load of on-board ordnance. External ordnance may shield components from projectiles and fragments, or it may react violently to a ballistic impact, possibly destroying the aircraft. Adverse reactions of any internally carried ordnance have an even greater probability of destroying the aircraft.

In general, the committee believes that the combination of analytical models, supported by live fire tests on components and subsystems, and the full-scale Live Fire Tests are mutually compatible in the vulnerability assessment and design of aircraft. They complement each other, and the whole is superior to the sum of the parts. However, more work is needed to unify these approaches in order to obtain the maximum benefit.

Review of the Requirements of the Live Fire Test Law. The committee notes that there is a major controversy among the various participants in the DoD Live Fire Test program regarding the law’s requirements. One point of contention is the requirement for testing the system. One interpretation is that the law does not explicitly state that a complete system must be tested; hence the law is satisfied by an LFT program only on components and

subsystems. Another interpretation is that the word “system” refers to the complete or full-scale system. Based upon the evidence gathered by the committee and its study of the law, the committee is unanimous in the opinion that the LFT law requires a full-scale, full-up aircraft to be tested, unless a waiver is granted. The committee bases its opinion upon the events that led to the law, the wording in the law, the accompanying discussion of the law in the FY 1988–1989 DoD Authorization Act Conference Report, the opinion of the congressional staff member who drafted the law, and the fact that a waiver is allowed. If full-scale, full-up tests were not required, no waiver would be necessary, and any live fire tests would suffice, provided they were realistic.

Review of the OSD Live Fire Test and Evaluation Program. Two OSD documents have been provided to the Services and their Program Managers to assist them in the planning and conduct of a Live Fire Test program: (1) the 1988 Live Fire Test and Evaluation Guidelines issued by the Test and Evaluation Committee, Office of the Secretary of Defense, and (2) the 1989 Live Fire Test and Evaluation Planning Guide issued by the Live Fire Test Office. The committee is concerned that the written guidance provided by the LFT&E Office does not provide sufficient detail, particularly with respect to the full-scale tests, to ensure that the Program Manager can satisfy the requirements of the OSD policy and the LFT law, and also design a cost-effective test plan that will ensure the system requirements are satisfied. The various definitions given in the 1988 LFT&E Guidelines have been interpreted by some to imply that full-scale, full-up Live Fire Tests do not have to be conducted, that is, the OSD LFT policy is satisfied by Live Fire Tests only on sub-scale targets, such as major portions or subassemblies of an aircraft. Furthermore, the committee is concerned about the official status, or lack thereof, of the LFT&E Guidelines and the Planning Guide.

Review of the Service Live Fire Test and Evaluation Programs. The committee reviewed the LFT&E policies and programs of the three Services, including the Army’s RAH-66, the Navy’s V-22 and A-12, and the Air Force’s F-22 and C-17. The committee believes it is important to point out the facts that, except for the RAH-66, all of these programs were well under way when the law was passed and the C-17 development had proceeded beyond the deadline for the application of a waiver. Furthermore, Congress did not provide transitional guidance for these programs or additional money to fund the tests.

The Navy and the Air Force have interpreted the 1988 LFT&E Guidelines to imply that full-scale, full-up tests are not required. Furthermore, the LFT&E policies presented to the committee by the Services do not consider such tests to be cost-effective, particularly if on-board ordnance is included. Consequently, they have not developed LFT&E programs that contain full-scale, full-up Live Fire Tests. However, both Services strongly support the conduct of sub-scale inert and full-up tests throughout the development process. The Army policy on LFT&E supports a “building-block” approach consisting of component testing through full-scale, full-up system testing that satisfies the Live Fire Test law. The emphasis of its LFT&E program is on sub-scale testing, with limited full-scale, full-up testing to confirm the results obtained from sub-scale testing. However, the Army LFT&E program for the RAH-66 did not contain firm plans for testing a full-scale, full-up helicopter. The full-scale testing was going to be conducted only if the sub-scale test results showed it to be necessary.

In addition to the controversy regarding the requirement for full-scale testing, there is a controversy regarding the munitions to be used in the Live Fire Tests. The specific munitions to be used for a particular aircraft are selected by the aircraft Program Office as part of its LFT&E program plan. Typically, the threats selected by the Program Office for Live Fire Testing are the threats the aircraft was designed to withstand, such as a single hit by an AP projectile or small-caliber HE round. The assumption is made that the more lethal overmatching threats, such as the larger gun projectiles and missiles, will be avoided and hence should

not be a part of the LFT&E program. The Live Fire Test Office has interpreted the phrase “munitions likely to be encountered in combat” to mean that those munitions the aircraft may encounter, including the latest directed energy weapons, should be included in the LFT&E program, regardless of the design threat for the aircraft.

The representatives from the live fire test organizations of the three Services and the vulnerability experts from industry appear to be in general agreement about the efficacy of the Live Fire Test law as it applies to aircraft. They do not consider it necessary to conduct full-scale, full-up tests in order to determine most of the design vulnerabilities. However, they do consider it essential to conduct many sub-scale live fire tests on components and sub-systems, both inert and full-up, in the development cycle of an aircraft. They believe the full-scale, full-up tests may be conducted too late in the development cycles to be of much value to the designer, and that the amount of information obtained from the tests is limited. The representatives do recognize the possibility of an unanticipated reaction, cascading damage, or synergism occurring in the full-scale aircraft. However, they believe that nearly all of the kill modes of an aircraft are known and can be anticipated.

The committee notes that not everyone who has observed live fire tests on sub-scale and full-scale test articles shares the views held by these testers and vulnerability experts. They believe that there have been unanticipated results from these tests. Furthermore, even when the response is as expected, the difference in the expected magnitude of the response and the observed magnitude often is too large to be acceptable. The testers have also overlooked the fact that the information from the full-scale tests is a valuable input to the acquisition decision makers at milestone reviews.

Perceptions of the LFT Law. The committee is aware of the strong differences of opinion held by various individuals and organizations concerning the efficacy of the Live Fire Test law and of the level of mutual distrust that has evolved as a result of these opinions. This distrust between the various participants of each other’s motives and actions is probably responsible for the ever-increasing tensions within the current Live Fire Test program. In the committee’s opinion, the attitudes of the major participants concerning the Live Fire Test law and its place in the acquisition process may be those described below.

In the committee’s opinion, the Program Manager (PM) may consider the full-scale, full-up testing mandated by the LFT law to be an unquantifiable, but potentially catastrophic, risk to his program. LFT has no quantitative contractual specifications or acceptance criteria at program initiation. No quantitative criteria for acceptable or unacceptable damage are included in the requirements process, milestone commitments, or contractor performance documents. LFT of the full-scale aircraft occurs late in the development phase of the program, and there may be neither adequate time nor money to conduct the tests or to make any changes required as a result of the tests. Further, the PM may believe that neither the definition of the tests, nor the conduct of the tests, nor the interpretation of the test results is totally under his control. The perceived jeopardy to his program created by LFT is exacerbated by the severe requirement to fit the program into a somewhat inflexible overall resource schedule, both in time and in dollars. In summary, LFT represents a considerable source of problems to the PM, in the form of an uncontrollable, potentially catastrophic uncertainty, as he attempts to successfully complete the development of a system, and should be avoided if at all possible.

In the committee’s opinion, the Services may believe that, as system developers and users, they know what is needed in the equipment they will take into the field, and that they, the Services, are directly responsible for the fate of the military personnel who use this equipment in combat. They are very apprehensive about any outside organization that can dilute their ability to define the necessary equipment testing and the procedures required to accomplish this testing. They appear to further believe that the Live Fire Test law gives to others not directly responsible for the delivered product inordinate control without any accompanying responsibility for the quality of the product or its cost.

In the committee’s opinion, the OSD may believe that there have been a sufficiently large number of prior experiences in the area of live fire testing to indicate that pressure by one of the Services for successful and rapid certification of its products under development can lead to inadequate live fire testing and to subsequent unnecessary combat vulnerabilities. The OSD, therefore, has chosen to exercise close control over the Live Fire Test programs and assumes the ultimate authority for approval of the equipment based on the program results.

In the committee’s opinion, the Congress may believe that there are sufficient proven instances of unnecessary combat vulnerability in DoD equipment previously delivered to the field to warrant legislative direction of DoD test certification to include live fire testing of full-scale, full-up systems using munitions likely to be encountered in combat. Congress further believes that it has the ultimate responsibility for the programs it authorizes and therefore has the obligation to exercise that legislative direction.

The committee believes that the current assessment procedure, which is supposed to result in an improved aircraft, does not guarantee that the U.S. armed forces will field cost-effective systems designed for reduced vulnerability. The intent of the LFT law to contribute to the creation of less vulnerable aircraft designs is valid; its execution to achieve this intent has been flawed. The committee believes that the problems with the LFT law are (1) ambiguities in the wording of the law’s requirements; (2) the lack of a clear and binding LFT policy directive; (3) the Services’ reluctance to ask for a waiver from full-scale, full-up LFT for those programs for which they believe LFT to be unreasonably expensive and impractical because of the fear of a stigma associated with the waiver; and (4) the absence of a formal waiver process that includes a procedure for identifying when the full-scale, full-up testing is or is not unreasonably expensive and impractical, which would eliminate any stigma associated with the waiver.

The Future of Vulnerability Assessment. The committee recognizes the limited prospects for both new program starts and product improvements as a result of the declining DoD budget. Although this new environment will lead to austere budgets for vulnerability assessment, the requirements for a vulnerability assessment of any particular system should not decline. However, the overall total requirements for vulnerability assessments will most likely decline due to the reduction in the number of active aircraft programs. In addition to individual program cost containment issues, the committee anticipates a reduction in both the analytical/ modeling and the test and evaluation infrastructure within OSD and each of the Services as the total DoD budget declines.

Three categories for cost reduction in vulnerability assessment while maintaining or improving the current capabilities were examined. The first consists of an increased reliance on analysis/modeling. The committee believes that there appears to be a sufficient start of a modeling capability, and of a weapons effects and materials data base, to warrant an increased dependence on analysis/modeling for future vulnerability assessments as an aid in design. However, the committee also believes that the current analytical methodology and supporting data bases are not yet sufficiently robust, correct, precise, and representative to permit a total dependence on this methodology. Much work needs to be accomplished in model development and in the accumulation of weapons effects and material vulnerability bases.

A reduction in assessment costs can be obtained by requesting a waiver from the full-scale tests. The major factor in the cost of vulnerability assessment is the requirement for the full-scale Live Fire Test program mandated by the Live Fire Test law. The law offers a waiver from the full-scale, full-up tests when they would be unreasonably expensive and impractical. Under the current LFT&E Guidelines and Planning Guide, there is no guidance as to what constitutes an unreasonably expensive and impractical Live Fire Test program. In the future, a procedure must be established for gathering the facts necessary to determine

if the full-scale, full-up Live Fire Tests are unreasonably expensive and impractical with respect to the critical vulnerability issues and, if they are, what other assessments should be conducted in place of the complete system tests.

A third possibility for cost reduction involves a consolidation of the vulnerability assessment infrastructure. Although the committee did not review in detail this aspect of the vulnerability assessment activities and capabilities in each of the three Services, the committee believes that until the recent DoD budget downturn, there were sufficient aircraft programs to warrant the continuation of more or less similar Service live fire test capabilities. However, in the decades ahead, the expected requirements for the vulnerability testing of new Service equipment will probably fall below the level at which a critical mass of broad-based facilities and knowledgeable staff can be maintained within any of the individual Services. Because considerable cost savings could be achieved by consolidating the capabilities of these facilities, some form of consolidation beyond that currently contemplated appears inevitable.

The future of vulnerability assessment will most likely involve one or more of these three categories.

After reviewing the vulnerability assessment methodologies; evaluating the cost, effectiveness, and deficiencies of these methodologies; and reviewing and evaluating the Live Fire Test law and the OSD and Service Live Fire Test & Evaluation programs, the committee has come to the following conclusions.

• Conclusions Regarding the Live Fire Test Law & DoD Programs

1. The committee believes that the requirements in the Live Fire Test law have been interpreted in several ways and that these different interpretations have caused confusion and tension in the Live Fire Test programs. Nevertheless, the committee believes that the law is a valuable contribution to vulnerability assessment and to the design of survivable aircraft. Furthermore, it is satisfactory in its present form because of the waiver process. The committee believes that the law has had a positive impact on the vulnerability design of aircraft and is sufficiently flexible, due to the waiver process, to apply to all aircraft. Furthermore, the committee believes that verification of vulnerability by live fire testing is necessary and that this law ensures that verification.

2. The committee believes that the 1987 congressional Live Fire Test law mandates live fire testing of full-scale, full-up aircraft, including on-board ordnance, unless a waiver is granted by the Secretary of Defense.

3. The committee believes that the 1988 Live Fire Test & Evaluation Guidelines and the 1989 Live Fire Test & Evaluation Planning Guide are not consistent with its interpretation of the LFT law.

4. Because all three Services apparently believe that an LFT&E program plan that contains only sub-scale testing is in compliance with the law as interpreted by the OSD 1988 LFT&E Guidelines, no current LFT program contains plans to conduct full-scale tests and no waivers have been requested. (The committee has been informed that on May 11, 1992, the Under Secretary certified to the Congress that live fire testing of the F/A-18E/F aircraft would be unreasonably expensive and impractical. The alternatives to the statutorily prescribed survivability testing are being prepared by the Navy.)

5. The committee believes that a waiver is required to omit the full-scale, full-up tests.

6. The committee believes that there are aircraft for which a full-scale, full-up test program is unreasonably expensive and impractical, and that there are aircraft for which a full-scale, full-up test program is neither unreasonably expensive nor impractical.

7. The committee believes there should be no stigma attached to a waiver because the waiver is an acceptable alternative LFT&E path.

8. A serious problem in both the analyses and the Joint Live Fire Testing of aircraft has been the omission of on-board ordnance as a critical component.

9. The stated intent of the system tests mandated by the Live Fire Test law is to aid in design by providing information on possible weaknesses sufficiently early in the design process to allow the weaknesses to be corrected.

10. The implied intent of the Live Fire Test law is to force the consideration of vulnerability during the design process.

11. The lack of a definition of the specific threat munitions to be used in design and in Live Fire Testing has resulted in considerable controversy regarding which threat weapons to use in the Services’ LFT programs.

12. The apparent separation of the oversight of vulnerability analysis from the oversight of live fire testing, both of which are part of the testing and evaluation process, has created a situation that is detrimental to the overall OSD vulnerability program.

• Conclusions Regarding the Vulnerability Assessment Methodologies

13. Based upon its review of the two methodologies, the committee concludes that both vulnerability analysis and live fire testing, including the mandated Live Fire Testing, are essential in a mix peculiar to each aircraft development program.

14. The committee believes that both methodologies need to be improved and that these improvements should be mutually beneficial.

• Conclusions Regarding the Vulnerability Programs for Aircraft

15. The vulnerability of currently fielded U.S. aircraft will become more important in the future.

16. There is insufficient attention given to the requirement to design for vulnerability.

17. The collection of actual combat data on the vulnerability of U.S. aircraft is not given proper emphasis.

• Conclusions Regarding the Vulnerability Infrastructure

18. The process of designing and testing for vulnerability is extremely complex and would benefit from continuous input and oversight from a broad range of experts in the vulnerability community.

19. The vulnerability community of the future most likely will become smaller in both the number of programs and the size of the infrastructure.

Based upon the results of the committee’s study and the conclusions given above, the committee makes the following recommendations:

• Recommendations Regarding the DoD Live Fire Test & Evaluation Program

1. The committee recommends that the Director, Test and Evaluation, issue Guidelines that replace the 1988 Live Fire Test & Evaluation Guidelines and that more clearly conform with the requirements for the full-scale, full-up tests mandated by the Live Fire Test law. The recommended directive should completely define the procedures and requirements for planning and conducting the LFT&E program for both sub-scale and full-scale tests. The directive should require the conduct of vulnerability tests under realistic combat conditions, first at the sub-scale level as sub-scale systems are developed, and later at the full-scale level mandated in the legislation. In addition, the directive should describe a formal process for requesting a waiver.

2. The committee recommends that the Director, Test and Evaluation, formalize the waiver process by developing a risk-benefit assessment methodology that can be used uniformly to determine whether a full-scale, full-up test program for any particular aircraft is “unreasonably expensive and impractical.” The methodology must also be applicable to the evaluation of the alternative Live Fire Test program for the sub-scale targets. The process for requesting a waiver, described in the DoD directive recommended above, should include a risk-benefit assessment methodology that quantifies the benefits

associated with full-scale, full-up Live Fire Tests and the risks associated with waiving these tests. Once the benefits and risks have been quantified, a decision can be made as to whether the full-scale, full-up tests are unreasonably expensive and impractical. The committee strongly believes that such a methodology would significantly improve the process of requesting a waiver.

1. The committee recommends that the Secretary of Defense take measures to ensure that (a) the LFT&E Guidelines are properly enforced by requiring either that covered systems be subjected to full-scale, full-up testing or that a waiver be obtained; (b) that any waiver be fully justified; (c) that the waiver process be uniformly applied; and (d) that no stigma be attached to the use of the waiver process.

2. The committee recommends, for the full-scale, full-up Live Fire Tests, that the specific “likely to be encountered” munitions referred to in the Live Fire Test law be the weapon(s) specified in the requirements documentation for the system, projected forward to the time when the system is to be fielded. Furthermore, the threat should be reviewed and updated periodically at the milestone decision points to ensure that the specified design weapon(s) is representative of the major “likely to be encountered” threat(s) to the system.

3. The committee recommends that the Director, Test and Evaluation, expand the charter of the Live Fire Test and Evaluation program from its current oversight of those tests that are part of the congressionally mandated Live Fire Test program to include oversight of vulnerability assessment.

• Recommendation Regarding the Vulnerability Assessment Methodologies

6. The committee recommends that both the analysis community and the live fire testing community routinely include on-board ordnance in their assessments. A waiver to allow full-scale Live Fire Tests without on-board ordnance should be granted only after an examination of the results from alternate live fire tests of sub-scale components and their integration into analyses of the full-up aircraft carrying such ordnance.

7. The committee recommends that the Under Secretary of Defense, Acquisition, direct the multi-Service coordinated development and authorization for use of improved analytical vulnerability assessment models that are applicable to all military aircraft. The current Joint Technical Coordinating Group on Aircraft Survivability (JTCG/AS) approved models could form the basis for the new models. The 1987 General Accounting Office study on Live Fire Testing provides many suggestions on how to improve these models.

8. The committee recommends that a long-term live fire test program be funded in which realistic components, subsystems, and systems are specifically tested to develop a data base to support the analytical models.

9. The committee recommends that the Secretary of Defense (a) establish a program to examine the combat data collected from Desert Storm for “lessons learned” regarding the susceptibility and vulnerability of U.S. and allied aircraft; and (b) develop formal, institutionalized procedures for collecting data in future conflicts, for ensuring that the data collectors have access to the theater, and for permanently storing the data. The combat survivability data collection program should reflect the importance of collecting and preserving the data and should be coordinated among the three Services through a joint agency, such as the JTCG/AS.

• Recommendations Regarding Vulnerability Programs for Aircraft

10. Because of the expected service life extension of currently fielded U.S. military aircraft, the committee recommends that the Under Secretary of Defense, Acquisition, establish a formal vulnerability assessment and reduction program for these aircraft. This program should require that all product improvement or upgrade programs to existing aircraft include vulnerability assessment and, if appropriate, reduction as major goals of the program.

11. The committee recommends (a) that a vulnerability assessment program be an

integral part of every aircraft acquisition program; (b) that vulnerability assessment and evaluation be a specific item examined at each formal milestone review; and (c) that adequate funds be appropriated to the program.

1. The committee recommends that aircraft programs that become “prototype” programs, such as the RAH-66, not be excluded from live fire testing. The committee is concerned that the RAH-66 might be developed as a prototype without adequate consideration or testing of its vulnerability. If the decision is made at a later date to go into production with the prototype, it will be too late to correct any design weaknesses.

2. The committee recommends that specific vulnerability requirements on the design be a part of the survivability objectives defined at Milestones I and II. These vulnerability requirements should be identified as part of the survivability characteristics and incorporated in the aircraft development contracts.

• Recommendations Regarding the Vulnerability Infrastructure

14. The committee recommends that the Director, Test and Evaluation, establish a permanent Senior Vulnerability Assessment Board comprised of senior Services’ technical leaders, high-level OSD officials, and nationally recognized experts from industry and academia. This board would be advisory to the Director, Test and Evaluation, and chartered to review annually the proposed vulnerability assessment programs and budgets of DoD and to review the vulnerability assessment programs on specific aircraft as the need arises. This board would be similar to the boards already formed for conduct of coordinated 6.1, 6.2, and 6.3a Tech Base programs in the DoD.

15. The committee recommends that studies be conducted to determine if the existing Army, Navy, and Air Force vulnerability analysis community, test facilities, and infrastructure can be reduced proportionally to the expected overall infrastructure reduction within DoD. Project Reliance, the existing senior joint Services’ R&D cooperation group, should be charged with conducting the studies of how best to accomplish a meaningful infrastructure reduction. The committee believes that this drawdown should be carried out very carefully to ensure that essential vulnerability assessment personnel, capabilities, and facilities are not lost in the process.

The committee recommends to the Secretary of Defense that the broad issue of how to both design and test for vulnerability in an austere future be studied. Present concepts of analyses and live fire testing for vulnerability may not be adequate in a future of reduced budgets, fewer fielded aircraft, fewer program starts, smaller procurement numbers, and more “storage on the shelf” of technology capabilities with less time to react to emergencies. When such a study has been completed and an effective process has been developed for vulnerability design and validation, OSD should consult with Congress regarding revisions to the LFT law that reflect this new process.