Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program (1993)

The committee recognizes that there will be a major change in the environment in which the Department of Defense (DoD) will acquire weapon systems in the future, given the dramatic changes in the world socio-political environment and the current difficulties with the U.S. economy. Almost certainly, the future DoD budgets for acquisition of new systems will be significantly reduced, with a concomitant reduction in the number of new program starts. Two recent examples of the types of changes to be expected are the reduction of B-2 aircraft buy from 75 to 20 and the restructuring of the RAH-66 COMANCHE program to postpone production while emphasizing and stretching out the prototype phase. Future budgets for improving existing systems most likely will be below—possibly much below—current levels. Future budgets for the technology base (not counting independent research and development) are difficult to predict at this time, but may be above current levels in order to maintain the current U.S. technology edge.

In spite of the significant reduction in the future DoD acquisition budgets, the requirement to maintain and technologically update fielded platforms over substantially longer lifetimes to counter new threats will remain. When new programs are initiated or prototypes moved into production, it probably will be in response to important new threats not able to be defeated by current platform improvements. In all cases, there will be a requirement to contain the costs of the new or prototype system. Although program cost containment has always been a concern within DoD, the committee believes that minimizing total program costs in the foreseeable future while improving weapon system abilities to counter and survive new threats will be an even more important issue.

In this regard, the committee notes that the restructured program for prototyping the RAH-66 has deferred the survivability and live fire tests. Because there is no intention to go into production at this time, the RAH-66 is not a candidate for Live Fire Testing. The committee is concerned that by deferring the survivability and live fire tests, the vulnerability of the prototype design will not receive the proper attention. A decision could be made in the future to move the prototype into production, with no further vulnerability assessments planned to determine any design weaknesses. However, once this decision is made, the helicopter becomes a candidate for Live Fire Testing. Any attempts to change the prototype design to reduce vulnerability once the decision has been made to go into production may be met with much resistance. This is going to create an adversarial situation again. It would be much better to find and correct any vulnerabilities in the design during development of the prototype than attempt to do so after the development is completed.

In addition to individual program cost containment issues,

the committee anticipates a reduction in the test and evaluation infrastructure within the Office of the Secretary of Defense (OSD) and in each of the Services as the total DoD budget declines. In particular, the independent charters, staffs, and facilities of the vulnerability assessment community may have to be changed, as well as the emphasis currently given to the analysis and test communities. The committee believes that new approaches to many of the current DoD weapon acquisition procedures may have to be developed to respond to the new fiscal threat environment, and among these approaches may be changes to the procedures and infrastructure for vulnerability assessment.

The committee believes that it is vital for the future effectiveness of U.S. military aircraft that any changes made to the vulnerability assessment procedures and infrastructure must not degrade the current capabilities for vulnerability assessment within DoD. Vulnerability assessment, which is an integral part of designing for survivability, will increase in importance as the numbers of front line aircraft decrease and the lifetime of each aircraft is extended. Furthermore, because fewer new systems will be developed in the future, the vulnerability of each of these new systems becomes more important. Consequently, more funding for vulnerability assessment may be required for the few systems that are developed, rather than less funding.¹ On the positive side of this situation, as more new starts are prototyped and development time is increased, there are more opportunities for better assessments. For example, the results from live fire tests on full-scale, full-up targets may be obtained sufficiently early in the stretched-out development time to influence the final design. An additional impetus for vulnerability assessment is the close and important connection between combat vulnerability and flight safety. Aircraft designed to take hits in combat and to survive crashes are inherently safer aircraft for the aircrew to fly in. Many design features included to reduce vulnerability, such as fire and explosion suppression, flight control reconfigurability, hydraulic power redundancy and separation, rotor blade toughness, and engine hardness, also prevent the loss of an aircraft due to peacetime malfunctions.

Based on the suppositions given above, the committee believes that the vulnerability assessment capabilities of the future, both analytical and testing, should be developed as follows:

1. Vulnerability assessment will continue to be required to design and validate the vulnerability expectations of new military platforms. Consequently, the current methodologies must be improved. However, the assessments and their improvements must use the available funds in the most efficient way.

2. There will also have to be a cost reduction within the DoD vulnerability analysis and test infrastructure, because there will be fewer aircraft to assess, while still maintaining necessary staff expertise and test facilities for the remaining programs.

3. A vulnerability test and evaluation procedure must be developed that provides, despite any cost reductions, a greater level of trust among the three Services, the OSD, and Congress than exists today.

The committee’s view is that unless a procedure for vulnerability assessment and design validation can be developed that increases the mutual trust among participants, there is very little chance that the requirement to reduce the funds expended for assessment while maintaining the capability to produce a less vulnerable and more survivable aircraft will be satisfied. This procedure must be a logically based method that removes all emotionalism and arbitrariness from the assessment process.

Category 1—Increased Reliance on Analysis/Modeling. One method for reducing the costs of vulnerability assessment would be to rely more on the analysis/modeling methodology.² Based on its own review, the committee believes that there appears to be a sufficient start of a modeling capability and weapons effects and materials data base to warrant an increased dependence on analysis/modeling for future vulnerability assessments as an aid in design. However, the committee also believes that the current analytical methodology and supporting data bases are not yet sufficiently robust, correct, precise, and representative to permit a total dependence on this methodology. Much work needs to be accomplished in the model development and in the accumulation of weapons effects and material P_k/h data bases. Consequently, much of live fire testing in the future should be oriented toward verifying the improved modeling procedures, extending the data base of weapons effects and material responses, and validating proposed design features and equipment for reducing vulnerability.

Category 2—Request a Waiver from the Live Fire Test (LFT) Law. The major factor in the cost of vulnerability assessment, which obviously provides the major opportunity for cost reduction, is the requirement for the full-scale, full-up Live Fire Test (LFT) program mandated by the LFT law. This law was written to prevent the neglect of vulnerability in the system design and was the result of the distrust among the Congress, OSD, and the Services examined in Chapter 4. The law offers a waiver from the Live Fire Tests. If this waiver is granted, considerable funds would be freed up to be used in vulnerability analyses and Live Fire Tests at the sub-scale level. Under the current Live Fire Test and Evaluation (LFT&E) Guidelines, there is no guidance as to what constitutes an unreasonably expensive and impractical Live Fire Test program and no instructions on which facts should be gathered together and used to make an unbiased, impartial, and logical waiver decision. Up to the present time, the Services have not applied for a waiver for any program, apparently because of the belief that their LFT&E programs were in compliance with the law and because they have been reluctant to take advantage of the waiver for fear of a stigma attaching itself to their program.

Many people, both in and out of the Services, are of the opinion that full-scale, full-up Live Fire Testing is unreasonably expensive and impractical for all aircraft, that there are more cost-effective ways to obtain a design with reduced vulnerability, and that this conclusion should be obvious to any one who has thought about it. Among the reasons given for this conclusion are the facts that the results come too late in the development cycle to influence the design, that expensive targets are at risk on every shot, that not enough data are obtained for statistical validity, that the wrong conclusion might be reached, and all the other disadvantages associated with Live Fire Testing presented in Chapter 2.

Others believe that full-scale LFT is unreasonably expensive and impractical only for some aircraft, such as large nontactical aircraft that are not likely to encounter threats to their survival, and is not unreasonably expensive and impractical for other aircraft, such as small tactical aircraft that are very likely to encounter threats to their survival; there are some who feel that LFT is not unreasonably expensive and impractical for any aircraft. They believe that only by testing the full-scale, full-up aircraft can certain design weaknesses be discovered, that it is never too late to change a design if it is inadequate, that a test schedule can be designed so that the full-scale target is not at risk for all shots, and all the other advantages associated with Live Fire Testing presented in Chapter 2. All of these opinions and beliefs on both sides of the fence are subjective; none are objective. No quantification of the benefits and risks associated with the combination of analyses and full-scale, full-up live fire tests compared to the benefits and risks associated with the combination of analyses and only sub-scale live fire testing has been attempted. Consequently, a decision to grant a waiver today will be based on subjective arguments.

In the future, a formal analytical procedure must be established for gathering the facts necessary to determine if the full-scale, full-up Live Fire Tests are unreasonably expensive and impractical with respect to the critical vulnerability issues and, if they are, what other assessments should be conducted in place of the complete system tests. With such a procedure in place, requesting and receiving a waiver, which should allow much of the vulnerability assessment budget to be transferred to other assessment tasks, will be an acceptable procedure. No stigma should be attached to the waiver because the procedure for obtaining a waiver is a rational one. Arbitrariness and emotionalism have been reduced or eliminated and replaced with objectivity.

Such a procedure, referred to here as a risk-benefit assessment, could be developed by using the principles of risk analysis. Risk analysis is a procedure that has been developed for projects that involve large capital outlays, significant new technology, uncertainty, and regulatory issues, such as the Live Fire Test law (Cooper and Chapman, 1987). This risk-benefit methodology will formalize and standardize the procedure for deciding if a waiver should be granted. A risk-benefit assessment would identify and quantify the risks and benefits associated with both conducting and not conducting full-scale, full-up testing. For example, the benefits (e.g., a reduction in vulnerability) associated with full-scale, full-up testing of a relatively inexpensive combat aircraft, many of which are very likely to be hit in combat, may outweigh the benefits (e.g., a reduction in expenditures) of not testing the full-scale, full-up aircraft. Not testing the full-scale, full-up aircraft puts too many aircraft at an unacceptable risk of destruction. On the other hand, the benefits associated with such tests on a relatively large support aircraft, very few of which will likely be hit in combat, may not outweigh the risks of not conducting the tests. The committee believes that such a methodology is essential to the process of requesting a waiver. If one is not developed, the arbitrary granting of a waiver will continue to be subject to considerable controversy, and attempts may be made to avoid all live fire testing.

Category 3—Consolidation of the Vulnerability Assessment Infrastructure. The third category for reducing costs is to consolidate the various Service live fire test facilities and vulnerability analysis/modeling organizations. Although the committee did not review in detail this aspect of the vulnerability assessment activities and capabilities in each of the three Services, it believes that until the recent DoD budget downturn, there were sufficient programs to warrant the continuation of more-or-less similar Service live fire test capabilities. Continuous review within the Services and by

OSD has already led to reduction of duplicative underutilized capabilities. However, in the decades ahead, the expected requirements for the vulnerability testing of new Service equipment will probably fall below that level where a critical mass of broad-based facilities and knowledgeable staff can be maintained within any of the individual Services. If considerable cost savings could be achieved by consolidating the capabilities of these facilities, some additional form of consolidation beyond that currently contemplated should be considered. Furthermore, an overall reduction in the cost of test facilities does not automatically imply a reduction in capabilities at every facility. In fact, by downsizing or eliminating some facilities, others can increase in capabilities. Perhaps a national vulnerability test center could be created that would serve all Services, with significantly more capabilities than currently exist at any one facility today, at a cost below that in effect today.

• Cooper, D. and Chapman, C., 1987. Risk Analysis for Large Projects, Models, Methods, & Cases, John Wiley & Sons Ltd.