National Academies Press: OpenBook

Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program (1993)

Chapter: 4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry

« Previous: 3 The OSD Live Fire Test and Evaluation Program
Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

4
Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry

The Army, Navy, and the Air Force have developed general Live Fire Test and Evaluation (LFT&E) policies in response to the passage of the 1987 Live Fire Test (LFT) law, and these policies are reflected in the LFT&E programs for the RAH-66 (Army), the V-22 and A-12 (Navy), and the F-22 and C-17 (Air Force). These policies and the individual aircraft LFT&E programs were presented to the committee on September 26–27, 1991, and are summarized below. Also contained in this chapter are the summaries of the presentations given to the committee on July 25, 1991 by the test community from each Service. The presenters represented the offices of U.S. Air Force, Office of the Director, Test & Evaluation; U.S. Army, Office of the Director, U.S. Army Test and Evaluation Management Agency; U.S. Navy, Office of Test and Evaluation and Technology Requirements. The highlights of all of these presentations are given below.

The committee believes it is important to point out before proceeding to the presentations that, except for the RAH-66, all of the programs described below were well under way when the law was passed and the C-17 development had proceeded beyond the deadline for the application of a waiver. Congress had given no transitional guidance for these programs. Furthermore, because the Services had only recently begun to test full-scale, full-up aircraft in the Joint Live Fire (JLF) program, they lacked adequate test facilities for conducting some of the tests; and they did not have any long-term experience in preparing the test plans for, and conducting, such tests.1 Finally, because of the confusion over the requirements of the law, no waivers were requested for any program because of the belief within the Services that the law was satisfied with less-than-full-scale, full-up tests and that a stigma would be attached to any waiver. This situation has complicated the preparation of a thorough LFT&E program that satisfies the letter of the law as well as its intent.

The Army LFT&E Programs

The Army gave two presentations on its Live Fire Testing program. One, given by a representative from the U.S. Army Test and Evaluation Management Agency, was entitled “Army Philosophy and Policy on Live Fire Testing.” The other, entitled “RAH-66 COMANCHE Ballistic Vulnerability DT&E Program Briefing,” was given by a representative from the COMANCHE Program Manager’s office. The presentations are summarized below.


Army Philosophy and Policy on Live Fire Testing. According to the presenter, the objectives of the Army Live

1

The 1987 General Accounting Office study presented in Appendix B addressed the issues of the status of the JLF test program, the methodological quality of the testing and evaluation process, and how to improve live fire testing.

Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

Fire test and evaluation (T&E) program are to demonstrate the ability of a system to provide survivability, to provide insight into the principal damage mechanisms and kill modes, and to provide insight into techniques for reducing personnel casualties and enhancing system survivability. The Army policy reiterates the legislative language. A draft policy document AR 73-XX, “Test and Evaluation Policy,” was released for interim use in December 1990. According to that document, LFT&E will be integrated into the overall T&E program strategy. The Live Fire Tests are part of technical testing, and the scope of the program will build on early testing of components and on modeling. The LFT&E strategy will be contained in the Test and Evaluation Master Plan (TEMP). A draft document Department of the Army Pamphlet (DA PAM) 73-XX, “Test and Evaluation Procedures Guidelines,” is being prepared. This document will incorporate the document “An Army Guide to Live Fire Test and Evaluation,” August 1990.

According to the presenter, the Army LFT&E strategy involves a complementary testing and modeling effort. Modeling is an essential part of the strategy for determining system vulnerability since it is impractical to test the total spectrum of weapon/aircraft interactions, and testing is required to provide the necessary data to develop the vulnerability model algorithms. The LFT&E strategy follows a “building-block” approach consisting of component testing; full-up subsystem testing; and full-scale, full-up system testing. The emphasis of the testing program is on front-end testing of the components and subsystems to ensure that the performance of these tested articles is understood. Limited full-scale, full-up testing is conducted to confirm this understanding.


Live Fire Testing of the RAH-66 COMANCHE Helicopter. The Program/Ballistics Detailed Schedule for the COMANCHE (presented to the committee in September 1991) is shown in Figure 4-1.2 The ballistic assessment tests began in April 1991 and ended in November 1991. A three-and-half year period of ballistic verification and demonstration tests and Live Fire Tests begins in March 1995 and ends in September 1998.

The presenter identified three major ballistic vulnerability issues that could result from expected threat encounters. The first issue deals with the hazards to the COMANCHE aircrew, the second issue deals with the vulnerabilities of the flight critical subsystems, and the third deals with the mission essential subsystems. Accordingly, the objectives of the development test and evaluation (DT&E) program for ballistic vulnerability are to verify adequate aircrew and aircraft protection, to reduce the vulnerability of the flight critical components, and to reduce the vulnerability of the mission essential components. All subsystems are to be evaluated in the step-by-step test program, starting from section/ coupon testing, proceeding through static/dynamic component testing and full-up component and subsystem testing, and finishing with full-up complete system Live Fire Testing. The early tests are design support tests, and the later tests are to establish specification compliance. Two examples of design support tests using coupons or engineering mockups are the hydraulic ram tests on fuel tank panels and the ballistic penetration tests on composite panels. Two of the test articles to be examined for specification compliance are the flight control linkage and the anti-torque system drive shaft. Most of the 26 tests scheduled will use the 12.7-millimeter API; five tests are scheduled for a larger-caliber projectile. The Milestone II exit criteria tests are to demonstrate the damage tolerance potential of selected critical components to the specified design threats. Some of the test articles are the main rotor hub, main rotor blade, fantail assembly, and the T800 engine. The focus of these tests is to verify the ability of the helicopter to fly for 30 minutes after a ballistic impact.

According to the presenter from the COMANCHE Program Office, the objective of the dedicated Live Fire Test program is to evaluate the vulnerability of the full-up production configuration components, subsystems, and if necessary, a representative COMANCHE air vehicle. The focus of these tests will be on postdamage tolerance and the evaluation of synergistic effects. According to the presenter, the LFT&E program uses the approved threats; schedules the tests in a sequential “building-block” approach; tests full-up components and subsystems; will test a full-scale, full-up system, if necessary; and is in compliance with the law.


Committee Comments. Several aspects of the COMANCHE LFT&E program are of concern to the committee. First, the schedule shows a Live Fire Test program being completed just before going into full-rate production but after 72 aircraft have been bought in low-rate production contract awards, which is approximately 15% of the planned buy. It may not be possible to retrofit the changes required to correct any design deficiencies discovered in the LFT&E program into these aircraft. Furthermore, if approval is given to go into full production, another 96 aircraft will be produced in the next year while the design changes are being prepared. This is contrary to the LFT law’s requirement that testing shall be carried out sufficiently early in the

2

In early 1992, the Army restructured the COMANCHE program, stretching out the prototype phase by two years through the summer of 1997. Plans for completing the development and going into production after 1997 have been dropped, and the survivability and live fire tests have been deferred. Apparently, this is part of the Defense Department’s plans to emphasize prototype research and development of new aircraft (Bond, 1992).

Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

FIGURE 4-1 Program/ballistics detailed schedule for the COMANCHE.

development phase of the system to allow any observed design deficiency to be corrected before proceeding beyond low-rate initial production (LRIP). There are two ways this situation could be improved. One is that the demonstration and validation (DEM/VAL) prototypes will have been flying for several years before the first of the low-rate aircraft will have been contracted for, and one of these could be used for Live Fire Testing. The other way is to slow down the production rate.

Second, the briefer was vague on the actual testing of a full-scale, full-up aircraft and, when questioned, said a full-scale aircraft was not expected to be needed. On the other hand, his summary says the program will operate in compliance with the current Live Fire Test law. The presenter from the Army Test and Evaluation Agency left no doubt of the Army’s position; it will do full-scale, full-up testing.

The COMANCHE briefer mentioned the on-board munitions, but did not dwell on its contribution to the vulnerable area of the aircraft. There was little discussion of what the internal stowage of ordnance might do to vulnerability, whether this would be addressed with some kind of protection. The committee believes that the consideration of ordnance in the assessment of vulnerability, both in modeling and in testing, is essential.3

It is the committee’s opinion that the policy level of the Army supports limited full-scale, full-up testing, but that this support has not been fully recognized at the program level.

The Navy LFT&E Programs

The Navy Live Fire Test philosophy and the program for the V-22 were presented to the committee by a representative of the Survivability Branch, Naval Air Systems Command.

3

This aspect of vulnerability is examined in Chapter 2.

Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

FIGURE 4-2 Live Fire Test and Evaluation of Navy aircraft.

The Deputy Director of the Navy’s Office of Test & Evaluation and Technology Requirements was present at the briefing. Preliminary information on the LFT&E program for the A-12 aircraft was provided during several committee meetings. The Navy’s presentation is summarized below.


The Navy Live Fire Test and Evaluation Policy. The Navy’s view of the Live Fire Test law is that it encourages full-up vulnerability testing, under realistic combat conditions, first at the sub-scale level and later at the full-scale level. Live fire testing is part of developmental testing and is a continuing process that contributes to the development and engineering of Navy aircraft. The Navy policy is to comply with the intent of the law by establishing realistic operational requirements and combat threat scenarios, establishing explicit Survivability and vulnerability design requirements, and requiring Live Fire Testing and Evaluation as part of the development process using a shoot-fix-shootagain approach. The Navy LFT&E program integrates Live Fire Testing with the development of the aircraft, as shown in Figure 4-2. The integration of the tests on simulators and sub-scale hardware early in the development provides design information in time to be useful. Later Live Fire Verification Testing using simulators and surrogates provides information on the design's compliance with the specifications.

According to the Navy presenter, the benefits identified for full-scale, full-up testing are the possible discovery of previously unknown reactions to the weapons and synergistic effects. The disadvantages include the fact that the results are very limited statistically; the full-scale aircraft is at risk on each test; the tests are very costly; and the full-scale, full-up tests do not eliminate the need for all of the earlier sub-scale tests. Furthermore, testing the full-scale aircraft can be accomplished only at a very late stage in the development, and it may not be possible or practical to correct any design vulnerabilities discovered in the tests. This is contrary to the requirement in the law that the testing shall be carried out sufficiently early to allow any design deficiency demonstrated by the testing to be corrected before proceeding beyond low-rate initial production. In summary, Live Fire Testing is a process that begins in the D/V phase; it is not a single pass/fail event. It involves both analyses and testing, with results available sufficiently early to aid in design.


Live Fire Testing of the V-22 Aircraft. A comprehensive plan for the Live Fire Testing of the V-22 was made available

Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

to the committee, and though it is somewhat dated it was a useful input. The program is old enough to be grandfathered for Live Fire Testing even though the summary page says the current program meets the Live Fire Test law.

The V-22 aircraft had specific design requirements on the allowable vulnerable area for several specific threat weapons, excluding hydraulic ram. A number of alternative vulnerability reduction designs were identified, and the reduction in vulnerable area and increase in weight were determined for each alternative. The most promising alternatives were tested. Some of the test articles used in the stage I ballistic tests included a wing structure, a fuselage structure, a pylon actuator, and some of the flight controls. Ballistic verification tests have been completed for phase I and phase II wing tank hydraulic ram and the propellar rotor gearbox components. A series of dry bay fire suppression tests has been conducted using a representative three-bay simulator. Four different suppression techniques were tested; a Halon system, power panels, foam, and an on-board inert gas generating system (OBIGGS). The threats were several ballistic projectiles. In the baseline tests without any protection, large, consistent fires with heavy fixture damage were observed. The results using the different suppression techniques ranged from several fires to no fires.


Committee Comments on the V-22. The V-22 aircraft has been under development for several years and has been under a cloud for many of them. The cloud is related to the need for the system rather than anything related to its vulnerability. The cancellation of the program by the Secretary of Defense and the reinstatement of the program by Congress tend to make systematic planning for various phases of Live Fire Testing difficult.

The purpose of the aircraft is to take troops and light material into the vicinity of danger, and for that mission the Joint Services Operational Requirement for ballistic tolerance seems inadequate. Furthermore, there is no discussion of the vulnerable area associated with on-board ordnance. The aircraft will be hauling U.S. Marines into combat, and they will put all of the weapons and ordnance they can load aboard. An analysis that shows how the load could be distributed, or how some light armoring could change the vulnerability of the aircraft to ground fire, would give more credence to the claim that the ballistic protection had been achieved.

The fate of the LFT&E program is dependent upon a decision to enter into low-rate production.


Live Fire Testing of the A-12 Aircraft. The A-12 was a highly classified program, and therefore, it was not a covered system according to the LFT law. Nevertheless, a member of the LFT Office was read into the program because of the fact that the aircraft would become a candidate for LFT when it no longer was a black program. According to one of the briefers, the A-12 live fire test program for components and subsystems was one of the best programs to date. Although the proposed LFT strategy did not include testing on the full-scale, full-up aircraft, it appeared to be reasonable according to the briefer. Apparently, informal approval was given to the proposed LFT strategy, and this strategy was in effect when the program was canceled. Two major criticisms of the program were the omission of testing of the on-board ordnance and the omission of consideration of threats other than projectiles and missiles, such as directed energy weapons.


Committee Comments on the A-12. It is the committee’s opinion that the policy levels of the Navy and the Program Offices appear to have no intention of doing any full-scale, full-up test on any aircraft before proceeding beyond low-rate initial production. The Navy considers full-scale, full-up testing of aircraft to be unreasonably expensive and impractical. Apparently as a consequence of this belief and the interpretation of the LFT requirements given in the LFT&E Guidelines, no full-scale, full-up LFT tests were proposed for the A-12. The committee’s opinion is that any approval of an LFT strategy for the A-12 that did not include tests on the full-scale, full-up aircraft and did not request a waiver from these tests would have been in error. The committee is concerned that future black programs will face the same problems with respect to the LFT law that occurred with the A-12. The Navy has initiated efforts on a replacement program for the A-12, presently called the AX. Because this also is a black program, it is not a covered system under the LFT law. However, it too will eventually become an unclassified program and at that time will be subject to the law’s provisions. This will probably occur after the program enters into full-scale engineering development, the deadline for the application for a waiver. If the requirements in the LFT law are ignored until the program comes out of the black, there most likely will be a confrontation with the LFT Office.

The Air Force LFT&E Programs

The Air Force Live Fire Test Policy. The Air Force policy on LFT&E was presented by a representative from Test & Evaluation, Air Force, and was entitled “Air Force Policy Considerations for Live Fire Test & Evaluation.” According to the presenter, the objective of Air Force LFT&E is to provide a timely and thorough assessment of the vulnerability and lethality of a system as it progresses through its development. This is accomplished by a balanced program

Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

of analysis and test. It is a systems approach, similar to development testing used in other areas. Because it is prohibitive to test all possible combinations of threat/aircraft/conditions, analysis must be an integral part of the LFT&E process. A general sequence of design-analyze-test is followed and repeated if necessary. The analytical models are those accepted by the Joint Technical Coordinating Group on Aircraft Survivability (JTCG/AS). The LFT&E program is initiated sufficiently early to allow the results to impact the system design. The benefits of LFT&E can be maximized only when the test results are used to aid in the design. The LFT&E strategy, which is included in the TEMP, ensures that all issues are addressed and integrated with the other elements of the system program. According to the presenter, the Air Force policy is in compliance with the reporting requirements of the LFT&E Office. The threat used for the Live Fire Tests is the threat(s) defined by the Operational Requirements Document (ORD). The ORD is the official source of user requirements, and the ORD process places the user in the role of integrating the System Threat Assessment Report (STAR), aircraft characteristics, employment concepts, etc., to derive the expected threat.

According to the presenter, the test hardware will be of sufficient size and quantity so that realistic test results, including synergism and secondary effects, will be obtained. Experience indicates that components, subsystems, and subassemblies are generally sufficient and more desirable than a complete system. Full consideration will be given to the use of actual hardware, replicas, and surrogates. However, use of these items must depend on the technical payoff, availability, and cost. The requirements for additional live fire test facilities are being identified in the Air Force’s Test Investment Planning Process. In summary, according to the Air Force, its LFT&E program is integral to the system design, development, test, and evaluation process. The Air Force believes it endorses an intelligent approach to LFT&E that considers all variables affecting aircraft effectiveness. It believes the Air Force T&E approach employs the most prudent combination of analyses and tests, is viable, and is producing positive results.


The Live Fire Test Program for the F-22 Aircraft. The Live Fire Test program was briefed to the committee by a representative from the system program office in a presentation entitled “F-22 Vulnerability Program.” The approach to aircraft survivability used by the F-22 is a combination of low susceptibility and reduced vulnerability. Directed energy weapons are considered, as well as the conventional gun and missile threats. The reduction in vulnerability is accomplished through incorporation of redundant subsystems and damage tolerant features. Vulnerability analyses will be used to establish any system weaknesses and to identify vulnerability reduction candidates. Vulnerability testing of materials, components, and subassemblies is used to verify the vulnerability reduction candidates and to identify issues that necessitate Live Fire Testing. The results from the analyses and the lower-level tests will determine the need for, and extent of, further validation tests. According to the presenter, the LFT&E program is a sound development and demonstration program that represents a balance of the technical merits, cost, and schedule. The methodology used in the program is shown in Figure 4-3, and the vulnerability program schedule and funding are given in Figure 4-4. According to the presenter, the budget and schedule supports the conduct of prudent Live Fire Testing. Some of the assessments to be conducted to verify the design include fuel tank inciting using a rig test, redundancy and separation studies using analyses and inspection, verification of the fragmentation resistance of pressure vessels using ballistic tests, and a static test of structure.


Committee Comments on the F-22. In the F-22 presenter’s discussion of vulnerability modeling for the engineering, manufacturing and development (EMD) study he stated that internal armament was treated. However, a chart of vulnerability requirements included a column for armaments that discussed vulnerable area as to be determined (TBD) and fire inciting, fire protection, and shielding as not applicable (N/A). The EMD vulnerability program shown in Figure 4-5 has a category for stored munitions, engine, and aircraft availability, but it is under concurrent JLF. When members of the committee questioned the possibility that this was the prototype aircraft, the briefer seemed unaware that a Live Fire Test would be done on a full-scale, full-up aircraft, and that the prototypes were so different from the production model that they would not be representative.


The Live Fire Test Program for the C-17 Aircraft. The program for the C-17 LFT&E was given by a representative from the C-17 System Program Office in a presentation entitled “C-17, Live Fire Test (LFT) Program.” According to the presenter, the C-17 is expected to be deployed to within 20 to 40 km of the Forward Edge of the Battle Area FEB A). The vulnerability requirement for the selected design threat in the FEBA is met by using redundancy with separation, damage tolerant components, and fire and explosion suppression. A number of design changes were made to meet the requirement as the result of analytical assessments. These included the relocation of the oxygen converter, rerouting of the hydraulic lines, and changes to the pitch trim actuator and aileron hinge fitting. The test methodology included design, analysis, test, redesign, and retest. The test articles included components, such as crew armor, pressure vessels, the upper wing skin, and a flap

Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

FIGURE 4-3 Vulnerability assessment methodology for the F-22.

Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

FIGURE 4-4 Vulnerability program for the F-22.

hinge fitting; and a functional 8-foot section of the wing leading edge. The leading edge section is to be Live Fire Tested to determine if there is a leading edge fire problem. The test article is designed from production drawings from the Douglas Aircraft Company and will be built by the 4950th Test Wing Modification Branch. The Live Fire Tests will be conducted by the Wright Laboratory’s Flight Dynamics Directorate.


Committee Comments on the C-17. The C-17 represents one of the most interesting challenges to the Live Fire Test law of 1987. It was in full scale development when the law was passed and hence could not legally request a waiver. It is a very large and expensive aircraft, although not the largest to enter the inventory. The largest, the C-5A, was delivered during the 1960s and was not subject to any live fire testing. Apparently, Congress considers the C-17 to be a Live Fire Test candidate and expects to see Live Fire Testing conducted on the C-17 aircraft (Bennett, 1991). The contention by Congress is that testing of components from a military vehicle in an essentially inert condition is no substitute for the firing of threat ammunition at that vehicle loaded for combat with the intended fuels, fluids, and ordnance on board and in place.

In the case of cargo aircraft such as the C-17, if the aircraft contains ordnance when configured for combat, the ordnance and its reaction to ballistic impact must be treated in the assessment. A sub-scale, but full-up, Live Fire Test of the C-17 would logically consist of testing a mock-up of an actual cargo compartment, with ammunition loaded in pallets on dunnage. If the shot created significant damage due to a reaction of the ammunition, shielding or armoring could be undertaken to bring the damage expected to an acceptable level.

During a temporary shifting around of personnel in the

Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

FIGURE 4-5 Vulnerability requirements for the F-22.

Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

Office of the Deputy Director, Defense Research and Engineering (Test & Evaluation), the Air Force’s proposal to test only a portion of the leading edge of the wing was accepted as being in compliance with the LFT law. This enforced the belief that the conduct of full-scale, full-up Live Fire Testing could be avoided by the acquisition arm of the Air Force. The Air Force user should have been put more of a decision-making position on this issue. The size of the C-17 is going to make it an attractive target for various potential enemies who might hold fire from smaller targets in order to concentrate fire on the larger aircraft. If an aircraft is to go into combat, and if combat damage is expected to be encountered, the aircraft is going to have to be tolerant of the damage or a price will be paid in manpower and material losses. Supporters of the Live Fire Test law believe that if the full-scale C-17 is too expensive to be subjected to live fire in a full-up condition, the aircraft is too expensive to risk losing by delivering cargo to a forward military zone. If that is the case, the upper limit on the cost a Service may be willing to put into an aircraft may have been reached. A reasonable question to ask is, would the aircraft be bought if it was to be used only to deliver military cargo to rear areas? If that answer is no, the importance of testing the aircraft against the weapons it is expected to encounter is paramount.

The Test Community’s View of Live Fire Testing

The live fire test community of the three Services appears to be in general agreement about the efficacy of the Live Fire Test law as it applies to aircraft. Members of that community do not consider it necessary to do full-scale, full-up tests in order to determine most of the design vulnerabilities. However, they do consider it essential to conduct many sub-scale tests on components and sub-systems, both inert and full-up, in the development cycle of an aircraft. The sub-scale test results are invaluable sources of information that can be used initially to aid the design and later to verify the design. The test community believes that the full-scale, full-up tests are conducted too late in the development cycle to be of much value to the designer and that the amount of information obtained from the tests is very limited.4 Every time a test is conducted a portion of the test article is damaged, and it may be impossible to restore the article to its original condition. Furthermore, these tests require very large budgets that could go toward the expansion of test facilities and capabilities and the conduct of many more sub-scale tests. The test community does recognize the possibility of an unanticipated reaction, cascading damage, or synergism occurring in the full-scale aircraft. However, its members believe that nearly all of the kill modes of an aircraft are known and can be anticipated; they state that they very seldom see a test result that was unanticipated. The magnitude of the response to the hit may be larger or smaller than expected, but the response was anticipated.5


Committee Comments. The committee notes that not everyone who has observed live fire tests on sub-scale and full-scale test articles shares the views held by these testers. They believe that there have been unanticipated results from these tests. Furthermore, even when the response is as expected, the difference in the expected magnitude of the response and the observed magnitude often is too large to be acceptable. The testers have also overlooked the fact that information from the full-scale tests is an input to acquisition decision makers at milestone reviews. The committee also believes that the test community has not given proper consideration to the on-board ordnance problem.

The Industry View of Live Fire Testing

The U.S. aircraft industry does not play a major role in LFT&E. However, the committee solicited its opinion because these are the users of the results of the Live Fire Tests. The four industry representatives were unanimous in their belief that the LFT law as described in the LFT Guidelines and Planning Guide does not require a full-scale, full-up aircraft to be tested. Furthermore, they do not believe it should. One of the major reasons they gave for not recommending full-scale testing of a “production” aircraft is that the design is usually frozen by the time the results become available.6 However, they were in full agreement with the test community on the necessity of sub-scale testing throughout the program development.

Controversy Regarding Which Munitions to Use in the Live Fire Testing Program

The Live Fire Test law stipulates that “munitions likely to be encountered in combat (or munitions with a capability similar to such munitions)” are to be used in the “realistic survivability testing” The specific munitions to be used in the Live Fire Testing of a particular aircraft are selected by the aircraft Program Office as part of its LFT&E program plan. Typically, the threats selected by the Program Office for Live Fire Testing are the threats the aircraft was designed to withstand, such as a single hit by an armor-piercing (API)

4

The test community is not against all testing of full-scale, full-up aircraft; it fully supports this type of testing on existing aircraft in programs such as JLF.

5

This attitude appears to be in contrast to the experience of the testers of ground vehicles, where the particular response to a hit was often unanticipated.

6

The committee notes that industry shares the same reluctance as the Services to change a design late in the development cycle.

Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

or small-caliber high explosive with incendiaries (HEI). The assumption is made that the more lethal overmatching threats, such as the larger anti-aircraft artillery (AAA) and the surface-to-air missiles (SAMs), will be avoided and hence should not be a part of the LFT&E program. For example, the Navy and the Air Force are currently developing stealth aircraft whose survivability is strongly related to not getting hit. They believe that some “credit” should be given for this aspect of survivability. Aircraft that do not get hit as often do not have to rely as much on their reduced vulnerability to survive. Furthermore, if these overmatching threats are included in the LFT&E program, they could destroy the only test article(s) available, as well as the test equipment, and perhaps some of the test facilities.

The Live Fire Test Office has interpreted the phrase “munitions likely to be encountered in combat” to mean that those munitions the aircraft may encounter, including the latest directed energy weapons, should be included in the LFT&E program, regardless of the design threat for the aircraft. This interpretation of the phrase “likely to be encountered in combat” made by the LFT Office conflicts with the interpretation made by the Program Managers, and this difference in interpretations has been the source of considerable controversy. The LFT Office argues that the user Service’s intention to avoid the most lethal threats in combat through susceptibility reduction may not always be achievable. Given that the intent of the LFT law is not to intentionally destroy aircraft, the LFT Office believes that a test involving a “potentially” overmatching weapon can be designed that will provide information on any design weaknesses with little likelihood of destroying the aircraft. For example, suppose the weapon is a guided missile with a large HE warhead and a proximity fuze. The lethality of this weapon depends upon the location of the warhead with respect to the aircraft at the time of detonation. If the detonation occurs next to the aircraft, the aircraft will most likely be killed. On the other hand, if the detonation occurs far from the aircraft, the aircraft will most likely not be killed. Consequently, the LFT Office believes that if information on any design weaknesses can be obtained from a distant detonation with little probability of destroying the aircraft, the test should be considered. The Services counter this argument with the argument that any information obtained from the limited number of tests is not statistically meaningful. This is, in turn, countered by the argument that one purpose of the full-scale tests is to discover vulnerabilities, not to quantify vulnerability.


Committee Comments. Today, military systems are designed to survive the threat they can expect to find in the field at the time they are fielded.7 The process of defining the future threat is called threat projection. The results of the threat projection appear in STAR. Historically, the threat community has been reluctant to commit to a firm projection of the specific threats a particular system will encounter 10 to 20 years in the future for several reasons. Consequently, a long list of threats that the system may encounter in combat is usually prepared, perhaps with some prioritization. However flawed the threat projection process may be, the user of the proposed system must select a design threat for the system. The developer then designs a system that will survive the threat described in the user’s requirements document.8

Note that the user is the organization that examines the threat projection and selects the design threat. The design threat selected may or may not be the most likely threat that will be encountered; usually it is but one of many threats that are likely to be encountered. The Live Fire Test Office does not consider the design threat selected by the user as the only threat likely to be encountered and strongly encourages the Program Office to test the system against the other threats that exist, or will exist, in the operating environment. The argument given by the LFT Office is that the threat selected for design is usually one that can be defeated without significant increases in weight and cost. Thus, given that it is important to verify the design’s ability to withstand the design threat, it is also important to determine, and correct if possible, any weaknesses in the design when subjected to the other “likely to be encountered” threats before the system goes beyond LRIP. An argument against testing against the other, possibly overmatching, threats is that even though the system was not required to defeat these weapons, any negative results from such tests may jeopardize the program, as well as siphon off much needed funding.

The C-17 LFT&E program is an example of the conflict between testing only against the design threat and testing against other threats that may be encountered in combat. The Operational Requirements Document for the C-17 stipulates a design that will withstand a single hit by a certain ballistic projectile. According to the C-17 presenter, this projectile may be encountered by the C-17 in the region of 20 to 40 km from the Forward Edge of the Battle Area. The Live Fire Test Office, on the other hand, believes that C-17s in the forward area of the battlefield and at austere landing sites near combat zones can expect threats that are

7

Aircraft have not always had specific design requirements to survive the expected threat, particularly on the vulnerability aspect of survivability. Perhaps the first aircraft to have a vulnerability requirement on the design were the U.S. Army’s UTTAS and AAH, now the UH-60 Black Hawk and AH-64 Apache. Both aircraft had to be designed to survive a single hit by a non-explosive ballistic projectile anywhere on the aircraft and fly for 30 minutes after the hit.

8

As a consequence of this procedure, systems are typically designed against today’s threats—and sometimes even against yesterday’s threats. Both practices result in systems being designed to survive yesterday’s threats when they finally appear in the field.

Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

more much lethal than the design threat selected for the C-17.9

A Program Manager’s View of the Live Fire Test Law

The committee, after listening to all of the presentations, believes the following description of a Program Manager’s (PM’s) view of the LFT law may be representative of current opinion:

The full-scale, full-up testing mandated by the LFT law is an unquantifiable, but potentially catastrophic, risk to his program. LFT has no quantitative contractual specification or acceptance criteria at program initiation. No quantitative criteria for acceptable or unacceptable damage are included in the requirements process, milestone commitments, or contractor performance documents. LFT of the full-scale aircraft occurs late in the development phase of the program, and there may be neither adequate time nor money to conduct the tests or to make any changes required as a result of the tests. Further, the PM may believe that neither the definition of the tests, nor the conduct of the tests, nor the interpretation of the test results is totally under his control. The perceived jeopardy to his program created by LFT is exacerbated by the severe requirement to fit the program into a somewhat inflexible overall resource schedule, both in time and in dollars. In summary, LFT represents a considerable source of problems to the PM, in the form of an uncontrollable, potentially catastrophic uncertainty, as he attempts to successfully complete the development of his system, and should be avoided if at all possible.

Issues Relating to Distrust Among the Participants in Live Fire Testing

The committee is aware of the strong differences of opinion held by various individuals and organizations concerning the efficacy of the Live Fire Test law and of the level of mutual distrust that has evolved as a result of these opinions. This distrust between the various participants of each other’s motives and actions is probably responsible for the ever-increasing tensions within the current Live Fire Test program. The attitudes of the major participants concerning the Live Fire Test law and its place in the acquisition process appear to be those described below.

In the committee’s opinion, the Services may believe that, as system developers and users, they know what is needed in the equipment they will take into the field, and that they, the Services, are directly responsible for the fate of the military personnel who use this equipment in combat. They are very apprehensive about any outside organization that can dilute their ability to define the necessary equipment testing and the procedures required to accomplish this testing. They appear to further believe that the Live Fire Test law gives to others not directly responsible for the delivered product inordinate control without any accompanying responsibility for the quality of the product or its cost.

In the committee’s opinion, the Office of the Secretary of Defense (OSD) may believe that there have been a sufficiently large number of prior experiences in the area of live fire testing to indicate that pressure by a Service for successful and rapid certification of its products under development can lead to inadequate live fire testing and to subsequent unnecessary combat vulnerabilities. The OSD, therefore, has chosen to exercise close control over the Live Fire Test programs and assumes the ultimate authority for approval of the equipment based on the program results.

In the committee’s opinion, the Congress may believe that there are sufficient numbers of proven instances of unnecessary combat vulnerability in the Department of Defense (DoD) equipment previously delivered to the field to warrant legislative direction of DoD test certification to include live fire testing of full-scale, full-up systems using munitions likely to be encountered in combat. Congress further believes that it has the ultimate responsibility for the programs it authorizes and therefore has the obligation to exercise that legislative direction.

Conclusion

The committee believes the common view of all parties knowledgeable in this business is that the current assessment procedure, including both analysis and Live Fire Testing, does not guarantee that the U.S. armed forces will field cost-effective systems designed for reduced vulnerability. The intent of the LFT law to contribute to the creation of less vulnerable aircraft designs is valid; its execution to achieve this intent has been flawed in several ways as identified in Chapters 2, 3, and 4. The committee believes that the crux of the problems with the LFT law is (1) the ambiguity of the law; (2) the lack of a clear and binding LFT policy directive; (3) the reluctance by the Services to—for fear of a stigma10—ask for a waiver from full-scale, full-up LFT for those programs which they believe LFT to be unreasonably expensive and impractical; and (4) the absence of a formal waiver process that includes a procedure for identifying when full-scale, full-up testing is or is not unreasonably

9

Military airlifters have flown in hostile environments in the past. Examination of the combat data from the Southeast Asia conflict reveals damage and losses to C-7s, C-123s, and C-130s from small arms, AAA, and shoulder-fired SAMs, and 14 airlifters were damaged by small arms in only 400 sorties in 1989 in Operation Just Cause (Ropelewski, 1990).

10

There is also the possibility that the Services do not consider it appropriate for Congress to become involved in this area of program management and acquisition.

Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

expensive and impractical which would eliminate any stigma associated with the waiver.

References

• Bennett, Charles E., 1991. Vice Chairman of the Armed Services Committee, Letter to the Honorable Charles E.Adolph, January 2.

• Bond, D.F., 1992. U.S. Army Revamps COMANCHE Plans, Considers Next Bid for More Funds,” Aviation Week & Space Technology, pp. 20–21, March 16.

• Ropelewski, R.R., 1990. Planning, Precision, and Surprise Led to Panama Successes,” Armed Forces Journal International, February, 26–32.

Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 42
Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 43
Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 44
Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 45
Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 46
Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 47
Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 48
Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 49
Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 50
Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 51
Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 52
Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 53
Suggested Citation:"4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 54
Next: 5 The Future of Vulnerability Assessment »
Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!