National Academies Press: OpenBook
« Previous: 2 Evaluation of the Cost, Effectiveness, and Deficiencies of These Methodologies
Suggested Citation:"3 The OSD Live Fire Test and Evaluation Program." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

3
The OSD Live Fire Test and Evaluation Program

The Live Fire Test law passed in 1987 stipulates that realistic survivability testing be conducted on covered systems. Realistic survivability testing is defined as “testing for vulnerability of the system in combat by firing munitions likely to be encountered in combat (or munitions with a capability similar to such munitions) at the system configured for combat, with the primary emphasis on testing vulnerability with respect to potential user casualties and taking into equal consideration the susceptibility to attack and combat performance of the system” (U.S. Congress, 1986–1989). Systems covered by the law are vehicles, weapon platforms, or conventional weapon systems that include features designed to provide some degree of protection to users in combat and are major systems within the meaning of that term in section 2303(5) of title 10, i.e., Acquisition Category I and II systems. The Secretary of Defense may waive the requirement for Live Fire Testing of candidate systems if, before the system enters full-scale development, the Secretary “certifies to the Congress that such testing would be unreasonably expensive and impractical” (U.S. Congress, 1986–1989). The request for a waiver will be prepared by the Service and submitted to the Secretary through the appropriate chain of command. If the waiver is granted, the Secretary shall include with any such certification a report explaining how the Secretary plans to evaluate the survivability or the lethality of the system or program and assessing possible alternatives to realistic survivability testing of the system or program (U.S. Congress, 1986–1989).

Personnel from the Live Fire Test Office and from the Deputy Director, Defense Research and Engineering (Test and Evaluation/Tactical Warfare Program) [DDDR&E(T&E)/TWP] presented the Office of the Secretary of Defense (OSD) Live Fire Test and Evaluation (LFT&E) program to the committee on July 24, 1991, and the Director of the Live Fire Office made a second presentation to the committee on January 15, 1992. Personnel from the Institute for Defense Analyses gave presentations to the committee on September 26, 1991.


Oversight and Published Guidance for LFT&E. When the Live Fire Test law was passed in 1987, the position of Director, Live Fire Testing, was established under the Office of the Director of Defense Research and Engineering (Test and Evaluation) and was given the responsibility for implementing the Live Fire Test (LFT) legislation, developing and issuing guidelines for test plan preparation, reviewing and approving Service-prepared plans, performing independent assessments of the test results, providing liaison with Congress on the test results, and working with the Services and the Department of Defense (DoD) in improving LFT methodology and instrumentation, and in general, ensuring

Suggested Citation:"3 The OSD Live Fire Test and Evaluation Program." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

that conduct of Live Fire Testing is within the spirit of congressional intent (O’Bryon, 1987). A series of written documents has been prepared by OSD to provide guidance regarding details not specifically contained in the law.

The first set of guidelines intended to implement the congressionally mandated Live Fire Test program were issued on May 7, 1987 (DDDR&E, 1987). These guidelines supplemented DoD Directive 5000.3-M-1, “Test and Evaluation Master Plan Guidelines,” October 1986, in areas pertaining to Live Fire Testing. They also defined the LFT&E plan requirements. According to these guidelines, “the essential feature of Live Fire Testing is that threat munitions are fired against a major U.S. system configured for combat to test its vulnerability, and/or that a major U.S. major munition or missile is fired against a threat target configured for combat to test the lethality of the munition or missile. Each (L)ive (F)ire (T)est plan must include testing of complete systems. However, a limited set of live fire tests may involve production components configured as a subsystem prior to full up testing. In such a case the components must be tested in the context of the complete system in that the test issues must be specific to both a threat system and a target system. In addition, at least a preliminary decision shall have been made to configure the production system with the components tested. Thus, (L)ive (F)ire (T)esting is not synonymous with traditional research and development vulnerability/lethality testing” (DDDR&E, 1987).


The 1988 Test and Evaluation Committee (LFT&E) Guidelines. The 1987 LFT&E Guidelines were superseded by the Live Fire Test and Evaluation Guidelines issued June 1, 1988, by the Test and Evaluation Committee (TEC), Office of the Secretary of Defense, in a memorandum to the Assistant Secretary of the Army (Research Development and Acquisition), the Assistant Secretary of the Navy (Research Development and Acquisition), and the Assistant Secretary of the Air Force (Acquisition) (OSD, 1988). The TEC memorandum stated that the enclosed guidelines implemented the congressionally-mandated Live Fire Test program within the DOD. The guidelines had been reviewed and coordinated by the individual Service deputies responsible for test and evaluation. According to the memorandum, “the LFT program is a highly visible and sometimes misunderstood program. These guidelines should eliminate much of that misunderstanding” (OSD, 1988). The guidelines describe the objectives of LFT&E, the scope of the guidelines, the implementation of the program, and the responsibilities of the DoD staff and the Services relative to LFT&E.


The 1989 LFT&E Planning Guide. A Live Fire Test and Evaluation (LFT&E) Planning Guide was issued in June 1989 by the Director, Live Fire Testing (DDDR&E, 1989). This Planning Guide is the current primary source of information on LFT&E provided by the LFT office to the Services and industry. The Planning Guide was intended to provide a good foundation for understanding the LFT&E program, and to be useful to system program offices and test agencies responsible for the testing and evaluation of systems identified as Live Fire candidates; nothing in the Planning Guide was intended to be inconsistent with the Live Fire legislation or any related DoD directives. The Planning Guide contains the TEC LFT&E guidelines of 1988; the Live Fire Testing legislation; and guidance concerning the definition of critical issues for Live Fire testing, the development of a strategy for LFT&E, the preparation of a detailed LFT&E plan, and the integration process involving contracting, budgeting, and scheduling. According to the Planning Guide, the intent of LFT&E (with regard to vulnerability) is “to assure that battle damage tolerance and damage control of our crew-carrying combat systems to actual threat weapons is known and acceptable…”.

The LFT Program Approach

In concert, the LFT&E Guidelines of 1988 and the LFT&E Planning Guide of 1989 require the services to take an ordered approach to LFT&E on all covered aircraft systems. This approach includes the following steps:

  1. identification of the critical issues;

  2. development of an LFT&E strategy, which must be an integral part of the Test and Evaluation Master Plan (TEMP) and is subject to OSD review and approval;

  3. integration of this strategy into the covered system program plan, budget, and schedule;

  4. development of a detailed LFT&E test plan that fully describes the tests to be conducted and criteria for measuring test results and is subject to OSD review and comment;

  5. conduct of the tests; and

  6. generation of a detailed test report subject to OSD review for submission to the Secretary of Defense and the congressional committees.

The identification of the critical issues is required to form a foundation upon which a strategy for the Live Fire Test program can be developed for each system. To fully understand the system and the threat, information that fully describes the requirements for the system, the environment in which it must operate, and concepts for operation of the system must be gathered. Vulnerability analyses must then be conducted to identify potential weaknesses in the system and to obtain first-order assessments of the ability of the system to meet its operational requirements. Given this information, vital vulnerability concerns can be identified and distilled into the critical Live Fire Test issues. Depending upon the system, there are several sources for Live Fire Test

Suggested Citation:"3 The OSD Live Fire Test and Evaluation Program." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

issues. For example, if the system is designed to replace an existing system, a Live Fire Test issue would be, How does the vulnerability of the new system compare with that of the existing system?

The next step in the planning process is the development of a strategy for conduct of the Live Fire Test program. This strategy shall include the establishment of measures of evaluation, procedures for the evaluation of the results, the data requirements, and the test objectives. This strategy shall be a part of the Test and Evaluation Master Plan.

Provision must next be made for the integration of the Live Fire Test strategy into the program plan for the development of the system. The Program Manager will ensure that the overall program plan, budget, and schedule provide adequate resources to assure a successful Live Fire Test program.

A detailed Live Fire Test plan is required that fully describes the tests to be conducted and the criteria for evaluating test results. This test plan shall be in sufficient detail to ensure that the tests will satisfy the strategy objectives. The test plan is subject to review and comment by the Director, Live Fire Test.

The conduct of the tests is to be witnessed by the Live Fire Test Office, which does an independent evaluation of the results. The Services prepare a test report that is reviewed by the Live Fire Test Office. The test report is then forwarded to the Secretary of Defense with the independent OSD evaluation of the results, and from there it is sent to the interested congressional committees.

The 1988 LFT&E Guidelines and the 1989 LFT&E Planning Guide are the current written guidance provided to the Services and their Program Managers to assist them in the planning and conduct of a Live Fire Test program that is guaranteed to satisfy the requirements of the OSD and, through the law, the Congress. In addition to this written material, the staff of the Live Fire Test office is available for regular consultation with the program offices in the preparation of the strategy for the TEMP and of the detailed test plan. Other methods used by the LFT Office to communicate the LFT program to the testing and evaluation (T&E) and acquisition communities include full membership for the LFT Director on the Defense Acquisition Board Committees, periodic one-on-one discussions with the Services’ Live Fire Test gate keepers, introduction of the program into the curriculum of the Defense System Management College, preparation of a video containing an overview of the program, public testimony before congressional defense subcommittees, sponsorship of six Live Fire Lessons Learned Workshops, participation in more than 25 T&E symposia, more than 40 on-site overview question-and-answer presentations to the major defense contractors, and publication in the open literature of more than 30 articles on Live Fire Testing (O’Bryon, 1991).

Controversy Regarding the Definition of “Realistic Survivability Testing”

The committee is concerned that the written guidance does not provide sufficient detail, particularly with respect to the full-scale tests, to ensure that the Program Manager can satisfy the requirements of the OSD policy and the law and also design a cost-effective test plan that will ensure system requirements are satisfied. Furthermore, the committee is concerned about the official status, or lack thereof, of the LFT&E Guidelines and Planning Guide.1

In addition to the questions regarding the level of detail and the current status of the guidance, there is a question regarding the interpretation of specific mandates contained in the Live Fire Test law. A major issue apparently exists between the Services and OSD, and indeed within OSD itself, regarding the definition and meaning of “realistic survivability testing” given in the law. The definition has been interpreted by the 1988 Live Fire Test and Evaluation Guidelines as follows:

  1. Live Fire Test: A test event within an overall LFT&E program which involves the firing of actual munitions at target components, target subsystems, target subassemblies, and/or full-scale targets to examine personnel casualty, vulnerability, and/or lethality issues.

  2. Full-up Testing: Firings against a full-scale target containing all the dangerous materials (e.g., ammunition, fuel, hydraulic fluids, etc.), system parts (e.g., electrical lines with operating voltages and currents applied, hydraulic lines containing appropriate fluids at operating pressures, etc.), and stowage items normally found on that target when operating in combat. Full-up testing includes firings against fullup components, full-up subsystems, full-up subassemblies, or full-up systems. The term “Full-up Testing” is synonymous with “realistic survivability testing” or “realistic lethality testing” as defined in the legislation covering Live Fire Testing.

The phrasing of these definitions has been interpreted by some to imply that full-scale, full-up Live Fire Tests do not have to be conducted (i.e., OSD policy is satisfied by Live Fire Tests only on sub-scale targets, such as major portions or subassemblies of an aircraft). The essence of the problem is the phrase in the law that states that realistic survivability testing means “testing for vulnerability of the system by.” What is the “system?” Does testing only one major subassembly constitute testing for vulnerability of the system? If it does, as some believe, then no waiver from the mandated system Live Fire Testing is necessary.

1

If action is taken to assign official status to these documents, they should first be modified to ensure that there is no ambiguity in the requirements for full-scale testing (see discussion below). One briefer indicated to the committee that guidelines were good for only one year and, beyond that, the “official status” would not be valid.

Suggested Citation:"3 The OSD Live Fire Test and Evaluation Program." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×

Although these definitions in the LFT&E Guidelines are apparently the official guidance on realistic survivability testing provided to the Services’ Program Managers, the Live Fire Test Office does not fully accept them. It believes the Guidelines make too liberal an interpretation of the intent of the Congress. It believes that the law requires the testing of complete systems, not just sub-systems or mock-ups, fully configured for combat. However, it appears to recognize that this can be unreasonably expensive and impractical, and that circumstances exist in which the testing of only subscale targets is justified. In those situations, it believes a waiver should be requested.

The committee believes, as stated in Chapter 1, that the Live Fire Test law requires that full-scale, full-up tests be conducted on covered systems, unless a waiver is granted. Furthermore, it believes that the definitions in the 1988 Guidelines and the guidance given in the 1989 Planning Guide are not sufficiently clear as to the law’s requirement that full-scale, full-up testing must be conducted. As a consequence of this misunderstanding, the Services have proceeded with sub-scale Live Fire Test programs on several weapon systems without making a provision for testing a complete system and without asking for a waiver because of the belief that no full-scale, full-up testing was required if early tests on sub-scale targets showed no design weaknesses.2

Conclusion

The committee believes that the Services, when they consider that full-scale, full-up Live Fire Tests are unreasonably expensive and impractical, should ask for a waiver. There should be no stigma attached to a waiver if a strong case can be made for one. According to the law, ''the Secretary of Defense may waive the application of the survivability and lethality tests of this section to a covered system, munitions program, or missile program if the Secretary, before the system enters full-scale engineering development, certifies to Congress that live-fire testing of such system or program would be unreasonably expensive and impractical” (U.S. Congress, 1986–1989). If a waiver is granted, the law states that “the Secretary shall include with any such certification a report explaining how the Secretary plans to evaluate the survivability or the lethality of the system or program and assessing possible alternatives to realistic survivability testing of the system or program” (U.S. Congress, 1986–1989). Thus, requesting and receiving a waiver from the requirement for a full-scale, full-up test program do not eliminate the requirement for vulnerability assessment; only for the full-scale, full-up testing portion of the assessment.

References

• Deputy Director, Defense Research and Engineering (Test & Evaluation), 1987. Live Fire Test and Evaluation (LFT&E Guidelines), Live Fire Testing Office.

• Deputy Director, Defense Research and Engineering (Test & Evaluation), 1989. Live Fire Test and Evaluation Planning Guide, Live Fire Testing Office.

• O’Bryon, James F., 1987. Live Fire Testing: the Legislation and Its Impact,” Army Research, Development, & Acquisition Bulletin, May-June.

• O’Bryon, James F., 1991. Presentation made to the Committee on Weapons Effects on Airborne Systems, July 24.

• Office of the Secretary of Defense (OSD), 1988. Live Fire Test and Evaluation Guidelines, Test and Evaluation Committee, Memorandum for Assistant Secretary of the Army (Research Development and Acquisition), Assistant Secretary of the Navy (Research Development and Acquisition), and Assistant Secretary of the Air Force (Acquisition).

• U.S. Congress, 1986–1989. Survivability and Lethality Testing of Major Systems, DoD Authorization Acts, FY86—Sec. 123, FY87—Sec. 910, FY88–89—Sec. 802.

2

These programs are examined in Chapter 4.

Suggested Citation:"3 The OSD Live Fire Test and Evaluation Program." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 38
Suggested Citation:"3 The OSD Live Fire Test and Evaluation Program." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 39
Suggested Citation:"3 The OSD Live Fire Test and Evaluation Program." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 40
Suggested Citation:"3 The OSD Live Fire Test and Evaluation Program." National Research Council. 1993. Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program. Washington, DC: The National Academies Press. doi: 10.17226/12470.
×
Page 41
Next: 4 Live Fire Test Programs of the Three Services, and Views of the Test Community and Industry »
Vulnerability Assessment of Aircraft: A Review of the Department of Defense Live Fire Test and Evaluation Program Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!