Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
1 Introduction This report provides an assessment of the U.S. Army's planned ini- tial operational test of the Stryker family of vehicles. It focuses on two aspects of the test design and evaluation: (1) the measures of performance and effectiveness used to compare the force equipped with the Stryker with a baseline force and (2) whether the current operational test design is consistent with state-of-the-art methods. ARMY'S NEED FOR AN INTERIM ARMORED VEHICLE (STRYKER) The United States Army anticipates increases in the number and types of asymmetric threats and will be required to support an expanding variety of missions (including military operations in urban terrain and operations other than war) that demand an effective combination of rapid deploy- ability, information superiority, and coordination of awareness and action. In order to respond to these threats and mission requirements, the Army has identified the need for a future combat system that leverages the capa- bilities of advancing technologies in such areas as vehicle power, sensors, weaponry, and information gathering and sharing. It will take years to develop and integrate these technologies into weapon systems that meet the needs of the Army of the future. The Army has therefore established a three-pronged plan to guide the transition of its weapons and forces, as illustrated in Figure 1-1. 11
12 Legacy Force Objective Force Interim Force IMPROVED OPERATIONAL TESTING AND EVALUATION 1 l l l , 1 1 , I 2000 First 2003 Interim BCT l First Unit Equipped Objective (°~ if_ FIGURE 1-1 Army plans for transition to the Objective Force equipped with the future combat system. Acronyms: B CT, brigade combat team; R&D, research and development; SOT, sci- ence and technology; THE, test and evaluation. SOURCE: ATEC briefing to the panel. The Army intends to use this transition plan to sustain and upgrade (but not expand) its existing weapon systems, which are characterized as its Legacy Force. Heavy armored units, a key element of the Legacy Force, are not, however, adequate to address the challenge of rapid deployability around the globe. An immediate and urgent need exists for an air transportable Interim Force capable of deployment to anywhere on the globe in a combat-ready configuration. Until the future combat system is developed and fully fielded to support the Objective Force (the Army of the future), the Army intends to rely on an Interim Force, the critical warfighting component of which is the Interim Brigade Combat Team (IBCT). The mobility of the IBCT is to be provided by the Stryker vehicle (until recently referred to as the in- terim armored vehicle). Stryker Configurations The range of tasks to be accomplished by the IBCT calls for the Stryker to be not just a single vehicle, but a family of vehicles that are air transport-
INTRODUCTION 13 able, are capable of immediate employment upon arrival in the area of operations, and have a great degree of commonality in order to decrease its logistical "footprint." Table 1-1 identifies the various Stryker vehicle con- figurations (as identified in the Stryker system evaluation plane, the key government-furnished equipment (GFE) items that will be integrated into the configuration, and the role of each configuration. Stryker Capabilities The Army has identified two different but clearly dependent capability requirements for the Stryker-supported IBCT: operational capabilities for the IBCT force that will rely on the Stryker, and system capabilities for the Stryker family of vehicles. IBCT Operational Capabilities The Army's Operational Requirements Document for the Stryker (ORD, 2000) defines the following top-level requirement for the IBCT: The IBCT is a full spectrum, combat force. It has utility, confirmed through extensive analysis, in all operational environments against all projected future threats, but it is designed and optimized primarily for employment in small scale contingency (SSC) operations in complex and urban terrain, confront- ing low-end and mid-range threats that may employ both conventional and asymmetric capabilities. The IBCT deploys very rapidly, executes early entry, and conducts effective combat operations immediately on arrival to prevent, contain, stabilize, or resolve a conflict through shaping and decisive opera- tions (section 1.a.~3~) . . . As a full spectrum combat force, the IBCT is capable of conducting all major doctrinal operations including offensive, de- fensive, stability, and support actions.... Properly integrated through a mo- bile robust C4ISR network, these core capabilities compensate for platform limitations that may exist in the close fight, leading to enhanced force effec- tiveness (section 1.a.~4~. System Capabilities Each configuration of the Stryker vehicle must be properly integrated . . . ~ . . . . Wit. ~ sensing, information processing, communications, weapons, ant other essential GFE that has been developed independently of the Stryker. The Army notes that the Stryker-GFE system must provide a particular capability, termed situation awareness, to offset the above-mentioned plat- r 1- torm llmltatlons:
14 IMPROVED OPERATIONAL TESTING AND EVALUATION TABLE 1-1 Stryker Configurations Co nfiguratio n Government- Furnished Name Equipment Role Infantry carrier Force XXI Battle Command Carry and protect a vehicle Brigade and Below (FBCB2), nine-man infantry Enhanced Position Location and squad and a crew of Reporting System (EPLRS), two personnel. Global Positioning System (GPS), Thermal Weapon Sight (TWS-H) Mortar carrier M121 120-mm mortar, XM95 Provide fire support to Mortar Fire Control System maneuver forces. Antitank guided TOW II missile system, TOW Defeat armored threats. . . ... mlssl. .e vet alcle acquisition system Reconnaissance Long Range Advanced Scout Enable scouts to vehicle Surveillance System (LRAS3) perform reconnaissance and surveillance. Fire support Mission Equipment Package Provide automation- vehicle (MEP), Handheld Terminal Unit enhanced target (HTU), Common Hardware- acquisition, Software- Lightweight Computer identification, and Unit (CHS-LCU) detection; ,% communicate lore support information. The IBCT will offset the lethality and survivability limitations of its plat- forms through the holistic integration of all other force capabilities, particu- larly the internetted actions of the combined arms company teams. The mounted systems equipped with Force XXI Battle Command, Brigade and Below (FBCB2) and other enhancements provide the IBCT a larger internetted web of situational awareness extending throughout the IBCT area of operations. The synergistic effects achieved by internetting highly trained soldiers and leaders with platforms and organizational design enable the force to avoid surprise, develop rapid decisions, control the time and place to en- gage in combat, conduct precision maneuver, shape the battlespace with pre- cision fires and effects, and achieve decisive outcomes. (ORD section 1.a.~7~. The Stryker ORD specifies several performance requirements that ap-
INTRODUCTION TABLE 1-1 Continued 15 Configuration Name Government- Furnished Equipment Role Engineer squad vehicle Commander's vehicle Medical evacuation vehicle Nuclear, biological, chemical (NBC) reconnaissance vehicle Mobile gun system FBCB2, EPLRS, GPS, I~WS-H All Source Analysis System (ASAS), Advanced Field Artillery Tactical Data System Command, Control, Communications and Computer (AFATDS Cal. MC-4 Medic's Aide NBC Sensor Suite Eyesafe Laser Range Finder (ELRF) Neutralize and mark obstacles, detect mines, . . transport engineering squad. Provide command, control, . . . communications, and computer attributes to enable commanders to direct the battle. Enable recovery and evacuation of casualties. Perform reconnaissance missions in NBC environment; detect NBC conditions. Provide weapons fire to support assaulting · r 1ntantry. ply across configurations of the Stryker system. Key performance require- ments, defined in more detail in the ORD, are that the Stryker vehicles must: · maximize commonality of components and subcomponents across few . configurations; · possess an "internetted interoperable capability" that enables it to host and integrate existing and planned Army command, control, commu- nications, computers, intelligence, surveillance, and reconnaissance (C4ISR) systems; · be transportable in a C-130 aircraft;
16 IMPROVED OPERATIONAL TESTING AND EVALUATION · have the ability to operate effectively 24 hours per day, including at night, in inclement weather, and during other periods of limited visibility in hot, temperate, and cold climates; · be mobile, demonstrable in terms of specified hard surface speeds, cross-country mobility, cruising range, ability to transverse obstacles and gaps, and maneuverability; · possess the capability of sustainability, indicated by specified abili- ties to tow and be towed, to be started with assistance, to be refueled rap- idly, and to provide auxiliary power in the event of loss of primary power; · be survivable, as evidenced by the ability to achieve specified accel- eration, to provide protection to its crew and equipment, to accept add-on armor, to mount weapons, and to rapidly self-obscure; · permit the survivability of its crew in external environments with nuclear, biological, and chemical contamination, by providing warning or protection; · possess the capability of lethality, demonstrated by its ability to in- flict lethal damage on opposing forces and weapon systems; · satisfy specified requirements for logistics and readiness, which con- tribute to its fundamental requirement to maintain the force in the field; · be transportable by air and by rail; · operate reliably (i.e., without critical failures) for a specified period of time and be maintainable within a specified period of time when failures do occur. The Army Test and Evaluation Command (ATEC) has been assigned the mission of testing, under operationally realistic conditions, and evaluat- ing the extent to which the IBCT force equipped with the Stryker system (IBCT/Stryker) meets its requirements, compared with a baseline force, the current Light Infantry Brigade (LIB), which will be augmented with transportation assets appropriate to assigned missions. Although we have major concerns about the appropriateness of using the LIB as an alternative comparison force, because our primary responsi- bility is to address broader statistical and test design issues, we have taken this choice of baseline to be a firm constraint. PANEL CHARGE The Stryker will soon be entering an 1 8-month period of operational test and evaluation to determine whether it is effective and suitable to enter
INTRODUCTION 17 into full-rate production, and how the vehicles already purchased can best be used to address Army needs. Typical of the operational test and evalua- tion of a major defense acquisition program, the test and evaluation of the Stryker is an extremely complicated undertaking involving several separate test events, the use of modeling and simulation, and a wide variety of re- quirements that need to be satisfied. Reacting to a very high level of congressional interest in the Stryker program, ATEC must develop and use an evaluation approach that applies statistical rigor to determining the contribution of the Stryker to the IBCT mission, as well as the effectiveness of the IBCT itself. Affirming the value of obtaining an independent assessment of its approach, and desiring assis- tance in developing innovative measures of effectiveness, ATEC requested that the National Research Council's (NRC) Committee on National Sta- tistics convene a panel of experts to examine its plans for the operational test design and subsequent test and evaluation for the IBCT/Stryker. This resulted in the formation of the Panel on Operational Test Design and Evaluation of the Interim Armored Vehicle. The panel was specifically charged to examine three aspects of the operational test and evaluation of the IBCT/Stryker: 1. the measures of performance and effectiveness used to compare the IBCT force equipped with the Stryker system to the baseline, the LIB, and the measures used to assess the extent to which the Stryker system meets its requirements; 2. the design of the operational test, to determine the extent to which the design is consistent with state-of-the art methods in statistical experi- mental design; and 3. the applicability of combining information models, as well as of combining information from testing and field use of related systems and from developmental test results for the Stryker, with operational test results for the Stryker. The panel was also asked to identify alternative measures (e.g., of situ- ation awareness) and experimental designs that could better reflect the ad- vantages and disadvantages of the IBCT force equipped with the Stryker system relative to the LIB force. In addition, the panel was asked to address the use of modeling and simulation as part of the program evaluation and the analysis strategy proposed for the evaluation of the IBCT/Stryker.
18 IMPROVED OPERATIONAL TESTING AND EVALUATION PANEL APPROACH In its 1998 report Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements, the NRC Committee on National Statistics' Panel on Testing and Evaluating Defense Systems established broad perspectives and fundamental principles applicable to the examination of statistical aspects of operational testing (National Research Council, 1998a). Our panel has adopted the findings, conclu- sions, and recommendations of that report as a key starting point for our deliberations. We also reviewed in detail all key government documents pertaining to the operational testing of the IBCT/Stryker, including: the Operational Requirements Document, · the System Evaluation Plan, · the Test and Evaluation Master Plan, · the Organizational and Operational Description, · the Failure Definition and Scoring Document, · the Mission Needs Statement, · the Operational Mode Summary and Mission Profile, and sample Operational Orders applicable to operational tests. With the cooperation of the management and staff of ATEC, the panel conducted two forums and two subgroup meetings at which ATEC staff presented, in response to panel queries: descriptions of measures of effec- tiveness, suitability, and survivability under consideration for the initial operational test; details of the proposed experimental design; planned use of modeling and simulation; and planned methods of analysis of data that will result from the testing. At these forums, panel members and ATEC staff engaged in interactive discussion of proposed and alternative mea- sures, test designs, and analytical methods. At the invitation of the panel, two forums one on measures and one on test designs were attended by representatives from the Office of the Director, Operational Test and Evalu- ation; the Institute for Defense Analysis; the U.S. Government Accounting Office; and the U.S. Military Academy at West Point. Beyond the specific recommendations and conclusions presented here, it is our view that the open and pointed discussions have created a process that in itself has had a salutary influence on the decision making and design for the testing of the Stryker system of vehicles and the IBCT.
INTRODUCTION 19 This report summarizes the panel's assessment regarding (1) the mea- sures of performance and effectiveness used to compare the IBCT force equipped with the Stryker system with the baseline force and the extent to which the Stryker system meets its requirements and (2) the experimental design of the operational test. This report also addresses measures for situ- ation awareness, alternative measures for force effectiveness, analysis strate- gies, and some issues pertaining to modeling and simulation. Box 1-1 presents a number of terms used in operational testing. After additional forums and deliberations, the panel intends to prepare a second report that addresses the applicability of combining information from other sources with that from the IBCT/Stryker initial operational test. Those sources include developmental tests for the Stryker, testing and field experience with related systems, and modeling and simulation. Our
20 IMPROVED OPERATIONAL TESTING AND EVALUATION
INTRODUCTION 21 final report will incorporate both Phase I and II reports and any additional developments. Finally, we wish to make clear that the panel was constituted to address the statistical questions raised by the selection of measures of performance and measures of effectiveness, and the selection of an experimental design, given the need to evaluate Stryker and the IBCT in scenarios identified in the OMS/MP. A number of other important issues (about which the panel provides some commentary) lie outside the panel's charge and expertise. These include an assessment of (a) the selection of the baseline system to compare with Stryker, (b) the problems raised by the simultaneous evalua- tion of the Stryker vehicle and the IBCT system that incorporates it, (c) whether the operational test can definitively answer specific tactical ques- tions, such as the degree to which the increased vulnerability of Stryker is offset by the availability of greater situational awareness, and (~) whether or not scenarios to be acted out by OPFOR represent a legitimate test suite. Let us elaborate each of these ancillary but important issues. The first is whether the current choice of a baseline system (or multiple baselines) is best from a military point of view, including whether a baseline system could have been tested taking advantage of the IBCT infrastructure, to help understand the value of Stryker without the IBCT system). It does not seem to be necessary to require that only a system that could be trans- ported as quickly as Stryker could serve as a baseline for comparison. The second issue (related to the first) is the extent to which the current test provides information not only about comparison of the IBCT/Stryker system with a baseline system, but also about comparison of the Stryker suite of vehicles with those used in the baseline. For example, how much more or less maneuverable is Stryker in rural versus urban terrain and what impact does that have on its utility in those environments? These questions require considerable military expertise to address. The third issue is whether the current operational test design can pro- vide adequate information on how to tactically employ the IBCT/Stryker system. For example, how should the greater situational awareness be taken advantage of, and how should the greater situational awareness be balanced against greater vulnerability for various types of environments and against various threats? Clearly, this issue is not fundamentally a technical statisti- cal one, but is rather an essential feature of scenario design that the panel was not constituted to evaluate. The final issue (related to the third) is whether the various missions, types of terrain, and intensity of conflict are the correct choices for opera-
22 IMPROVED OPERATIONAL TESTING AND EVALUATION tional testing to support the decision on whether to pass Stryker to full-rate production. One can imagine other missions, types of terrain, intensities, and other factors that are not varied in the current test design that might have an impact on the performance of Stryker, the baseline system, or both. These factors include temperature, precipitation, the density of buildings, the height of buildings, types of roads, etc. Moreover, there are the serious problems raised by the unavailability of add-on armor for the early stages of the operational test. The panel has been obligated to take the OMS/MP as given, but it is not clear whether additional factors that might have an important impact on performance should have been included as test fac- tors. All of these issues are raised here in order to emphasize their impor- tance and worthiness for consideration by other groups better constituted to address them. Thus, the panel wishes to make very clear that this assessment of the operational test as currently designed reflects only its statistical merits. It is certainly possible that the IBCT/Stryker operational test may be deficient in other respects, some of them listed above, that may subordinate the statistical aspects of the test. Even if the statistical issues addressed in this report were to be mitigated, we cannot determine whether the resulting operational test design would be fully informative as to whether Stryker should be promoted to full-rate production.
