Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 11
1
Introduction
This report provides an assessment of the U.S. Army's planned ini-
tial operational test of the Stryker family of vehicles. It focuses on
two aspects of the test design and evaluation: (1) the measures of
performance and effectiveness used to compare the force equipped with the
Stryker with a baseline force and (2) whether the current operational test
design is consistent with state-of-the-art methods.
ARMY'S NEED FOR AN INTERIM
ARMORED VEHICLE (STRYKER)
The United States Army anticipates increases in the number and types
of asymmetric threats and will be required to support an expanding variety
of missions (including military operations in urban terrain and operations
other than war) that demand an effective combination of rapid deploy-
ability, information superiority, and coordination of awareness and action.
In order to respond to these threats and mission requirements, the Army
has identified the need for a future combat system that leverages the capa-
bilities of advancing technologies in such areas as vehicle power, sensors,
weaponry, and information gathering and sharing. It will take years to
develop and integrate these technologies into weapon systems that meet the
needs of the Army of the future. The Army has therefore established a
three-pronged plan to guide the transition of its weapons and forces, as
illustrated in Figure 1-1.
11
OCR for page 12
12
Legacy
Force
Objective
Force
Interim
Force
IMPROVED OPERATIONAL TESTING AND EVALUATION
1
l
l
l
,
1 1
, I
2000 First 2003
Interim
BCT
l
First Unit
Equipped
Objective
(°~
if_
FIGURE 1-1 Army plans for transition to the Objective Force equipped with the
future combat system.
Acronyms: B CT, brigade combat team; R&D, research and development; SOT, sci-
ence and technology; THE, test and evaluation.
SOURCE: ATEC briefing to the panel.
The Army intends to use this transition plan to sustain and upgrade
(but not expand) its existing weapon systems, which are characterized as its
Legacy Force. Heavy armored units, a key element of the Legacy Force, are
not, however, adequate to address the challenge of rapid deployability
around the globe.
An immediate and urgent need exists for an air transportable Interim
Force capable of deployment to anywhere on the globe in a combat-ready
configuration. Until the future combat system is developed and fully fielded
to support the Objective Force (the Army of the future), the Army intends
to rely on an Interim Force, the critical warfighting component of which is
the Interim Brigade Combat Team (IBCT). The mobility of the IBCT is
to be provided by the Stryker vehicle (until recently referred to as the in-
terim armored vehicle).
Stryker Configurations
The range of tasks to be accomplished by the IBCT calls for the Stryker
to be not just a single vehicle, but a family of vehicles that are air transport-
OCR for page 13
INTRODUCTION
13
able, are capable of immediate employment upon arrival in the area of
operations, and have a great degree of commonality in order to decrease its
logistical "footprint." Table 1-1 identifies the various Stryker vehicle con-
figurations (as identified in the Stryker system evaluation plane, the key
government-furnished equipment (GFE) items that will be integrated into
the configuration, and the role of each configuration.
Stryker Capabilities
The Army has identified two different but clearly dependent capability
requirements for the Stryker-supported IBCT: operational capabilities for
the IBCT force that will rely on the Stryker, and system capabilities for the
Stryker family of vehicles.
IBCT Operational Capabilities
The Army's Operational Requirements Document for the Stryker
(ORD, 2000) defines the following top-level requirement for the IBCT:
The IBCT is a full spectrum, combat force. It has utility, confirmed through
extensive analysis, in all operational environments against all projected future
threats, but it is designed and optimized primarily for employment in small
scale contingency (SSC) operations in complex and urban terrain, confront-
ing low-end and mid-range threats that may employ both conventional and
asymmetric capabilities. The IBCT deploys very rapidly, executes early entry,
and conducts effective combat operations immediately on arrival to prevent,
contain, stabilize, or resolve a conflict through shaping and decisive opera-
tions (section 1.a.~3~) . . . As a full spectrum combat force, the IBCT is
capable of conducting all major doctrinal operations including offensive, de-
fensive, stability, and support actions.... Properly integrated through a mo-
bile robust C4ISR network, these core capabilities compensate for platform
limitations that may exist in the close fight, leading to enhanced force effec-
tiveness (section 1.a.~4~.
System Capabilities
Each configuration of the Stryker vehicle must be properly integrated
. . . ~ . . . .
Wit. ~ sensing, information processing, communications, weapons, ant
other essential GFE that has been developed independently of the Stryker.
The Army notes that the Stryker-GFE system must provide a particular
capability, termed situation awareness, to offset the above-mentioned plat-
r 1-
torm llmltatlons:
OCR for page 14
14
IMPROVED OPERATIONAL TESTING AND EVALUATION
TABLE 1-1 Stryker Configurations
Co nfiguratio n Government- Furnished
Name Equipment Role
Infantry carrier Force XXI Battle Command Carry and protect a
vehicle Brigade and Below (FBCB2), nine-man infantry
Enhanced Position Location and squad and a crew of
Reporting System (EPLRS), two personnel.
Global Positioning System
(GPS), Thermal Weapon Sight
(TWS-H)
Mortar carrier M121 120-mm mortar, XM95 Provide fire support to
Mortar Fire Control System maneuver forces.
Antitank guided TOW II missile system, TOW Defeat armored threats.
. . ...
mlssl. .e vet alcle acquisition system
Reconnaissance Long Range Advanced Scout Enable scouts to
vehicle Surveillance System (LRAS3) perform reconnaissance
and surveillance.
Fire support Mission Equipment Package Provide automation-
vehicle (MEP), Handheld Terminal Unit enhanced target
(HTU), Common Hardware- acquisition,
Software- Lightweight Computer identification, and
Unit (CHS-LCU) detection;
,%
communicate lore
support information.
The IBCT will offset the lethality and survivability limitations of its plat-
forms through the holistic integration of all other force capabilities, particu-
larly the internetted actions of the combined arms company teams. The
mounted systems equipped with Force XXI Battle Command, Brigade and
Below (FBCB2) and other enhancements provide the IBCT a larger
internetted web of situational awareness extending throughout the IBCT area
of operations. The synergistic effects achieved by internetting highly trained
soldiers and leaders with platforms and organizational design enable the force
to avoid surprise, develop rapid decisions, control the time and place to en-
gage in combat, conduct precision maneuver, shape the battlespace with pre-
cision fires and effects, and achieve decisive outcomes. (ORD section 1.a.~7~.
The Stryker ORD specifies several performance requirements that ap-
OCR for page 15
INTRODUCTION
TABLE 1-1 Continued
15
Configuration
Name
Government- Furnished
Equipment
Role
Engineer squad
vehicle
Commander's
vehicle
Medical evacuation
vehicle
Nuclear, biological,
chemical (NBC)
reconnaissance
vehicle
Mobile gun system
FBCB2, EPLRS, GPS, I~WS-H
All Source Analysis System
(ASAS), Advanced Field Artillery
Tactical Data System Command,
Control, Communications and
Computer (AFATDS Cal.
MC-4 Medic's Aide
NBC Sensor Suite
Eyesafe Laser Range Finder (ELRF)
Neutralize and mark
obstacles, detect mines,
. .
transport engineering
squad.
Provide command,
control,
. . .
communications, and
computer attributes to
enable commanders to
direct the battle.
Enable recovery and
evacuation of casualties.
Perform reconnaissance
missions in NBC
environment; detect
NBC conditions.
Provide weapons fire to
support assaulting
· r
1ntantry.
ply across configurations of the Stryker system. Key performance require-
ments, defined in more detail in the ORD, are that the Stryker vehicles
must:
· maximize commonality of components and subcomponents across
few .
configurations;
· possess an "internetted interoperable capability" that enables it to
host and integrate existing and planned Army command, control, commu-
nications, computers, intelligence, surveillance, and reconnaissance
(C4ISR) systems;
· be transportable in a C-130 aircraft;
OCR for page 16
16
IMPROVED OPERATIONAL TESTING AND EVALUATION
· have the ability to operate effectively 24 hours per day, including at
night, in inclement weather, and during other periods of limited visibility
in hot, temperate, and cold climates;
· be mobile, demonstrable in terms of specified hard surface speeds,
cross-country mobility, cruising range, ability to transverse obstacles and
gaps, and maneuverability;
· possess the capability of sustainability, indicated by specified abili-
ties to tow and be towed, to be started with assistance, to be refueled rap-
idly, and to provide auxiliary power in the event of loss of primary power;
· be survivable, as evidenced by the ability to achieve specified accel-
eration, to provide protection to its crew and equipment, to accept add-on
armor, to mount weapons, and to rapidly self-obscure;
· permit the survivability of its crew in external environments with
nuclear, biological, and chemical contamination, by providing warning or
protection;
· possess the capability of lethality, demonstrated by its ability to in-
flict lethal damage on opposing forces and weapon systems;
· satisfy specified requirements for logistics and readiness, which con-
tribute to its fundamental requirement to maintain the force in the field;
· be transportable by air and by rail;
· operate reliably (i.e., without critical failures) for a specified period
of time and be maintainable within a specified period of time when failures
do occur.
The Army Test and Evaluation Command (ATEC) has been assigned
the mission of testing, under operationally realistic conditions, and evaluat-
ing the extent to which the IBCT force equipped with the Stryker system
(IBCT/Stryker) meets its requirements, compared with a baseline force,
the current Light Infantry Brigade (LIB), which will be augmented with
transportation assets appropriate to assigned missions.
Although we have major concerns about the appropriateness of using
the LIB as an alternative comparison force, because our primary responsi-
bility is to address broader statistical and test design issues, we have taken
this choice of baseline to be a firm constraint.
PANEL CHARGE
The Stryker will soon be entering an 1 8-month period of operational
test and evaluation to determine whether it is effective and suitable to enter
OCR for page 17
INTRODUCTION 17
into full-rate production, and how the vehicles already purchased can best
be used to address Army needs. Typical of the operational test and evalua-
tion of a major defense acquisition program, the test and evaluation of the
Stryker is an extremely complicated undertaking involving several separate
test events, the use of modeling and simulation, and a wide variety of re-
quirements that need to be satisfied.
Reacting to a very high level of congressional interest in the Stryker
program, ATEC must develop and use an evaluation approach that applies
statistical rigor to determining the contribution of the Stryker to the IBCT
mission, as well as the effectiveness of the IBCT itself. Affirming the value
of obtaining an independent assessment of its approach, and desiring assis-
tance in developing innovative measures of effectiveness, ATEC requested
that the National Research Council's (NRC) Committee on National Sta-
tistics convene a panel of experts to examine its plans for the operational
test design and subsequent test and evaluation for the IBCT/Stryker. This
resulted in the formation of the Panel on Operational Test Design and
Evaluation of the Interim Armored Vehicle. The panel was specifically
charged to examine three aspects of the operational test and evaluation of
the IBCT/Stryker:
1. the measures of performance and effectiveness used to compare the
IBCT force equipped with the Stryker system to the baseline, the LIB, and
the measures used to assess the extent to which the Stryker system meets its
requirements;
2. the design of the operational test, to determine the extent to which
the design is consistent with state-of-the art methods in statistical experi-
mental design; and
3. the applicability of combining information models, as well as of
combining information from testing and field use of related systems and
from developmental test results for the Stryker, with operational test results
for the Stryker.
The panel was also asked to identify alternative measures (e.g., of situ-
ation awareness) and experimental designs that could better reflect the ad-
vantages and disadvantages of the IBCT force equipped with the Stryker
system relative to the LIB force. In addition, the panel was asked to address
the use of modeling and simulation as part of the program evaluation and
the analysis strategy proposed for the evaluation of the IBCT/Stryker.
OCR for page 18
18
IMPROVED OPERATIONAL TESTING AND EVALUATION
PANEL APPROACH
In its 1998 report Statistics, Testing, and Defense Acquisition:
New Approaches and Methodological Improvements, the NRC Committee
on National Statistics' Panel on Testing and Evaluating Defense Systems
established broad perspectives and fundamental principles applicable to
the examination of statistical aspects of operational testing (National
Research Council, 1998a). Our panel has adopted the findings, conclu-
sions, and recommendations of that report as a key starting point for our
deliberations.
We also reviewed in detail all key government documents pertaining to
the operational testing of the IBCT/Stryker, including:
the Operational Requirements Document,
· the System Evaluation Plan,
· the Test and Evaluation Master Plan,
· the Organizational and Operational Description,
· the Failure Definition and Scoring Document,
· the Mission Needs Statement,
· the Operational Mode Summary and Mission Profile, and
sample Operational Orders applicable to operational tests.
With the cooperation of the management and staff of ATEC, the panel
conducted two forums and two subgroup meetings at which ATEC staff
presented, in response to panel queries: descriptions of measures of effec-
tiveness, suitability, and survivability under consideration for the initial
operational test; details of the proposed experimental design; planned use
of modeling and simulation; and planned methods of analysis of data that
will result from the testing. At these forums, panel members and ATEC
staff engaged in interactive discussion of proposed and alternative mea-
sures, test designs, and analytical methods. At the invitation of the panel,
two forums one on measures and one on test designs were attended by
representatives from the Office of the Director, Operational Test and Evalu-
ation; the Institute for Defense Analysis; the U.S. Government Accounting
Office; and the U.S. Military Academy at West Point. Beyond the specific
recommendations and conclusions presented here, it is our view that the
open and pointed discussions have created a process that in itself has had a
salutary influence on the decision making and design for the testing of the
Stryker system of vehicles and the IBCT.
OCR for page 19
INTRODUCTION
19
This report summarizes the panel's assessment regarding (1) the mea-
sures of performance and effectiveness used to compare the IBCT force
equipped with the Stryker system with the baseline force and the extent to
which the Stryker system meets its requirements and (2) the experimental
design of the operational test. This report also addresses measures for situ-
ation awareness, alternative measures for force effectiveness, analysis strate-
gies, and some issues pertaining to modeling and simulation. Box 1-1
presents a number of terms used in operational testing.
After additional forums and deliberations, the panel intends to prepare
a second report that addresses the applicability of combining information
from other sources with that from the IBCT/Stryker initial operational
test. Those sources include developmental tests for the Stryker, testing and
field experience with related systems, and modeling and simulation. Our
OCR for page 20
20
IMPROVED OPERATIONAL TESTING AND EVALUATION
OCR for page 21
INTRODUCTION
21
final report will incorporate both Phase I and II reports and any additional
developments.
Finally, we wish to make clear that the panel was constituted to address
the statistical questions raised by the selection of measures of performance
and measures of effectiveness, and the selection of an experimental design,
given the need to evaluate Stryker and the IBCT in scenarios identified in
the OMS/MP. A number of other important issues (about which the panel
provides some commentary) lie outside the panel's charge and expertise.
These include an assessment of (a) the selection of the baseline system to
compare with Stryker, (b) the problems raised by the simultaneous evalua-
tion of the Stryker vehicle and the IBCT system that incorporates it, (c)
whether the operational test can definitively answer specific tactical ques-
tions, such as the degree to which the increased vulnerability of Stryker is
offset by the availability of greater situational awareness, and (~) whether or
not scenarios to be acted out by OPFOR represent a legitimate test suite.
Let us elaborate each of these ancillary but important issues.
The first is whether the current choice of a baseline system (or multiple
baselines) is best from a military point of view, including whether a baseline
system could have been tested taking advantage of the IBCT infrastructure,
to help understand the value of Stryker without the IBCT system). It does
not seem to be necessary to require that only a system that could be trans-
ported as quickly as Stryker could serve as a baseline for comparison.
The second issue (related to the first) is the extent to which the current
test provides information not only about comparison of the IBCT/Stryker
system with a baseline system, but also about comparison of the Stryker
suite of vehicles with those used in the baseline. For example, how much
more or less maneuverable is Stryker in rural versus urban terrain and what
impact does that have on its utility in those environments? These questions
require considerable military expertise to address.
The third issue is whether the current operational test design can pro-
vide adequate information on how to tactically employ the IBCT/Stryker
system. For example, how should the greater situational awareness be taken
advantage of, and how should the greater situational awareness be balanced
against greater vulnerability for various types of environments and against
various threats? Clearly, this issue is not fundamentally a technical statisti-
cal one, but is rather an essential feature of scenario design that the panel
was not constituted to evaluate.
The final issue (related to the third) is whether the various missions,
types of terrain, and intensity of conflict are the correct choices for opera-
OCR for page 22
22
IMPROVED OPERATIONAL TESTING AND EVALUATION
tional testing to support the decision on whether to pass Stryker to full-rate
production. One can imagine other missions, types of terrain, intensities,
and other factors that are not varied in the current test design that might
have an impact on the performance of Stryker, the baseline system, or both.
These factors include temperature, precipitation, the density of buildings,
the height of buildings, types of roads, etc. Moreover, there are the serious
problems raised by the unavailability of add-on armor for the early stages of
the operational test. The panel has been obligated to take the OMS/MP as
given, but it is not clear whether additional factors that might have an
important impact on performance should have been included as test fac-
tors. All of these issues are raised here in order to emphasize their impor-
tance and worthiness for consideration by other groups better constituted
to address them.
Thus, the panel wishes to make very clear that this assessment of the
operational test as currently designed reflects only its statistical merits. It is
certainly possible that the IBCT/Stryker operational test may be deficient
in other respects, some of them listed above, that may subordinate the
statistical aspects of the test. Even if the statistical issues addressed in this
report were to be mitigated, we cannot determine whether the resulting
operational test design would be fully informative as to whether Stryker
should be promoted to full-rate production.
Representative terms from entire chapter:
operational testing
