Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
THE NATIONAL ACADEMIES Advisers to [he Nation on Science, Engineering, and Medirine Division of Behavioral and Social Sciences and Education Committee on National Statistics Panel on Operational Test Design and Evaluation of the Interim Armored Vehicle (IAV) October 10, 2002 Frank John Apicella Technical Director Army Evaluation Center U.S. Army Test and Evaluation Command 4501 Forc3 Avenue Alexandria, VA 22302-1458 Dear Mr. Apicella: 500 Fifth Street, NW Washington, DC 20001 Phone: 202 334 3408 Fax: 202 334 3584 E-mail: jmcgee~nas.edu AS you know, at the request of the Army Test and Evaluation Command (ATEC) the Committee on National Statistics has convened a pane} to examine ATEC's plans for the operational test design and evaluation of the Interim Armored Vehicle, now referred to as the Stryker. The panel is currently engaged in its tasks of focusing on three aspects of the operational test design ant! evaluation ofthe Stryker: (~) the measures of performance and effectiveness user! to compare the Interim Brigade Combat Team (IBCT), equipped with the Stryker, against a baseline force; (2) whether the current operational test design is consistent with state-of-the-art methods in statistical experimental design; and (3) the applicability of models for combining information from testing and field use of related systems ant! from clevelopmental test results for the Stryker with operational test results for the Stryker. ATEC has asked the pane] to comment on ATEC's current plans ant! to suggest alternatives. The work performance plan includes the preparation of three reports: . . The first interim report (clue in November 2002) will address two topics: (~) the measures of performance and effectiveness used to compare the Stryker-equipped IBCT against the baseline force, ant! (2) whether the current operational test design is consistent with state-of-the-art methods in statistical experimental design. The second interim report (due in March 2003) will address the topic of the applicability of mo(lels for combining information from testing and field use of related systems and from developmental test results for the Stryker with operational test results for the Stryker. The final report (due in July 2003) will integrate the two interim reports and add any additional findings of the panel. The reports have been sequenced and timed for delivery to support ATEC's time- critical schedule for developing plans for designing and implementing operational tests and for performing analyses and evaluations of the test results. NATIONAL ACADEMY OF SCIENCES · NATIONAL ACADEMY OF ENGINEERING · INSTITUTE OF MEDICINE · NATIONAL RESEARCH COUNCIL
Two specific purposes of the initial operational test of the Stryker are to determine whether the Interim Brigade Combat Team (IBCT) equipped with the Stryker performs more effectively than a baseline force (Light Infantry Brigade), ant! whether the Stryker meets its performance requirements. The results of the initial operational test contribute to the Army's decisions of whether and how to employ the Stryker ant! the IBCT. The panel's first interim report will aciciress in cletai] factors relating to the effectiveness and performance of the Stryker- equippecl IBCT and of the Stryker; effective experimental designs and procedures for testing these forces and their systems uncler relevant operational conditions, missions, and scenarios; subjective and objective measures of performance and effectiveness for criteria of suitability, force effectiveness, and survivability; and analytical procedures and methods appropriate to assessing whether ant! why the Stryker-equipped IBCT compares well (or not well) against the baseline force, and whether ant} why the Stryker meets (or does not meet) its performance requirements. -1 r r ~ ~ ~ 1 - ~ -C7 -- - --- r- In the process of deliberations toward! proclucing the first interim report that will address this broad sweep of issues relevant to operational test design and to measures of performance ant! effectiveness, the pane] has cliscerned two issues with long lead times to which, in the opinion of the panel, ATEC should begin attending immediately, so that resources can be identified, mustered, and applied in time to address them: early development of a "strawman" (hypothetical draft) Test and Evaluation Report (which will support the development of measures and the test design as well as the subsequent analytical efforts) and the scheduling of test participation by the Stryker-equipped force and the baseline force so as to remove an obvious test confounder of different seasonal conditions. The general purpose of the initial operational test (JOT) is to provide information to decision makers about the utility of and the remaining challenges to the IBCT and the Stryker system. This information is to be generated through the analysis of TOT results. In order to highlight areas for which data are lacking, the panel strongly recommends that immediate effort be focused on specifying how the test data will be analyzes! to address relevant decision issues and questions. Specifically, a strawman Test Evaluation Report (TER) shouIct be prepared as if the TOT had been completed. It should include examples of how the representative data will be analyze(l, specific presentation formats (inclucling graphs) with expected results, insights one might develop from the data, draft recommendations, etc. The content of this strawman report should be based on the experience and intuition of the analysts ant! what they think the results of the TOT might look like. Overall, this couIct serve to provide a set of hypotheses that wouIc! be tested with the actual results. Preparation of this strawman TER will help ATEC assess those issues that cannot be informer} by the operational tests as currently planned, will expose areas for which needed data is lacking, ant! will allow appropriate revision of the current operational test plan. The current test design calls for the execution of the IBCT/Stryker vs. the opposing force (OPFOR) trials and the baseline vs. the OPFOR trials to be scheduled for different seasons. This design totally confounds time of year with the primary measure of interest: the difference in effectiveness between the baseline force and the IBCT/Stryker force. The pane! believes that the factors that are present in seasonal variations--weather, foliage density, light level, temperature, etc.--may have a greater effect on the differences between the measures of the two forces than
the abilities of the two forces themselves. We therefore recommend that serious consideration be given to scheduling these events as closely in time as possible. One way to address the potential confounding of seasonal affects, as well as possible effects of learning by blue forces and by the OPFOR, would be to intersperse activities of the baseline force ant! the TBCT/Stryker force over time. The pane] remains eager to assist ATEC in improving its plans and processes for operational test and evaluation of the IBCT/Stryker. We are grateful for the support and information you ant! your staff have consistently provided cluring our efforts to date. It is the panel's hope that delivering to you the recommendations in this brief letter in a timely fashion will encourage ATEC to begin drafting a strawman Test Evaluation Report in time to influence operational test activities and to implement the change in test plan to allow the compared forces to undergo testing in the same season. Sincerely yours Stephen Pollock, Chair Panel on Operational Test Design and Evaluation ofthe Interim Armored Vehicle Attachment: List of pane] members