Executive Summary

Biodetector validation is based on a range of metrics that should increase confidence in the system and its ability to meet specific performance goals. In the end the warfighter wants confidence in a system’s performance and the sponsor wants assurance that the funds are well spent. Both of these goals require managing risks.

In its Statement of Work (see Appendix) the committee was asked to “review current biological point-detection system testing protocols and integrated evaluation methodologies”, and “assess the feasibility and benefits of the DOT&E [Director, Operational Test and Evaluation] requirement for a systems-level, active agent testing capability.” Despite the importance of the Joint Biological Point Detection System (JBPDS) production decision, the committee is not convinced that the planned Whole System Live Agent Testing (WSLAT) facility alone will provide the information required for that decision. Without an integrated test and evaluation methodology and a suite of optimized and dependable simulants, the agent/simulant correlation measurements intended to be conducted in WSLAT and elsewhere will not be meaningful. Also, changes in threat requirements and unexpected results with live agents in WSLAT or other facilities will require dependable simulants and a consistent test methodology for resolution. The WSLAT performance requirements document equivocates on critical threat details, and was changed during the committee’s work.

A preliminary review and assessment of these issues was made by the committee in its interim report. In Chapter 1 of this report, the committee provides a general introduction and a summary of the Interim Report that addressed the first item of the committee’s Statement of Work.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Review of Testing and Evaluation Methodology for Biological Point Detectors: Abbreviated Summary Executive Summary Biodetector validation is based on a range of metrics that should increase confidence in the system and its ability to meet specific performance goals. In the end the warfighter wants confidence in a system’s performance and the sponsor wants assurance that the funds are well spent. Both of these goals require managing risks. In its Statement of Work (see Appendix) the committee was asked to “review current biological point-detection system testing protocols and integrated evaluation methodologies”, and “assess the feasibility and benefits of the DOT&E [Director, Operational Test and Evaluation] requirement for a systems-level, active agent testing capability.” Despite the importance of the Joint Biological Point Detection System (JBPDS) production decision, the committee is not convinced that the planned Whole System Live Agent Testing (WSLAT) facility alone will provide the information required for that decision. Without an integrated test and evaluation methodology and a suite of optimized and dependable simulants, the agent/simulant correlation measurements intended to be conducted in WSLAT and elsewhere will not be meaningful. Also, changes in threat requirements and unexpected results with live agents in WSLAT or other facilities will require dependable simulants and a consistent test methodology for resolution. The WSLAT performance requirements document equivocates on critical threat details, and was changed during the committee’s work. A preliminary review and assessment of these issues was made by the committee in its interim report. In Chapter 1 of this report, the committee provides a general introduction and a summary of the Interim Report that addressed the first item of the committee’s Statement of Work.

OCR for page 1
Review of Testing and Evaluation Methodology for Biological Point Detectors: Abbreviated Summary With additional data available to the committee, these issues are examined in further detail in Chapter 2. Additionally, the committee was asked in its Statement of Work to review the Whole System Live Agent Testing proposal provided under a study from the West Desert Test Center, Dugway Proving Ground. This review is provided in Chapter 3 of this report. In its overall evaluation the committee agrees with the concept of WSLAT, or a WSLAT-like facility, in which biological detectors will be evaluated against threat representative challenges. However, the committee concludes that the purposes and need for WSLAT must be clarified and defined more rigorously, since the proposed WSLAT program carries high schedule risk, high cost and questionable feasibility of adequate system performance. The WSLAT program does not present a clear plan for the testing to be done, data to be produced, or how the data will be analyzed and applied to the decision. These issues have not yet been addressed and included in schedule and budget plans for WSLAT. The committee concludes that in parallel with the planned WSLAT program, an essential matching effort is needed by the DOD to develop a well-defined testing and evaluation methodology. In Chapter 2, the committee provides the following three high-level recommendations (recommendations 2-1, 2-2, and 2-3), and justification relative to the Interim Report and information provided to the committee subsequent to the release of the Interim Report. Recommendation 2-1: A scientifically credible integrated test and evaluation methodology should be established and implemented to guide sensor validation and subsequent production decisions. This test and evaluation methodology is detailed further in recommendations 2-4, 2-5, and 2-6. Recommendation 2-2: Facilities are needed for validation of JBPDS and other biodetection systems, and should be developed in a phased manner where each phase adds to the ability to predict performance capability of the system under testing. This facility could be a WSLAT-like facility that evolves from the original concept to support changes in the test and evaluation methodology and to address shortcomings identified by the committee in its review. The recommended strategy is to work backwards from the integrated T&E plan to the data and analysis needed and then to specification of appropriate facilities and infrastructure. A mobile whole system T&E capability is recommended to address T&E in representative environments and backgrounds. Recommendation 2-3: An independent expert advisory committee should be created to provide guidance on JBPDS testing and subsequent detection system development.

OCR for page 1
Review of Testing and Evaluation Methodology for Biological Point Detectors: Abbreviated Summary Based on the collective expertise and experience of the committee, the technical challenges inherent in the development and implementation of evaluation methodology for the JBPDS warrant outside scientific advice through an independent team. Because this team could be composed of individuals both inside and outside DOD, more wide ranging scientific expertise can assist project managers in moving forward with additional confidence in the scientific rationale for the decisions that DOD needs to make. The integrated T&E plan should be externally peer reviewed as should any anticipated changes to the WSLAT proposal. Each of the following recommendations related to testing and evaluation methodology is developed in detail in Chapter 2. Recommendation 2-4: A suite of optimized and dependable simulants should be developed. Recommendation 2-5: Modeling and correlation analyses of components and whole systems should be developed. Recommendation 2-6: Testing and modeling should be constructed in complex, operationally realistic environments. With the exception of component testing of simulants and live agents, it is clear that Dugway Proving Ground does not currently have adequate test facilities. It is for this reason that the committee recommends that a Whole System Killed Related Agent Testing (WSKRAT) facility be constructed and used to begin implementation of the test methodology described above. Recommendation 2-7: A WSKRAT facility should be constructed that would permit the development of predictive models and the testing of the adequacy of killed related-strain agents as simulants for live agents. The use of killed related strain agents provide the best opportunity to both mimic live agents and to pass regulatory hurdles that may limit testing. Another advantage of such a facility is that WSKRAT would be a fully contained system with no open-air release, implying that some testing can begin immediately. In addition, by avoiding the use of live agents, it should be possible to construct a mobile WSKRAT facility that can test in realistic operational environments. Recommendation 2-8: A WSKRAT mobile test facility should be constructed to assess the ability of system models to predict whole system behavior in operational environments. While the committee is convinced that the development of WSKRAT is essential as part of a complete set of test and evaluation facilities, a WSLAT-like facility also will be necessary. However, without the needed data provided by a

OCR for page 1
Review of Testing and Evaluation Methodology for Biological Point Detectors: Abbreviated Summary graded approach that includes WSKRAT, a WSLAT itself will not represent an adequate test strategy. Recommendation 2-9: A test facility should be constructed for final validation of JBPDS or any future point detection system. In Chapter 3, the committee provides its evaluation of the WSLAT feasibility study, the second item in the committee’s Statement of Work, including the following recommendation and a summary of the justification developed more fully in Chapter 3. Recommendation 3-1: An analysis of alternative WSLAT design approaches that would include comparisons of the various key performance requirements in each design approach should be undertaken. The analysis should include a thorough discussion and analysis of risk, mitigation methods, and development of a sound risk mitigation plan. The committee is not optimistic that the proposed WSLAT approach will succeed. The design has major unresolved technical questions. The schedule is highly compressed and is already behind. The committee also concluded that risk identification, analysis, and proposed management were particularly weak. The committee’s observations on the feasibility study are presented in three groups. The first addresses the requirements analysis, the second addresses the scientific and engineering approach, and the third group of comments addresses regulatory and policy concerns. In addition to these comments specifically about the WSLAT feasibility study, the committee notes that a credible testing program requires more than testing facilities. Supporting methodology, materials (e.g., agents and simulants), and validated processes are also needed. To be useful, WSLAT should be operated according to an overall, integrated test and evaluation methodology. The live agents it will use must be validated and correlated with a suite of optimized and dependable simulants. Neither the methodology nor the suite of simulants is adequate, and WSLAT alone will not suffice. In Chapter 4, the committee suggests an alternative and complementary process for the testing and evaluation of biopoint detection systems in light of the recommendations and evaluations provided by the committee in this report. Assuming that simulants can be selected that mimic the biological agents1 of concern, the predictive capability of the system performance testing protocol increases in the following order (this list is in order of increasing confidence levels): 1   Biological agents are live microorganisms or toxins that can incapacitate or kill humans and animals, and damage crops.

OCR for page 1
Review of Testing and Evaluation Methodology for Biological Point Detectors: Abbreviated Summary Level 1 Component testing with simulants Level 2 Component testing with live agents Level 3 Whole system testing with simulants alone Level 4 Whole system testing with live agents Level 5 Whole system testing with simulants under environmentally representative conditions Level 6 Whole system testing with live agents under environmentally representative conditions Recommendation 4-1: Assuming good simulants exist, the committee recommends testing to proceed from component testing with simulants (Level 1), and live agents (Level 2), through whole system testing with simulants (Level 3), and then to whole system testing with simulants under environmentally representative conditions (Level 5). With an ideal simulant, there will be sufficient confidence in the simulant’s ability to faithfully mimic the agent(s) of concern that whole system live agent testing will not be required. If such simulants do not exist then Level 4 (WSLAT) is necessary before going to Level 5. The committee’s position is that testing at Level 5 should be the minimum for system certification. The committee assumes that Level 6 will not be permitted as part of any evaluation procedure. A WSLAT-like capability likely may be necessary in the future because adequate simulants for all agents of concern will not be found, or because adequate simulants may prove to be too time consuming or costly to develop for all agents of concern, or because new agents of concern may arise for which a WSLAT capability is needed to support the warfighter in the field on short notice, or because the modeling may not develop dependable relationships in all cases. A WSLAT facility alone will not obviate the needs for an integrated T&E plan, method development including better simulants, and modeling and analysis for predicting operational performance against live agents. The commitment to develop a feasible WSLAT plan, therefore, necessitates parallel investments in these other areas.