Review of Testing and Evaluation Methodology for Biological Point Detectors: Abbreviated Summary (2005)

Biodetector validation is based on a range of metrics that should increase confidence in the system and its ability to meet specific performance goals. In the end the warfighter wants confidence in a system’s performance and the sponsor wants assurance that the funds are well spent. Both of these goals require managing risks.

In its Statement of Work (see Appendix) the committee was asked to “review current biological point-detection system testing protocols and integrated evaluation methodologies”, and “assess the feasibility and benefits of the DOT&E [Director, Operational Test and Evaluation] requirement for a systems-level, active agent testing capability.” Despite the importance of the Joint Biological Point Detection System (JBPDS) production decision, the committee is not convinced that the planned Whole System Live Agent Testing (WSLAT) facility alone will provide the information required for that decision. Without an integrated test and evaluation methodology and a suite of optimized and dependable simulants, the agent/simulant correlation measurements intended to be conducted in WSLAT and elsewhere will not be meaningful. Also, changes in threat requirements and unexpected results with live agents in WSLAT or other facilities will require dependable simulants and a consistent test methodology for resolution. The WSLAT performance requirements document equivocates on critical threat details, and was changed during the committee’s work.

A preliminary review and assessment of these issues was made by the committee in its interim report. In Chapter 1 of this report, the committee provides a general introduction and a summary of the Interim Report that addressed the first item of the committee’s Statement of Work.

With additional data available to the committee, these issues are examined in further detail in Chapter 2. Additionally, the committee was asked in its Statement of Work to review the Whole System Live Agent Testing proposal provided under a study from the West Desert Test Center, Dugway Proving Ground. This review is provided in Chapter 3 of this report.

In its overall evaluation the committee agrees with the concept of WSLAT, or a WSLAT-like facility, in which biological detectors will be evaluated against threat representative challenges. However, the committee concludes that the purposes and need for WSLAT must be clarified and defined more rigorously, since the proposed WSLAT program carries high schedule risk, high cost and questionable feasibility of adequate system performance. The WSLAT program does not present a clear plan for the testing to be done, data to be produced, or how the data will be analyzed and applied to the decision. These issues have not yet been addressed and included in schedule and budget plans for WSLAT.

The committee concludes that in parallel with the planned WSLAT program, an essential matching effort is needed by the DOD to develop a well-defined testing and evaluation methodology.

In Chapter 2, the committee provides the following three high-level recommendations (recommendations 2-1, 2-2, and 2-3), and justification relative to the Interim Report and information provided to the committee subsequent to the release of the Interim Report.

This test and evaluation methodology is detailed further in recommendations 2-4, 2-5, and 2-6.

This facility could be a WSLAT-like facility that evolves from the original concept to support changes in the test and evaluation methodology and to address shortcomings identified by the committee in its review. The recommended strategy is to work backwards from the integrated T&E plan to the data and analysis needed and then to specification of appropriate facilities and infrastructure. A mobile whole system T&E capability is recommended to address T&E in representative environments and backgrounds.

Based on the collective expertise and experience of the committee, the technical challenges inherent in the development and implementation of evaluation methodology for the JBPDS warrant outside scientific advice through an independent team. Because this team could be composed of individuals both inside and outside DOD, more wide ranging scientific expertise can assist project managers in moving forward with additional confidence in the scientific rationale for the decisions that DOD needs to make.

The integrated T&E plan should be externally peer reviewed as should any anticipated changes to the WSLAT proposal.

Each of the following recommendations related to testing and evaluation methodology is developed in detail in Chapter 2.

With the exception of component testing of simulants and live agents, it is clear that Dugway Proving Ground does not currently have adequate test facilities. It is for this reason that the committee recommends that a Whole System Killed Related Agent Testing (WSKRAT) facility be constructed and used to begin implementation of the test methodology described above.

The use of killed related strain agents provide the best opportunity to both mimic live agents and to pass regulatory hurdles that may limit testing. Another advantage of such a facility is that WSKRAT would be a fully contained system with no open-air release, implying that some testing can begin immediately. In addition, by avoiding the use of live agents, it should be possible to construct a mobile WSKRAT facility that can test in realistic operational environments.

While the committee is convinced that the development of WSKRAT is essential as part of a complete set of test and evaluation facilities, a WSLAT-like facility also will be necessary. However, without the needed data provided by a

graded approach that includes WSKRAT, a WSLAT itself will not represent an adequate test strategy.

In Chapter 3, the committee provides its evaluation of the WSLAT feasibility study, the second item in the committee’s Statement of Work, including the following recommendation and a summary of the justification developed more fully in Chapter 3.

The committee is not optimistic that the proposed WSLAT approach will succeed. The design has major unresolved technical questions. The schedule is highly compressed and is already behind. The committee also concluded that risk identification, analysis, and proposed management were particularly weak. The committee’s observations on the feasibility study are presented in three groups. The first addresses the requirements analysis, the second addresses the scientific and engineering approach, and the third group of comments addresses regulatory and policy concerns.

In addition to these comments specifically about the WSLAT feasibility study, the committee notes that a credible testing program requires more than testing facilities. Supporting methodology, materials (e.g., agents and simulants), and validated processes are also needed. To be useful, WSLAT should be operated according to an overall, integrated test and evaluation methodology. The live agents it will use must be validated and correlated with a suite of optimized and dependable simulants. Neither the methodology nor the suite of simulants is adequate, and WSLAT alone will not suffice.

In Chapter 4, the committee suggests an alternative and complementary process for the testing and evaluation of biopoint detection systems in light of the recommendations and evaluations provided by the committee in this report.

Assuming that simulants can be selected that mimic the biological agents¹ of concern, the predictive capability of the system performance testing protocol increases in the following order (this list is in order of increasing confidence levels):

With an ideal simulant, there will be sufficient confidence in the simulant’s ability to faithfully mimic the agent(s) of concern that whole system live agent testing will not be required. If such simulants do not exist then Level 4 (WSLAT) is necessary before going to Level 5. The committee’s position is that testing at Level 5 should be the minimum for system certification. The committee assumes that Level 6 will not be permitted as part of any evaluation procedure.

A WSLAT-like capability likely may be necessary in the future because adequate simulants for all agents of concern will not be found, or because adequate simulants may prove to be too time consuming or costly to develop for all agents of concern, or because new agents of concern may arise for which a WSLAT capability is needed to support the warfighter in the field on short notice, or because the modeling may not develop dependable relationships in all cases.

A WSLAT facility alone will not obviate the needs for an integrated T&E plan, method development including better simulants, and modeling and analysis for predicting operational performance against live agents. The commitment to develop a feasible WSLAT plan, therefore, necessitates parallel investments in these other areas.