Review of Testing and Evaluation Methodology for Biological Point Detectors: Abbreviated Summary (2005)

There are three high-level recommendations that derive from the committee’s assessment of the current testing and evaluation methodologies.

Each of these recommendations will be discussed in detail in the following sections. Additional recommendations related to these high-level recommendations are also in these sections.

As the committee received information on the process by which the JBPDS program had identified performance requirements, presented testing criteria, and evaluated early sensor constructs, it became clear that a systematic and scientifically credible test methodology would still be needed to guide production deci-

sions for JBPDS. As detailed in the interim report, this test methodology should comprise

• component testing with optimized simulants and live agents;

• whole system testing with simulants and, where necessary, live agents;

• predictive correlation analyses of component to whole system performance; and

• testing and correlation analyses in realistic operational environments.

Although these testing elements appear straightforward, and much has been done consistent with the committee’s recommendations, it appears that these testing elements have not been applied in a systematic and rigorous manner to the evaluation process as a whole. Therefore, the committee examined each of these testing elements in order to provide further guidance and elaboration on their justification and implementation.

The committee was not provided protocols and corresponding data to assess the performance of JBPDS components or whole system upon which to base a decision for operational deployment of these systems. The first fundamental need for a credible integrated test methodology for JBPDS is the development of a suite of well-correlated simulants for each live agent of concern or for appropriate classes of live agents. The second need is the development of predictive models and correlation analyses between component response and whole system performance.

It is important that the term “simulant” be clearly defined for the purposes of this report. Simulants are nonpathogenic organisms that are employed to minimize the exposure risks to testing personnel. In fact, simulants are anything that provides useful, evaluative information on the performance of the system under test, short of the actual biologically active, weaponized warfare agents themselves. Other terms used frequently in this report are defined as follows:

Live Agent—One or more of the viable pathogenic organisms or active toxic compounds as cited in the International Task Force-6 (ITF-6) report of 9 February 1990.

Killed Agent—The nonviable or inactive forms of the aforementioned live agents.

Live Related-Strain Agent—One or more viable organisms or active compounds that closely resembles, genetically or chemically, a live agent as described above EXCEPT that it is of low to zero pathogenicity, virulence or toxicity.

Killed Related-Strain Agents—Nonviable or inactivated form of related-strain agent.

The better the simulant (or surrogate) mimics the biological warfare agents of concern (i.e., creates the same response in the system under test) the more confidence evaluators and warfighters will have in the system’s ability to fulfill its mission. Equally important to the biological verisimilitude of the simulant is the ability for testers to employ the simulant in a threat representative manner and in operationally relevant environments. In addition, simulants should be easily quantifiable so that evaluators can in turn provide a quantitative measure of the tested system’s performance. In summary, the ideal simulant will have the following properties:

• It will interact with the system under test in a manner that can be directly correlated with the analogous warfare agent.

• It can be presented to the system under test in a threat representative manner, and under operationally relevant environments.

• It supports quantitative measure of simulant challenges to the system under test.

• It should provide greater flexibility in whole system testing and evaluation compared with actual agents.

The committee’s recommendations for better simulants derives entirely from the need to exercise the system in as realistic a fashion as possible so that results derived from simulant testing can be used to predict live-agent performance. Optimization of the actual JBPDS performance itself should be an ongoing but separate effort.

The committee suggests that the best simulants to mimic the behavior of live agents would be killed agents or killed related strains of live agents.

Provided that the biodetection component response to these killed related strain agents can be correlated with that of their pathogenic live counterparts, their relative ease of growth, preparation, and handling will provide for more realistic and frequent whole-system testing.

Substantial emphasis was placed in the Interim Report on the importance of developing both component and whole system performance prediction models. The assumptions are twofold: (1) The component response to simulants can be statistically correlated to those of live agents, and (2) whole system models can be developed with good predictive value even in realistic environments. If these assumptions are true, then component-to-whole-system modeling may enable live agent performance prediction from simulant whole system testing, and therefore obviate the need for WSLAT under some conditions.

The committee notes that additional expertise in system modeling may be required to supplement the current team that is slated to execute the tests; options for providing this essential expertise should be explored.

Degradation of performance when complex systems are taken to the field is the rule rather than the exception, and this is the reason for the committee’s consensus that testing in realistic environments is as important, if not more important, than whole system testing against live agents. The integrated test methodology should include a path forward for eventual tests in more challenging operational environments.

While the committee recognizes the complexity of such modeling and analyses outlined above, we believe they are within the state of practice in the field. The critical question then becomes, “Which facilities and expertise are necessary and sufficient to support the testing and model validation?”

In reviewing the available test facilities at DPG and in assessing what will be needed for system validation it is clear that with the exception of component testing of simulants and live agents, DPG does not currently have adequate test facilities. It is for this reason that the committee recommends that a Whole System Killed Related Agent Testing (WSKRAT) facility be constructed and used to begin implementation of the test methodology presented in the previous section. The WSKRAT would be a fully contained system with no open-air release. Techniques for introduction of transient environmental interferents should be developed to provide more realistic conditions for whole system testing.

As noted previously, it is essential to ultimately test in realistic operational environments. While much can be learned from the introduction of collected background air into the WSKRAT facility, it is difficult to reproduce the natural variation found in the broad range of likely JBPDS deployment environments. Therefore a mobile test facility should also be constructed that can accommodate simulants that mimic live agents. The mobile WSKRAT would enable optimized simulant challenges (to mimic live agents) in varying complex operational environments. This system would offer capability in assessing operational performance as well as in validating the whole system correlation models.

The cornerstones of the committee’s recommended approach are high correlation between component results for killed related strain and live agents and the ability of predictive models to reproduce whole system behavior in operational environments. A phased approach to test facility development has the advantages of immediately proceeding with implementation of the systematic test and evaluation methodology proposed while also providing valuable data to guide the development of an end-point facility.

While the committee is convinced that the development and construction of both stationary and mobile WSKRAT facilities are essential, we suspect that a WSLAT facility also will be necessary as part of a complete set of test and evaluation facilities. However, while WSLAT may become necessary, by itself it will not represent an adequate test strategy. The proposed graded approach—including WSKRAT—will provide the needed data.

The arguments for a WSLAT-like facility are to

1. complete the validation of system models.

2. test systems or agents not amenable to modeling.

3. test agents not amenable to the killed related strain strategy.

4. establish the facilities and expertise.

The committee stresses that design and construction of an appropriate test facility requires generation of an extensive scientific body of evidence, as well as substantial analysis and modeling, before an effective WSLAT facility can be developed; reliance on the current WSLAT proposal alone will not adequately address the needs for test validation to support JBPDS testing, evaluation, and production. In summary, the committee proposes that the order of priorities in test and evaluation should be

1. implementation of a scientifically credible, integrated test and evaluation methodology;

2. component testing of simulants and live agents (and the development of models to correlate the two);

3. component and whole-system testing in WSKRAT (and the development of predictive system performance models);

4. testing and evaluation in complex operational environments (with model refinement as required); and

5. whole system live-agent testing and evaluation in a WSLAT-like facility (with model validation).

The need for an independent expert committee to advise the JPEO on the development of credible integrated test and evaluation methodology was apparent by the lack of submission of any protocols and systematic methodologies for the committee to review. The scientific, management, and regulatory demands of such methods development are daunting, and JPEO would be well served by a formal advisory committee to accelerate the progress and enhance the utility of future products of the program.

The overall evaluation methodology for the JBPDS (and biological point detectors in general) should be documented and submitted for peer review.

Based on the collective expertise and experience of the committee, the technical challenges inherent in the development and implementation of evaluation methodology for the JBPDS warrant outside scientific advice through an independent team. Because this team could be composed of individuals both inside and outside DOD, more wide ranging scientific expertise can assist project managers in moving forward with additional confidence in the scientific rationale for the decisions that DOD needs to make.

This team should include a broad spectrum of expertise spanning the technical breadth required for the development, testing, and evaluation of systems such as the JBPDS. Although the details of the charter for this team should be determined by the JPEO-CBD and DOT&E, the committee recommends that this advisory body be used to assess future proposals for test facilities and test and evaluation protocols for JBPDS and next generation systems. The committee expects that the proposed independent team will complement and greatly assist the expertise resident in the existing programs.