Review of Testing and Evaluation Methodology for Biological Point Detectors: Abbreviated Summary (2005)

As discussed in Chapter 2, the committee’s two primary concerns are that the current simulant suite does not adequately mimic live agents and that whole system testing in pristine environments will not predict successful operation in the field. Many chemical and biological detectors that perform well under laboratory conditions fail when faced with real-world challenges. Thus, in addition to the evaluation of the current WSLAT proposal as described in Chapter 3, the committee has suggested alternative and complementary protocols for the evaluation of biopoint detection systems.

The Statement of Work for the committee directs it to assess the WSLAT proposal’s ability “to support test and evaluation of current and near-future biological point detection systems.”¹ The question that derives from this directive is this: “How does the WSLAT fit into the Government’s total evaluation plan?” The committee’s alternative protocols were built around this central question.

It is imperative then that the top-level question of how the WSLAT contributes to the overall system assessment be addressed by DOD before committing to an expensive and high-risk course of action. It is the committee’s belief that the OTA’s concerns of evaluating the JBPDS in an operationally relevant environment, and with threat representative challenges, can best be met in the near future

with a WSKRAT-type facility, a facility that will also serve as a mechanism for developing critical enabling methodologies for a WSLAT-like test facility.

The committee notes that killed² related agents, such as those used for vaccines, are readily available and provide agent simulants (or surrogates) that most nearly mimic agent activity without pathogenecity. In addition, these killed related agents can be evaluated in BSL-2 facilities where the detectors can be challenged with representative environments.

Assuming that simulants can be selected that mimic the agents of concern, the predictive capability of the system performance testing protocol increases in the following order (this list is in order of increasing confidence levels):

With an ideal simulant, there will be sufficient confidence in the simulant’s ability to faithfully mimic the agent(s) of concern that whole system live agent testing will not be required. If such simulants do not exist then Level 4 (WSLAT) is necessary before going to Level 5. The committee’s position is that testing at Level 5 should be the minimum for system certification. The committee assumes that Level 6 will not be permitted as part of any evaluation procedure.

A WSLAT-like capability likely may be necessary in the future because adequate simulants for all agents of concern will not be found, or because adequate simulants may prove to be too time consuming or costly to develop for all agents of concern, or because new agents of concern may arise for which a WSLAT capability is needed to support the warfighter in the field on short notice, or because the modeling may not develop dependable relationships in all cases.

An additional advantage of the correlated simulant approach is that transportable test facilities (referred to as “mobile WSKRAT” in this report) can be taken to a specific location and the detector tested directly under real environmental conditions.

The committee understands that creating a representative environment inside a BSL-3 or higher facility will be expensive and difficult to accomplish. Nevertheless, a WSLAT facility in which detectors can be challenged with representative environments would provide a high level of confidence in the detector. The committee repeats its earlier recommendation, however, that a WSLAT-like facility should be considered an end stage in a progressively more realistic test facility development and should not be viewed as sufficient in and of itself to provide detection system evaluation.

The committee’s proposed strategy offers significant advantages and risk reductions over a strategy based on the WSLAT program alone. The committee stresses that with its strategy constructive work could begin immediately on the development of an evaluation plan, challenge definition and methodology development. At a future time work on testing could take place to support the full rate production decision.

This page intentionally left blank.