4
Recommended Evaluation Strategies

As discussed in Chapter 2, the committee’s two primary concerns are that the current simulant suite does not adequately mimic live agents and that whole system testing in pristine environments will not predict successful operation in the field. Many chemical and biological detectors that perform well under laboratory conditions fail when faced with real-world challenges. Thus, in addition to the evaluation of the current WSLAT proposal as described in Chapter 3, the committee has suggested alternative and complementary protocols for the evaluation of biopoint detection systems.

The Statement of Work for the committee directs it to assess the WSLAT proposal’s ability “to support test and evaluation of current and near-future biological point detection systems.”1 The question that derives from this directive is this: “How does the WSLAT fit into the Government’s total evaluation plan?” The committee’s alternative protocols were built around this central question.

DEVELOPMENT OF OVERALL SYSTEM ASSESSMENT

It is imperative then that the top-level question of how the WSLAT contributes to the overall system assessment be addressed by DOD before committing to an expensive and high-risk course of action. It is the committee’s belief that the OTA’s concerns of evaluating the JBPDS in an operationally relevant environment, and with threat representative challenges, can best be met in the near future

1  

Statement of Work from the JPEO-CBD to the National Research Council. (see Appendix A).



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 19
Review of Testing and Evaluation Methodology for Biological Point Detectors: Abbreviated Summary 4 Recommended Evaluation Strategies As discussed in Chapter 2, the committee’s two primary concerns are that the current simulant suite does not adequately mimic live agents and that whole system testing in pristine environments will not predict successful operation in the field. Many chemical and biological detectors that perform well under laboratory conditions fail when faced with real-world challenges. Thus, in addition to the evaluation of the current WSLAT proposal as described in Chapter 3, the committee has suggested alternative and complementary protocols for the evaluation of biopoint detection systems. The Statement of Work for the committee directs it to assess the WSLAT proposal’s ability “to support test and evaluation of current and near-future biological point detection systems.”1 The question that derives from this directive is this: “How does the WSLAT fit into the Government’s total evaluation plan?” The committee’s alternative protocols were built around this central question. DEVELOPMENT OF OVERALL SYSTEM ASSESSMENT It is imperative then that the top-level question of how the WSLAT contributes to the overall system assessment be addressed by DOD before committing to an expensive and high-risk course of action. It is the committee’s belief that the OTA’s concerns of evaluating the JBPDS in an operationally relevant environment, and with threat representative challenges, can best be met in the near future 1   Statement of Work from the JPEO-CBD to the National Research Council. (see Appendix A).

OCR for page 19
Review of Testing and Evaluation Methodology for Biological Point Detectors: Abbreviated Summary with a WSKRAT-type facility, a facility that will also serve as a mechanism for developing critical enabling methodologies for a WSLAT-like test facility. The committee notes that killed2 related agents, such as those used for vaccines, are readily available and provide agent simulants (or surrogates) that most nearly mimic agent activity without pathogenecity. In addition, these killed related agents can be evaluated in BSL-2 facilities where the detectors can be challenged with representative environments. Assuming that simulants can be selected that mimic the agents of concern, the predictive capability of the system performance testing protocol increases in the following order (this list is in order of increasing confidence levels): Level 1 Component testing with simulants Level 2 Component testing with live agents Level 3 Whole system testing with simulants alone Level 4 Whole system testing with live agents (WSLAT) Level 5 Whole system testing with simulants under environmentally representative conditions Level 6 Whole system testing with live agents under environmentally representative conditions Recommendation 4-1: Assuming good simulants exist, the committee recommends testing to proceed from component testing with simulants (Level 1), and live agents (Level 2), through whole system testing with simulants (Level 3), and then to whole system testing with simulants under environmentally representative conditions (Level 5). With an ideal simulant, there will be sufficient confidence in the simulant’s ability to faithfully mimic the agent(s) of concern that whole system live agent testing will not be required. If such simulants do not exist then Level 4 (WSLAT) is necessary before going to Level 5. The committee’s position is that testing at Level 5 should be the minimum for system certification. The committee assumes that Level 6 will not be permitted as part of any evaluation procedure. A WSLAT-like capability likely may be necessary in the future because adequate simulants for all agents of concern will not be found, or because adequate simulants may prove to be too time consuming or costly to develop for all agents of concern, or because new agents of concern may arise for which a WSLAT capability is needed to support the warfighter in the field on short notice, or because the modeling may not develop dependable relationships in all cases. 2   “Killed” agents are microbes or toxins that are rendered biologically inactive, and hence non-pathogenic and non-reproducing. Typical “kill” or inactivation protocols use gamma-irradiation or chemical inactivation. The goal is to render the biologics inactive and non-pathogenic/non-toxic, but still identifiable by the sensor under test. “Live” agents retain their pathogenic/toxic and/or reproductive capacities.

OCR for page 19
Review of Testing and Evaluation Methodology for Biological Point Detectors: Abbreviated Summary An additional advantage of the correlated simulant approach is that transportable test facilities (referred to as “mobile WSKRAT” in this report) can be taken to a specific location and the detector tested directly under real environmental conditions. The committee understands that creating a representative environment inside a BSL-3 or higher facility will be expensive and difficult to accomplish. Nevertheless, a WSLAT facility in which detectors can be challenged with representative environments would provide a high level of confidence in the detector. The committee repeats its earlier recommendation, however, that a WSLAT-like facility should be considered an end stage in a progressively more realistic test facility development and should not be viewed as sufficient in and of itself to provide detection system evaluation. SUMMARY: THE ARGUMENT FOR AN IMMEDIATE START ON WSKRAT The committee’s proposed strategy offers significant advantages and risk reductions over a strategy based on the WSLAT program alone. The committee stresses that with its strategy constructive work could begin immediately on the development of an evaluation plan, challenge definition and methodology development. At a future time work on testing could take place to support the full rate production decision.

OCR for page 19
Review of Testing and Evaluation Methodology for Biological Point Detectors: Abbreviated Summary This page intentionally left blank.