National Academies Press: OpenBook
« Previous: Front Matter
Suggested Citation:"Executive Summary." National Research Council. 2003. Testing and Evaluation of Standoff Chemical Agent Detectors. Washington, DC: The National Academies Press. doi: 10.17226/10645.
×

Executive Summary

The detection of chemical warfare agents is a topic that attracts a good deal of attention in the current environment of the war on terrorism. It is well, therefore, to point out at the outset what this report does and does not cover because the broad topic is a critical one with many aspects that need to be addressed urgently. Testing and evaluation of standoff chemical warfare agent (CWA)1 detectors to certify their suitability for field applications present two major challenges: (1) these detectors cannot readily be tested in a field environment with CWAs without extensive precautions and legal approvals by top government officials and (2) the wide variety of environments in which the detectors might be used may significantly affect the instrument performance. The Defense Threat Reduction Agency (DTRA) of the Department of Defense (DoD) sponsored a study by the Battelle Memorial Institute entitled The Use of Chemical Agent Simulants in Standoff Detection Testing. This study issued a final report in October 2001 which stated that “using simulants instead of chemical agents provides an effective means for conducting outdoor operational testing of standoff detection instruments.”2,3 Review of the report by the DoD raised concerns that a testing protocol, absent field testing with CWAs, might not adequately predict the response of these detectors to CWAs in battlefield and homeland defense applications.

DTRA requested that the National Research Council undertake a study of the testing and evaluation of standoff detectors to provide an independent assessment of suitable test protocols that might be useful and reliable. In its Statement of Work, the committee was asked to answer the following:

1  

Appendix D contains a list of acronyms and a glossary of terms.

2  

A.R. Blackburn. K.S.K. Chinn, S.D. Fortney, W.A. Ivancic, A.K. Judd, B.D. Lerner, and J. Ontiveros. 2001. The Use of Chemical Agent Simulants in Standoff Detection Testing. Battelle Memorial Institute, Columbus, Ohio, CBIAC Contract No. SPO700-00-D-3180. CBIAC Task No. 75.

3  

Simulants are molecules having either spectroscopic or physical properties similar to the corresponding properties of a CWA.

Suggested Citation:"Executive Summary." National Research Council. 2003. Testing and Evaluation of Standoff Chemical Agent Detectors. Washington, DC: The National Academies Press. doi: 10.17226/10645.
×
  1. What test protocols should be adopted to ensure that standoff CWA detectors will meet operational requirements and why. Consideration should be given to a variety of options to include chamber testing, chamber and simulant testing, and live-agent open-air testing.

  2. Identify the challenges associated with executing the recommended protocols.

  3. Identify the risks associated with not doing open-air testing using live agents.

  4. Using Multi-Service Tactics, Techniques, and Procedures for Risk Management as described in FM 3-100.12, Air Land Sea Risk Management dated February 2001, (ref Annex D Risk Assessment Matrix), assess the risk associated with operationally employing standoff CWA detectors that have been tested at three possible levels: (1) baseline—live-agent chamber challenges combined with simulant open-air challenges; (2) baseline plus challenges in a test facility capable of enclosed, long-range live-agent challenges; and (3) baseline plus live-agent open-air challenges. If NAS does not feel qualified to assess the severity component of such a risk assessment, NAS may provide the probability component and defer risk assessment to the DoD.

This report is a narrow and very specific study of the testing and evaluation of infrared-type standoff detectors for CWAs in military situations. It is not a study of various possible methods of detecting chemical warfare agents, nor is it a study of the use of such detectors in broader applications such as homeland security. Rather, it is a study of the best possible way to evaluate such detectors in a realistic way that will ensure that they will detect CWAs, primarily in a wartime scenario.

The committee addressed a number of critical issues:

  • Which types of detectors were to be included in any test protocols. The committee determined that two types of detectors, passive infrared and active infrared based on lidar technology, would incorporate technology presently used as well as technology likely to be incorporated in the not-too-distant future.

  • The value of field-testing of these detectors with CWAs. The present study provides complex— but necessary—test protocols that eliminate the need for field testing with CWAs.

  • The risk assessment of responses from these detectors. The response of these detectors in active field applications has a certain error associated with it. The results (an alarm or no alarm) from these detectors have to be interpreted in the context of command actions that must be taken. The report discusses some examples of the risk assessment and illustrates how the information from the detectors must fit into the decision-making process.

The complexities of the protocols recommended in this report are primarily driven by two factors: the wide variety of backgrounds in which the CWAs are to be detected and the variable chemical nature of the CWAs that results in a range of infrared absorption bands on which their detection can be based. Detecting and measuring the spectral features of a CWA in the presence of thousands of background spectra using complex chemometric techniques requires statistical testing of simulants in both laboratory and field environments as well as laboratory testing of the various CWAs. It is these factors that result in the large amount of testing which has to be done to assure detector performance.

The committee’s protocols do not call for field testing with CWAs, and the committee presents justification to show that this step is not necessary to provide statistically valid protocols.

SIMULANTS

Simulants are chemicals with lower toxicity than CWAs that can be used in the testing and

Suggested Citation:"Executive Summary." National Research Council. 2003. Testing and Evaluation of Standoff Chemical Agent Detectors. Washington, DC: The National Academies Press. doi: 10.17226/10645.
×

evaluation of standoff detectors, in both the laboratory and the field. Two broad categories of simulants can be identified—spectral simulants and aerosol simulants.

For spectral simulants the spectra of these compounds should ideally have their maximum absorption within 20 cm–1 of the CWA of interest, and the spectra of the simulants should not have absorption bands stronger than the two strongest bands of the spectrum of interest.

Development of a generic simulant for CWA aerosols and droplets poses challenges that have yet to be solved. Rigorous simulation of a CWA aerosol or droplet requires matching (1) the optical properties of the material, (2) the physical properties that determine the manner in which the agent is dispersed, and (3) the dispersal method in the battlefield environment.

SUMMARY OF RECOMMENDATIONS

From its study of the problem, the committee makes a series of recommendations regarding appropriate test protocols and risk assessment in standoff detector testing and evaluation. Because they are inherently different, the two categories of detectors, passive and active, that are based on infrared spectral measurements require somewhat different test protocols.

TESTING PROTOCOL FOR PASSIVE DETECTORS

The committee concluded that the protocol recommended in the Battelle report is inadequate and that the assumptions on which it is based are incorrect. The major flaw in the protocol is the lack of information about the highly variable backgrounds that will be observed in the field. The committee proposes an alternative protocol that first acquires a large sampling of background spectra under a variety of field conditions. Then laboratory chamber spectra are taken of simulants, CWAs, concomitants, and possible interferents. Combining these, spectra can be synthesized that would be observed under field conditions with simulants or agents, concomitants, and interferents.4 The large collection of synthesized spectra provides a database for the training and verification of a signal-processing model. This model can be tested first with known concentrations of simulants, interferents, and possible concomitants under ideal conditions in a chamber. If validated under these conditions, the same signal-processing model can then be tested in the field using simulants, interferents, and concomitants. Successful verification of this model using simulants in field measurements validates the transfer of the signal-processing model from the chamber to field and gives high confidence that a similar model for CWAs, if successful in chamber measurements, would transfer to the field without the necessity of field measurements using CWAs.

Recommendation: A detailed test protocol for testing and evaluating passive standoff detec tors is recommended that recognizes the importance of the highly variable background in which such detectors will be employed. Simulants are used to provide validation of the signal-processing model in both the laboratory and the field. A similar model, developed with only laboratory testing of CWAs, can be used for the detection of live CWAs in the field.

Application environments for the detector are defined in the proposed test protocol.

4  

Concomitants are compounds that may be present under battlefield conditions—such as adhesives, thickeners, and propellants—whose spectra may interfere with detection of the target chemicals. Interferents are chemicals that in some way interfere with the detection of the target chemicals. This interference can be spectral, chemical, or physical.

Suggested Citation:"Executive Summary." National Research Council. 2003. Testing and Evaluation of Standoff Chemical Agent Detectors. Washington, DC: The National Academies Press. doi: 10.17226/10645.
×

TESTING PROTOCOL FOR ACTIVE DETECTORS

Active detection systems are inherently easier to model using physical data on the various components that make up the field of measurement in such systems. In fact, modeling is necessary to study the detectability of simulants and CWAs with active detectors. This results in a somewhat different approach to the testing of these devices.

Recommendation: This test protocol will be used to validate a model for the measurement and detection of CWAs in the field. The model will be based on measurements and understanding of background, simulants, interferents, and concomitants as well as characteristics of the instrumentation.

Validation of the model with simulants in the laboratory and field along with laboratory validation with CWAs will provide a high degree of confidence in the ability of the instrument to detect CWAs in the field under battlefield or homeland security conditions.

Limitation of application environments for these detectors will be predicted using the validated model. Detection sensitivity for the CWAs as a function of environmental factors in the model, such as levels of fog, dust, etc., can be predicted for the model and define the applicability limits of the detector.

RESEARCH AREAS TO SUPPORT TEST PROTOCOLS FOR STANDOFF DETECTORS

The committee believes that additional information and data are needed to fully support the proposed test protocols. These are described in Chapter 6. Among these research areas are the following related to the development of an algorithm for signal processing.

Recommendation: It is recommended that algorithm development be a multigroup effort that will result in robust, upgradeable, transferable software that will utilize multiple, differenti ated chemometric approaches to the problem.

Robust algorithms are an essential element in the success of these detectors. This effort should ensure not only that the algorithm is effective but also that they can be modified as necessary by software experts in different locations as needed for the application.

DECISION MAKING AND RISK

A potential concern with the deployment of a standoff detector that has only been tested with simulants—but never with “live” CWAs— under realistic field conditions is that this might lower the degree of trust in its results. However, the principal detection challenge—whether for simulant or agent—is whether the spectral signatures can be unequivocally distinguished from background radiation and confounding spectral features associated with interferents and concomitants in the atmosphere. The rigorous test protocols developed by the committee provide the data necessary to develop algorithms that will be able to discriminate CWAs in the complex spectra acquired by a given instrument. If an instrument can achieve the demanding level of performance required by the applicable test protocol using simulants in both laboratory and field tests, there is a high degree of confidence that it would also detect CWAs in actual field conditions.

The ultimate proof of a standoff detector’s worth will be its performance under field conditions. The measure of the value of a detector for making command decisions in the field is to analyze the improvement in such decisions made with information from the detector. The detector’s value lies in

Suggested Citation:"Executive Summary." National Research Council. 2003. Testing and Evaluation of Standoff Chemical Agent Detectors. Washington, DC: The National Academies Press. doi: 10.17226/10645.
×

providing information that will permit accurate prediction and decision to protect troops in an actual CWA attack, as well as the degree to which unproductive protective measures can be avoided when there is no attack.

Recommendation: Decision and risk training in the context of standoff detectors should be given to personnel who will be associated with the operation, use, and analysis of information generated by the standoff detectors in the field.

Good decisions do not consist of following the detector’s alarms blindly, but in using the detector’s information to improve the ability to characterize the chances of a CWA attack in a given threat situation. Understanding the false positive and false negative rates, the situations that can lead to them, and the pitfalls of deciding what to do based only on the detector’s readings are all key to the effective use of the detector and to optimal avoidance of losses. Training in the proper use of detectors in the context of the larger decision-making setting would help make the best use of the detectors’ information.

FIELD TESTING WITH CWAS

The value of open-air testing of detectors with CWAs, if any, would lie in the degree to which false positive and false negative rates are better characterized and understood than could be achieved by testing with simulants alone. Arguing against CWA testing is the fact that any such testing must necessarily be limited to a fairly narrow range of conditions and CWAs. Thus, testing with CWAs in the field would add little, if anything, to the confidence in the device. Such limited testing could also create an unwarranted degree of confidence if the detector “works” under field testing conditions that may turn out to be much more favorable than those actually encountered in battle.

Recommendation: Testing CWAs in the field is not recommended. The added value of infor mation from such tests, recognizing that they would be limited in number, would not provide a significant improvement in the confidence level about instrument performance.

A detailed discussion of risk assessment and the value of information analysis is provided in Appendix C.

While the focus of this study was the testing and evaluation of standoff detectors for military applications, the protocols established and described may have applicability to the broader use of such detectors in areas such as homeland defense, for example. Such applications would require more knowledge of deployment strategies for detectors in such applications and tailoring of the recommended protocols for these different scenarios.

Suggested Citation:"Executive Summary." National Research Council. 2003. Testing and Evaluation of Standoff Chemical Agent Detectors. Washington, DC: The National Academies Press. doi: 10.17226/10645.
×
Page 1
Suggested Citation:"Executive Summary." National Research Council. 2003. Testing and Evaluation of Standoff Chemical Agent Detectors. Washington, DC: The National Academies Press. doi: 10.17226/10645.
×
Page 2
Suggested Citation:"Executive Summary." National Research Council. 2003. Testing and Evaluation of Standoff Chemical Agent Detectors. Washington, DC: The National Academies Press. doi: 10.17226/10645.
×
Page 3
Suggested Citation:"Executive Summary." National Research Council. 2003. Testing and Evaluation of Standoff Chemical Agent Detectors. Washington, DC: The National Academies Press. doi: 10.17226/10645.
×
Page 4
Suggested Citation:"Executive Summary." National Research Council. 2003. Testing and Evaluation of Standoff Chemical Agent Detectors. Washington, DC: The National Academies Press. doi: 10.17226/10645.
×
Page 5
Next: 1 Introduction »
Testing and Evaluation of Standoff Chemical Agent Detectors Get This Book
×
Buy Paperback | $47.00 Buy Ebook | $37.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The report provides an independent assessment of suitable test protocols that might be useful and reliable for the testing and evaluation of standoff chemical agent detectors. The report proposes two testing protocols, one for passive detectors and one for active detectors, to help ensure the reliable detection of a release of chemical warfare agents. The report determined that testing these detectors by release of chemical warfare agents into the atmosphere would not provide additional useful information on the effectiveness of these detectors than would a rigorous testing protocol using chemical agents in the laboratory combined with atmospheric release of simulated chemical warfare agents.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!