National Academies Press: OpenBook

Testing and Evaluation of Standoff Chemical Agent Detectors (2003)

Chapter: 4 Recommended Test Protocol and Decision Tree for Active Detectors (Lidar)

« Previous: 3 Recommended Test Protocol and Decision Tree for Passive Detectors
Suggested Citation:"4 Recommended Test Protocol and Decision Tree for Active Detectors (Lidar)." National Research Council. 2003. Testing and Evaluation of Standoff Chemical Agent Detectors. Washington, DC: The National Academies Press. doi: 10.17226/10645.
×

4
Recommended Test Protocol and Decision Tree for Active Detectors (Lidar)

Active detectors in this context are called lidar, or light detection and ranging detectors. They utilize a laser excitation source that can be either fixed or tuned wavelength and a co-located receiver system to collect the scattered laser radiation (with or without wavelength shifting) from the atmosphere and/or topographic target. The successful design and use of this detection system, i.e., a system which sends out its own “interrogating” radiation from a source and records the reflected radiation that has been attenuated by interaction with the “field,” depends on accurately modeling the system, including all components, both instrumental and in the field, as part of the model. The testing and evaluation experiments are then used to refine the model such that ultimately it can reliably predict field behavior of the instrument. References given later in this discussion provide examples of this methodology for other applications.

ASSUMPTIONS

This protocol assumes that:

  1. field testing of chemical warfare agents (CWAs) can be avoided if adequate testing of simulants is performed in conjunction with a validated measurement uncertainty model that can realistically and reliably predict lidar performance for CWA measurements in battlefield or home defense environments;

  2. no single simulant exists that can simultaneously emulate CWAs in optical and physical characteristics;

  3. all battlefield conditions cannot be reproduced in field testing for evaluating lidar CWA measurement capabilities; and

  4. steps in the protocol must be completed in sequence (unless otherwise stated), and if a step cannot be met to the degree required by the Operational Requirements Document (ORD), then a change to the ORD must be agreed to before proceeding to the next step.

Suggested Citation:"4 Recommended Test Protocol and Decision Tree for Active Detectors (Lidar)." National Research Council. 2003. Testing and Evaluation of Standoff Chemical Agent Detectors. Washington, DC: The National Academies Press. doi: 10.17226/10645.
×

FiGURE 3 Test protocol for active detectors.

TESTING AND EVALUATION PROTOCOL (FIGURE 3)

Step:

  1. Develop measurement uncertainty model. A comprehensive end-to-end measurement uncertainty model must be developed that includes all of the components necessary to realistically model the CWA measurement performance of the lidar system under conditions in which it is to be used. This must include all the factors that can contribute to uncertainties in the lidar CWA measurement performance. These factors entail battlefield environmental factors, including atmospheric scattering and attenuation from various sources; reflected and emitted background radiation sources; vapor, aerosol, and droplet characteristics of CWA releases and their spatial and temporal variation; spectral and spatial characteristics of topographic scattering; and lidar system characteristics as they pertain to measurement performance, such as transmitted energy per pulse, number of pulses averaged per measurement, wavelength stability of the laser source, range to measurement, size of receiver, field of view (FOV), collection and transmission efficiency (including optical filters), detector system noise, measurement sampling differences, and errors in application of the various lidar measurement techniques to be used. This model must be sufficiently comprehensive to be able to assess the sensitivity of the lidar system performance to variations in CWA release scenarios, changes in battlefield environments, meteorological and target variations, etc., and it must be able to be validated in comprehensive simulant experiments described below. Examples of this type of modeling for lidar measurements of gases were first discussed by

Suggested Citation:"4 Recommended Test Protocol and Decision Tree for Active Detectors (Lidar)." National Research Council. 2003. Testing and Evaluation of Standoff Chemical Agent Detectors. Washington, DC: The National Academies Press. doi: 10.17226/10645.
×

Schotland for water vapor measurements16,17and by Byer and Garbuny for measurements of pollutant gases in the infrared.18

  1. Determine and incorporate model input parameters. All of the components of the model that are to be in place in the application of this lidar system must be accurately determined for the model to represent correctly the CWA measurement uncertainties of the system. These include the following:

  • characteristics of all types of topographic scatterers that will be used by this system, including albedo characteristics, spectral dependence of scattering, and spatial variations (mixtures of surfaces within FOV);

  • amount and wavelength dependence of background radiation from all sources within expected lidar FOV under conditions that the system will be used;

  • relevant ambient environmental factors that would affect measurements such as atmospheric temperature, density, and wind;

  • spatial distribution and physical and optical characteristics of all the different aerosols along the measurement FOV;

  • spectral absorption characteristics of CWA, concomitant, simulant, and interferent vapors at the relevant spectral resolution for the lidar measurement;

  • size distribution and refractive index of CWA, concomitant, simulant, and interferent aerosols and droplets;

  • lidar system parameters that affect measurement of signal-to-noise ratio;

  • technique-specific operational issues that could contribute to measurement error sources in the model, such as from spectral impurity, laser wavelength jitter, and Doppler broadening of backscattered signal, etc.; and

  • modeling of the characteristics and uncertainties of the data analysis technique that will be used in the separation of multivariate components to arrive at the concentration of the CWAs such as principal component analyses and neural networks.

Note: Some of the atmospheric and topographic scattering and attenuation characteristics will be augmented by lidar measurements made as part of Step 6, and the CWA and simulant parameters will be defined as part of Step 3.

  1. Compile a database of relevant CWA and simulant properties. Obtain accurate spectroscopic properties of CWA, simulant, and interferent vapors that are appropriate for the lidar system parameters (laser wavelengths and line widths, atmospheric conditions, etc.). Obtain accurate CWA, simulant, and interferent aerosol (and droplet) optical (complex refractive index) and physical (spatial and temporal variations in size distributions) characteristics in the range of atmospheric CWA releases to be considered as required by the ORD. (Note: The aerosol and droplet characteristics may be delivery mode dependent; if so, this information needs to be determined and tested in subsequent steps for each mode.) In most cases these measurements will require chamber tests using the lidar system to directly obtain the

16  

R.M. Schotland. 1966. Some observations of the vertical profile of water vapor by means of a laser optical radar. Fourth Symposium on Remote Sensing of Environment. Environmental Research Institute of Michigan. Ann Arbor, Michigan. Pp. 273-283.

17  

R.M. Schotland. 1974. Errors in the lidar measurement of atmospheric gases by differential absorption, J. Appl. Meteorol. 13:71-72.

18  

R.L. Byer and M. Garbuny. 1973. Pollutant Detection by Absorption Using Mie Scattering and Topographic Targets as Retroreflectors. Appl. Opt. 12:1496-1505.

    Suggested Citation:"4 Recommended Test Protocol and Decision Tree for Active Detectors (Lidar)." National Research Council. 2003. Testing and Evaluation of Standoff Chemical Agent Detectors. Washington, DC: The National Academies Press. doi: 10.17226/10645.
    ×

    optical parameters of the vapors, aerosols, and droplets, which are needed for model simulations of atmospheric measurements.

    1. Develop a data-training algorithm. If a data-processing technique such as a neural network is used in the CWA measurement, it should be trained with the lidar in the high signal-to-noise environment of the chamber or through some other appropriate means (e.g., high-resolution spectroscopic technique).

    2. Match CWAs and simulants. Based on the results of Step 3, determine which simulants are similar to which CWAs in their vapor absorption properties and aerosol and droplet scattering and absorption properties at operating wavelengths of the lidar and their aerosol and droplet physical characteristics.

    3. Test system operational signal and noise levels in realistic environments. Operate the lidar system in a wide range of battlefield environments with ambient atmospheric scatterers, interferents, and a wide range of topographical targets over expected ranges. This will provide a test of the model to predict range-dependent signal and noise characteristics with different aerosol distributions, backgrounds, and topographic targets and their spectral dependencies. If the model does not accurately predict the lidar signals and noise, it must be fixed before going to the next step. It is expected that the atmospheric conditions and topographical characteristics will be sufficiently different than the assumed parameters used in the initial phase of the model development and that empirical changes to the model input parameters will need to be made to bring model predictions in line with actual conditions encountered in the battlefield environment. If the model cannot predict the uncertainty in the lidar measurements without a simulant present, return to Step 1 to determine and correct the fundamental reasons for the model deficiencies. Every step in the protocol must be examined to discover and correct the model or model input parameters.

    4. Simulate instrument field performance. With the model, simulate an atmospheric measurement with a simulant release placed at the expected range for the lidar measurement in an expected battlefield condition (similar to what will be conducted in atmospheric tests). The absorption and scattering characteristics as well as the spatial and temporal variations of the simulant need to be included. The resulting modeled signals at the various lidar wavelengths are then input to the data-processing system for determination of the resulting measurement uncertainty for the simulant. A series of experimental simulations covering the range of simulants, mixture of simulants, and interferents need to be conducted to determine realistically the measurement uncertainty for a simulant. Experimental simulations should be conducted for all simulant testing to represent CWA vapor, aerosol, and droplet releases. Sensitivity studies should also be performed using the measurement simulation model to determine what are the most critical parameters to simulate in atmospheric tests to ensure the highest sensitivity to CWA characteristics.

    5. Test instrument detection with controlled releases. Conduct atmospheric tests with simulant and interferent vapors, aerosols, and droplets used in chamber tests to assess lidar system performance at ranges of detection and concentration levels required by the ORD. The range of tests must cover the simulation of all CWA vapors, aerosols, and droplets required by the ORD under a wide range of atmospheric and background conditions with different natural and battlefield aerosol distributions, topographical targets, and CWA release modes, which might affect the encountered aerosol size distributions in different delivery modes. Measurement results will be compared to model predictions based on “ground-truth” observations simultaneously made during the tests by different sensor techniques for providing the atmospheric parameters, background conditions, simulant concentrations and distributions (vapors and aerosols/droplets), topographic albedo, etc., that are needed as inputs to the measurement simulation model and for direct comparison with the lidar measurement.

    Suggested Citation:"4 Recommended Test Protocol and Decision Tree for Active Detectors (Lidar)." National Research Council. 2003. Testing and Evaluation of Standoff Chemical Agent Detectors. Washington, DC: The National Academies Press. doi: 10.17226/10645.
    ×
    1. Compare model results with controlled release results. The results from Step 8 are compared with the measurement simulation model results. The model must be shown to reliably predict the performance of the lidar system for the atmospheric simulant measurements under a wide range of conditions. Once the model is determined to accurately predict the lidar performance, a series of sensitivity studies can be conducted using measured and assumed parameters for atmospheric conditions, battlefield interferents, backgrounds, topographic scatterers, ranges of measurements, CWA concentrations and delivery methods, etc., to predict the performance of the lidar system in satisfying the ORD. Preliminary studies of this type can be conducted with the model prior to this step; however, it is not until this step that it can be said with confidence that the model is complete enough or accurate enough to predict the lidar system performance for CWA measurements. If the model fails this requirement, return to Step 1 to determine and correct the fundamental reasons for the model deficiencies. Every step in the protocol must be examined to discover and correct the model or the model input parameters. Step 7 must be repeated to provide updated predictions of atmospheric simulation test results. If these new results indicate that the atmospheric tests should be done differently, Step 8 must be repeated. If no new tests are indicated, the results from the previous Step 8 can be used in comparison with the model results. If the model still cannot provide the required performance, a final repeat of Steps 1 through 9 can be attempted before terminating testing of the detector.

    2. Conduct sensitivity studies to determine CWA measurement sensitivity. Once the model is determined to accurately predict lidar performance, a series of sensitivity studies can be conducted using measured and assumed parameters for atmospheric conditions, battlefield interferents, backgrounds, topographic scatterers, ranges of measurements, CWA concentrations and delivery methods, etc., to predict the performance of the lidar system in satisfying the ORD. Preliminary studies of this type can be conducted with the model prior to this step; however, it is not until this step that it can be said with confidence that the model is complete enough or accurate enough to predict the lidar system performance for CWA measurements.

    Failure of an instrument to meet the desired false positive/false negative rates in Step 10 with the signal-processing model developed in Steps 1 through 6 is considered failure. However, the learning in Steps 7 and 8 with a particular instrument may suggest adjustments to the model that might improve it. If this is the case, the instrument may be retested with an improved model. If such improvements to the model are not possible, the instrument is rejected as not meeting the criteria.

    POTENTIAL PROBLEM AREAS

    1. The aerosol and droplet size distributions and their time histories for CWAs in various modes of delivery may not be known and cannot be established.

    2. It may be difficult to generate representative size and number densities of aerosols or droplets of simulated CWAs in a large chamber for sensor testing.

    3. Some aerosol simulants that have physical characteristics similar to CWAs may not have suitable refractive indices or spectral dependence to their absorption to simulate the CWA aerosols or droplets (see Chapter 5).

    Suggested Citation:"4 Recommended Test Protocol and Decision Tree for Active Detectors (Lidar)." National Research Council. 2003. Testing and Evaluation of Standoff Chemical Agent Detectors. Washington, DC: The National Academies Press. doi: 10.17226/10645.
    ×
    Page 19
    Suggested Citation:"4 Recommended Test Protocol and Decision Tree for Active Detectors (Lidar)." National Research Council. 2003. Testing and Evaluation of Standoff Chemical Agent Detectors. Washington, DC: The National Academies Press. doi: 10.17226/10645.
    ×
    Page 20
    Suggested Citation:"4 Recommended Test Protocol and Decision Tree for Active Detectors (Lidar)." National Research Council. 2003. Testing and Evaluation of Standoff Chemical Agent Detectors. Washington, DC: The National Academies Press. doi: 10.17226/10645.
    ×
    Page 21
    Suggested Citation:"4 Recommended Test Protocol and Decision Tree for Active Detectors (Lidar)." National Research Council. 2003. Testing and Evaluation of Standoff Chemical Agent Detectors. Washington, DC: The National Academies Press. doi: 10.17226/10645.
    ×
    Page 22
    Suggested Citation:"4 Recommended Test Protocol and Decision Tree for Active Detectors (Lidar)." National Research Council. 2003. Testing and Evaluation of Standoff Chemical Agent Detectors. Washington, DC: The National Academies Press. doi: 10.17226/10645.
    ×
    Page 23
    Next: 5 Simulant Characteristics and Specifics »
    Testing and Evaluation of Standoff Chemical Agent Detectors Get This Book
    ×
    Buy Paperback | $47.00 Buy Ebook | $37.99
    MyNAP members save 10% online.
    Login or Register to save!
    Download Free PDF

    The report provides an independent assessment of suitable test protocols that might be useful and reliable for the testing and evaluation of standoff chemical agent detectors. The report proposes two testing protocols, one for passive detectors and one for active detectors, to help ensure the reliable detection of a release of chemical warfare agents. The report determined that testing these detectors by release of chemical warfare agents into the atmosphere would not provide additional useful information on the effectiveness of these detectors than would a rigorous testing protocol using chemical agents in the laboratory combined with atmospheric release of simulated chemical warfare agents.

    1. ×

      Welcome to OpenBook!

      You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

      Do you want to take a quick tour of the OpenBook's features?

      No Thanks Take a Tour »
    2. ×

      Show this book's table of contents, where you can jump to any chapter by name.

      « Back Next »
    3. ×

      ...or use these buttons to go back to the previous chapter or skip to the next one.

      « Back Next »
    4. ×

      Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

      « Back Next »
    5. ×

      Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

      « Back Next »
    6. ×

      To search the entire text of this book, type in your search term here and press Enter.

      « Back Next »
    7. ×

      Share a link to this book page on your preferred social network or via email.

      « Back Next »
    8. ×

      View our suggested citation for this chapter.

      « Back Next »
    9. ×

      Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

      « Back Next »
    Stay Connected!