Testing and Evaluation of Standoff Chemical Agent Detectors (2003)

The use of simulants is the basis of the protocol described in the Battelle report for validation of the Joint Service Lightweight Standoff Chemical Agent Detector (JSLSCAD), the passive standoff detection system for chemical warfare agents (CWAs).¹⁰ These are chemical species that have a similar spectral response to the CWA of interest, with a known factor relating the absorption cross-sections. Signal-processing models are developed in chamber measurements for simulants to establish a detection limit, both alone and in the presence of possible interferents. The detection limit for the CWA is then assumed to be related to the detection limits for the simulants by the factor derived from the relative absorptivities. The signal-processing model is then checked with simulants in live-field measurements. The assumption is that similar results would be obtained in live-field tests of a CWA once the factor of the relative absorptivities is taken into account.

The committee concluded that the protocol recommended in the Battelle report is inadequate and that the assumptions on which it is based are incorrect. The major flaw in the protocol is the lack of information about the highly variable backgrounds that will be observed in the field. The background for chamber tests is an extended blackbody source of known temperature, whereas the background for field measurements is likely to be highly variable and could include forests, hillsides, sky, grasslands, lakes, etc., for which the emissivity will not only be less than 1 but will almost certainly vary across the spectrum. If this high variability results in adjustments to the model for simulants developed from chamber testing where the backgrounds are constant (though adjustable in terms of temperature of the blackbody source), there is no way to adjust the model for agents without running live-field tests.

The committee proposes an alternate protocol that it believes will address some if not all of these

concerns. The protocol involves the use of simulants but in a significantly different fashion. Since the highly variable background issue is likely to be dominant, the protocol starts there. The initial task is to obtain a large sampling of background spectra acquired under field conditions. Enough spectra must be taken to fully span the space of all background conditions likely to be encountered in use of the standoff detectors. Background spectra must include field interferents such as smoke, diesel soot, dust, and propellants. Laboratory spectra are then taken of all CWAs, simulants, concomitants (e.g., adhesives, thickeners, propellants), and possible interferents (e.g., smoke, diesel soot).¹¹

The spectral data obtained in this way can be used with the acquired field background spectra to synthesize spectra that would be observed under field conditions with said simulants, concomitants, interferents, and agents present. The simulation of spectra over a range of temperature differentials will require knowledge of the approximate background temperature of the scene being used. This large collection of background spectra provides data for training and verification of a signal-processing model to extract concentrations from measured field spectra. Since the chamber has a constant blackbody background, this corresponds to an ideal representation of the field measurements. Thus with data from the chamber, the model can now be used to verify that various real mixtures of simulants, interferents, and possibly concomitants can be measured under ideal conditions. If the signal-processing model can be validated for a variety of simulants—representing the spectral properties of known or expected CWAs—from chamber measurements, it can then move on to field measurements of simulants with interferents present in the field using the same signal-processing model. Successful validation of the model using these simulants and interferents in field measurements validates the transfer of the signal model from chamber to field. This is the underlying basis for the use of simulants in this protocol. This result would give relatively high confidence that the signal-processing model for CWAs, if successful in chamber measurements, would transfer successfully to field measurements without actually making live-agent field measurements. Furthermore, field measurements with CWAs would not provide statistically significant verification of the robustness of the model unless a very large number of live-agent tests were conducted under a wide variety of conditions.

Infrared spectral emission and absorption are related linearly to both the product C × L (for low concentrations, i.e., the sample is considered to be “optically thin”), where C is the concentration and L is the path length, as well as the temperature differential, ?T, of the CWA relative to the background. In field applications of such detectors, the CL product will provide important sensitivity limitations to the detection provided ?T is not zero and ideally greater than 3°C.¹² The delivery and release mechanisms for CWAs (explosive shells, fogging from an aircraft, release from a compressed state) will likely provide sufficient temperature differential for second or minutes, to allow such detectors to detect the infrared emission and absorption of the released material.

Flow charts of the test protocols are given in Figures 1 and 2 to make it easier to visualize the steps and decision points in the protocol. The following text and figures are keyed to be used together.

Step:

1. First, a very large number of background spectra (on the order of thousands of independent spectra) should be measured under as wide a range of battlefield conditions as possible. Spectra might, for example, be collected during military training maneuvers in a variety of locales. While these background spectra are being measured, a videotape of the scene should also be recorded so that descriptive scene information can be used as a reference. No CWAs or simulants should contribute to the spectra. Backgrounds should have as wide a range of emissivity as possible (from near-blackbody to sky). Spectra measured with the JSLSCAD should be measured both at a resolution of 16 cm^–1 while the detector gimbal is rotating and at a resolution of 4 cm^–1 while the gimbal is static. If other instruments are developed in the future, corresponding measurements should be made at both low and high resolutions. This will permit a low-resolution rapid scan to sense the “presence or absence” of CWAs and a high resolution scan for final decision making. It is suggested that every time new nonchemical weapons are being tested, at least four JSLSCAD instruments should be mounted so that as many background spectra as possible can be acquired. The collected spectra should be divided into two sets—one set (A) for training the signal-processing model and one set (B) for the validation of the model, with Set A being about twice as large as Set B. The spectra in Set A should all have been measured from different sites than those in Set B.

2. Acquire reference transmission spectra of as many simulants, concomitants, and interferents as possible under lab conditions (i.e., using an incandescent source and with the sample contained in a long-path cell at a known partial pressure and temperature). Make up all samples to 101 kPa (760 torr) with dry nitrogen. These spectra should be acquired at a resolution no worse than 4 cm^–1. The original interferograms should be retained so that the spectra can be readily degraded to other resolutions without the need to re-measure the spectra. Ensure that simulants of low volatility are included in these reference spectra. In addition, spectra should be obtained at various temperatures to determine any temperature effects on key vibrational features.¹³

3. Using the data acquired from Steps 1 and 2, synthesize two sets of spectra of simulants (Set A and Set B) under field conditions so that a wide range of CL products and ?Ts are included, where C is the concentration of the sample (ppm), L is the effective path length (m), and ?T is the difference in temperature between the sample and the background (ºC). Some of the background spectra should have been measured when common interferents (e.g., smoke, diesel soot) were present. At least 50 times as many spectra should be synthesized in this way than there are background spectra in sets A and B, using at least 50 spectra calculated with different CL products and ?Ts for each background spectrum. Linear dependence of the data should be avoided. For half of the spectra in each set, one of the simulants should be present and for the other half no simulants should be present. Set A will be used for training and Set B for validation in a manner analogous to the protocol developed by Yang and Griffiths.¹⁴ It is possible that a number of different sets with different ranges of CL products and ?Ts for each simulant will need to be synthesized in this step. In this case, single background types (e.g., sky) should be used first and complexity added at a later stage of the training.

FIGURE 2 Test protocol with CWA for passive detectors.

4. Using the training spectra synthesized above (i.e., the calculated spectra in Set A), train an appropriate algorithm to recognize the presence of each simulant. The model(s) obtained in this way will be the basis of all subsequent tests. Initially, training and validation spectra should be calculated from similar backgrounds (e.g., sky and hillside backgrounds should not be included in the same set). The background spectra should then be expanded with the hope of calculating a universal model. If a universal signal-processing model is not feasible, software designed to classify background spectra into appropriate sets should be developed.

5. Validate the algorithms and models developed in Step 4 using spectra calculated from Set B. Find the limits of CL and ?T for each simulant. If the algorithm is validated in this step, then repeat steps 1 through 5 using a more complex background set. This cycle is repeated until a background set is used that results in the failure of the algorithm. The last validation of the algorithm (n–1) thus defines the limits of applicability of the instrument in terms of its successful operation with increasingly complex backgrounds.

6. Acquire passive spectra of simulants and interferents in a chamber with a blackbody source to validate the model developed in Steps 4 and 5 using the same type of instrument used to collect the background spectra operated at the same resolution(s) used in the field.

7. Acquire spectra of simulants in the presence and absence of interferents (e.g., smoke, diesel soot)

in the field in order to validate the model developed in Step 4 and tested in Steps 5 and 6 using at least four different JSLSCAD instruments. At the same time, “ground truth” transmission spectra should be measured in two perpendicular directions through the simulant plume to find how CL varies throughout the series of measurements independently of ?T. If possible, the temperature of the plume and the relevant background should be measured so that ?T is known. (In practice, however, such measurements may be difficult to make.)

8. Acquire reference transmission spectra of pure agents and mixtures of agents along with expected concomitants (e.g., adhesives, thickeners, propellants), using the same conditions as in Step 2.

9. Repeat Steps 3 through 6, including agents of interest. The range of CL products and ?Ts should be specified by the Defense Threat Reduction Agency.

Repeat the tests established for vapor-phase measurements (Steps 1 through 9) for low-volatility simulants where aerosol formation is known to occur. A number of reference spectra of monodispersed aerosols with different particle sizes should be measured if possible. As wide a range of droplet diameters as possible should be covered. In addition, spectra of polydispersed aerosols with known ranges of diameters should be measured in order to compare spectra computed by adding the appropriate spectra of monodispersed aerosols of different sizes with the spectra of polydispersed aerosols. The simulants should ideally have physical and spectral characteristics similar to the CWA they are designed to simulate. If there is a need to choose between physical and spectral characteristics, it is more important to match the physical characteristics than the spectrum.

For chamber testing, make sure that simulants and concomitants with as wide a range of physical properties (Henry’s law constants, viscosity, surface tension, etc.) are represented.

The “pass/fail” criteria for the performance of the instrument in this test protocol are the false positive and false negative rates, as defined in the Operational Requirements Document (ORD) for the device.¹⁵ While this rate is defined in the present documentation, different acceptance requirements might be appropriate for different circumstances (i.e., military vs. civilian population protection, etc.). In all cases the acceptable false positive/false negative rates are the figures of merit for the test protocols.

Several points in the test protocol have decision nodes (“diamond” symbols in Figures 1 and 2). The first of these (Step 5) defines the scope of application of the instrument with each succeeding “Yes” loop. With each loop, additional and more complex background spectra are added to the “training” and “verification” sets. When or if the instrument finally fails to “pass” this node, the accumulated background scenarios for which the instrument performed acceptably define the application limits on the instrument. There is also an option at this node that a change to the ORD can be implemented before continuing the test to subsequent steps. This would either loosen or tighten the acceptance criteria, depending on the trade-offs desired in the utility of the equipment.

Failure to pass subsequent decision nodes in the test protocol indicates that the particular instrument is not capable of adequate field performance against the ORD. Successful performance through all subsequent decision nodes indicates that the instrument will provide field detection of CWAs with the expected false positive/false negative rates in the field conditions as defined in Step 5. As will be

subsequently discussed under the risk assessment (Chapter 7 and Appendix C), it needs to be recognized that the essentially infinite array of field conditions will likely cause some deterioration of instrument performance against the metrics used in testing, and there is no practical way of estimating this deterioration beyond the background testing that has been included in the signal-processing model, short of adding even more background spectra to the model.