Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 1
Test and Evaluation of Biological Standoff Detection Systems: Abbreviated Version Summary A biological warfare agent (BWA) is a microorganism (or a toxin derived from a living organism) that causes disease in humans, plants, or animals or that causes the deterioration of material (NATO 1996). The effectiveness of a BWA is greatly reduced if the attack is detected in time for the target population to take appropriate defensive measures. Therefore, the ability to detect a BWA, in particular to detect it before the target population is exposed, will be a valuable asset to defense against biological attacks. The ideal detection system will have quick response (detection on the order of seconds) and be able to detect a threat plume at a distance (on the order of kilometers) from the target population.4 The development of reliable biological standoff detection systems—standoff detection refers to detection at a distance from the aerosol or plume or detector5—therefore is a key goal. Detection of biological agents is complex. This report focuses on one aspect of the biological defense problem: How can we test whether a biological standoff detection system fulfills its mission reliably if we cannot conduct open-air field tests with live BWAs? Test and evaluation (T&E) of biological standoff detection systems to certify that they fulfill their mission is difficult because open-air field tests with BWAs are not permitted under international conventions and because the wide variety of environments in which detectors might be used may affect their performance. Further, a T&E protocol should provide the opportunity to demonstrate that a biological standoff detection technology can reliably differentiate between a plume that is benign and one that contains BWAs. Current and near-term standoff detection technologies are based on lidar (light detection and ranging). Lidar uses pulsed lasers to optically detect and characterize aerosols6 at a distance. The basic components of lidar are a transmitter (laser), a receiver, and a detector. Lidar uses the laser radiation that is scattered by aerosol particles to determine some property of the aerosol. The major types of lidar currently relevant to biological standoff detection are elastic-backscatter lidar, ultraviolet-laser-induced fluorescence lidar, high-spectral-resolution lidar, Doppler lidar, differential-scatter lidar, and depolarization lidar. Different lidar technologies attempt to exploit different signatures to infer characteristics of an aerosol. That is difficult because aerosol particles interact with laser radiation in different ways. Aerosol particles consist of many molecules and are therefore much larger than molecules. Their varied microphysical properties (that is, particle size distribution, shape, and composition) affect the extinction and backscatter intensity of the signal used to probe a BWA. 4 The committee recognizes that all BW threats are not aerosols, but for purposes of this report the committee considered all BW threats as aerosols, hence the emphasis on plumes. 5 The actual standoff detection distance may vary based on the operational requirements for a specific technology. For example, the Joint Biological Standoff Detection System (JBSDS) is required to detect aerosol plumes at a distance of up to 5 km and discriminate BWA from naturally occurring organisms from 1to 3 km. For the purposes of this report, standoff distance refers to that which is required by the JBSDS operational requirements document. 6 An aerosol is a system of particles suspended in a medium (in this case the atmosphere). For the purposes of this report, BWAs are assumed to be disseminated as aerosol particles.
OCR for page 2
Test and Evaluation of Biological Standoff Detection Systems: Abbreviated Version Once a plume has been detected, the varied characteristics of BWAs make discrimination between threat and non-threat plumes even more difficult. The conventional wisdom several years ago was that fluorescence cross sections of BWAs depended on the specific agents; for example, dipicolinic acid is a fluorophore that is present in Bacillus anthracis spores but not Francisella tularensis. However, recent laboratory work has shown that interspecies differences can be obscured by other effects (Hargis et al. 2007). This critical finding suggests that a standoff capability to detect the signatures associated with those effects may be more valuable than one that can detect only specific agents. Such information is important to both developers and evaluators in assessing test conditions. Department of Defense Approach The Department of Defense (DOD) is developing biological standoff detection technology in phases. The first objective is to be able to detect an approaching plume of biological material, whether human-made or natural, and determine its size and extent. The goal is to provide longer warning times than are provided by biological point detectors, which can detect biological agents based on direct samples of agent collected at the “point” at which the detector is placed. Point and standoff detectors have different abilities to determine whether a plume contains a BWA. Standoff detectors that can detect a plume but not determine its nature might be deployed in conjunction with point detectors placed upwind from a site that is to be protected. That would provide two independent kinds of warning: one that a plume is approaching and one that a plume contains a BWA. Standoff detection systems can be highly sensitive to various features in the atmosphere, such as clouds, dust, and the like. A military unit might react to information that a plume was approaching without knowing whether it was a natural aerosol plume, a dust storm, some other phenomenon, or an actual threat; soldiers might be ordered repeatedly to don personal protective equipment, although no biological threat is present, and this would reduce their operational effectiveness. In order to be effective, biological standoff detection requires a capability to at least identify the approach of concentrations of biological substances. If the false-alarm rate is not too high, and in conjunction with other intelligence or warnings, it might be acceptable to act on information that some sort of biological material is approaching even if it turns out to be only natural spores from nearby fields or trees. Under such a scenario, it might still be valuable to detect an approaching plume of biological material and order troops to find protection, even if the standoff detector could not determine whether the approaching biological material is dangerous. However, it would be important to know that the standoff detection system is capable of spotting enemy-made biological agents and not only natural airborne materials. Therefore, T&E of such a system would necessarily include a demonstration that the system is not blind to enemy-made biological agents either alone or in the presence of other materials, such as contaminants or interferents. Ultimately, DOD hopes to be able with standoff detectors to identify species in a plume, that is, to identify specific BWAs. To test and evaluate a detector with such capability, it would be necessary to demonstrate that the standoff detection system could dependably detect perhaps a dozen or more enemy-made biological agents in the presence of other atmospheric phenomena,
OCR for page 3
Test and Evaluation of Biological Standoff Detection Systems: Abbreviated Version contaminants, and interferents.7 T&E would thus require surrogates that could be released to imitate the behavior of live, enemy-made agents in the field; in the absence of such surrogates, tests with actual live BWAs would be necessary. Tests would need to be conducted in a facility with reliable safety features to contain the BWAs in question. It also would require test facilities to duplicate realistic field conditions. In some cases, the test facilities might have to be large so that the limitations of the test facilities do not interfere with the measurements being made. Test and Evaluation of Biological Standoff Detection Systems T&E needs to be tailored to the level of operational capability being tested. The T&E required for a biological standoff detection system that identifies a threat plume by species is substantially different from and more complex than for a detection system that only discriminates between biological and nonbiological aerosols, which requires more complex T&E than a system that only detects a plume. Therefore, T&E needs to occur in a graduated manner from detecting a plume, to discriminating between biological and nonbiological aerosol, and finally to identifying a specific BWA. Similarly, laboratory tests of a system should be used to inform the design and conduct of tunnel tests, whose results should be used to inform the design and conduct of open-air tests. Laboratory testing allows the determination of the impact of other effects on available signatures. From laboratory testing the T&E strategy should expand to examine the influences of the environment and different delivery systems on the system’s detection capability. Testing at this level may indicate new limitations (for example, if the presence of some urban contaminants masks BWA signatures or increases false-alarm rates) or indicate opportunities to enhance detection capabilities (for example, if it is found that cueing by other detectors to focus on a particular event might allow better interrogation of a plume by the standoff detector system, which may require a change in system requirements). This testing offers another chance to assess where programmatic and test strategy decisions should be reviewed. Standoff detectors with no ability to identify specific biological agents require less elaborate test facilities than would highly capable futuristic systems. Although less elaborate test facilities are required for the T&E of a standoff detector that is intended to only measure the extent of an approaching plume and determine its dimensions and velocity, the T&E capabilities required are still challenging. DOD does not have an adequate set of tools to test biological standoff detection systems fully under both laboratory and field conditions and at component, subsystem, and full-system levels and then to correlate the results. It has invested in developing test facilities for standoff systems that can detect plumes, but the facilities are not suitable for standoff experiments requiring containment, although some may have limited containment capabilities. The technical and engineering difficulties of developing a set of facilities, complemented by modeling and simulation tools, that can accommodate active standoff detection system development while providing necessary containment are substantial and will make it expensive to develop such facilities. 7 The committee recognizes that this assumes the threat agent to be detected is actually known and cataloged. In reality the committee understands that by definition “enemy-made” could be an engineered biological agent not previously known.
OCR for page 4
Test and Evaluation of Biological Standoff Detection Systems: Abbreviated Version The committee believes that the technology to provide standoff identification of specific BWAs is not likely to be practicable for many years. Current research indicates that the signals detectable by current technologies associated with other effects can overwhelm the signals from specific biological species. This critical finding means that substantial work will be required to characterize those effects before a standoff capability to identify specific BWAs is possible. Not only is the technology for standoff species identification challenging, but the T&E of such systems would be difficult and complex. Finally, being able to identify specific threat species assumes that there is both a known set of threat agents and that certain procedures associated with those agents are known. Characterization of novel unanticipated threats is not easily imagined. Findings and Recommendations The findings and recommendations presented below are intended to provide DOD and the office of the Product Director for Test Equipment, Strategy, and Support with a clear path to developing a robust process for testing and evaluation of biological standoff detection systems. FINDING: The Department of Defense requires an integrated approach to development, testing, and evaluation of biological standoff detection systems. DOD needs a comprehensive T&E process that uses design of experiments and integrated statistical analyses to dynamically and iteratively combine modeling, simulation, theory, and laboratory efforts with the results of field testing. The process should quantify, through models and data, the performance of every hardware and software component and the interaction of the interrogation methods with the atmosphere and with aerosol particles. The performance quantification needs to include uncertainty quantification. The process needs to integrate diverse types of information into a single framework, including data from laboratory, tunnel, and open-air testing and data from modeling and simulation. All aspects of T&E should be integrated and coordinated with the development of biological standoff detection systems. T&E should inform the development of biological standoff detection systems, and interactions between the development and T&E communities should be a routine part of the research and development process. Tracking opportunities for development will allow the T&E community to insert new test technology into the T&E process and develop new T&E capabilities to address evolving needs. FINDING: Current understanding of the relationship between detection signals and the properties of biological aerosols is insufficient to allow reliable correlation between surrogates and agents. A process for building a body of knowledge about the interaction of laser radiation and other active interrogation techniques with aerosol (especially biological) particles and plumes needs to be developed. ALOs and simulants have been used as surrogates for live BWAs in testing, but the corresponding T&E results are valid only if a detection signal from a live BWA can be predicted from the detection signal of a surrogate. Developing a robust model for prediction requires an understanding of how a detection signal correlates with biological aerosol properties. Achieving a high level of confidence requires comparison of predicted and measured signals in a laboratory test bed that can accommodate live BWAs. There appears to have been
OCR for page 5
Test and Evaluation of Biological Standoff Detection Systems: Abbreviated Version insufficient emphasis on integrating laboratory-scale T&E to support such studies, which can include conventional and scaled-down test beds. FINDING: Referee systems need to be more accurate and precise than the system under test. Calibrated lidar is necessary to compare past, present, and future lidar data. Calibrated lidar provides backscatter intensity and extinction data in physical units (for example, per meter per steradian) that allow the data to be used in models of the integrated system. In recognition of the importance of lidar as a key instrument in T&E of biological standoff detection systems, the understanding, calibration, and reproducibility of lidar data should be emphasized. The most accurate technique to calibrate aerosol lidar is high-spectral-resolution lidar (HSRL;(Eloranta 2005), and there are eye-safe HSRL systems. Tunnel and range facilities require several well-calibrated nephelometers8 and visibility sensors to document scattering and extinction of both plume and background. A combination of point-based referee systems9 is necessary in tunnel and scaled-down testing to measure the size distribution of the aerosol plume and background as accurately as possible. The current Aerodynamic Particle Sizer can measure the size distribution only down to a diameter of 0.5 μm. Smaller particles in the plume require alternative detection technologies, such as the Scanning Mobility Particle Sizer. Such measurement capability is critical for understanding how aerosol particles contribute to the overall signal detected by a standoff system. RECOMMENDATION: The Department of Defense should develop an integrated framework for test and evaluation of biological standoff detection systems that includes modeling and simulation; uncertainty quantification; and laboratory, tunnel, and open-air testing. Viewing the T&E process as an integrated array of capabilities—as a “T&E system”—can improve overall T&E capabilities substantially. Design of experiments should be used to guide the integrated T&E framework, and uncertainty quantification should be an inherent goal. Uncertainty quantification is a vital component of the integrated framework. Its purpose is to measure system performance and to measure errors present in each combination of test conditions and test modality (laboratory, tunnel, open-air, and modeling and simulation testing). The evaluation process is limited without an appropriate quantification of both system performance and the associated uncertainties. Once the evaluation process is associated with quantified uncertainties, the T&E community needs to make decisions in light of the uncertainties. Each decision should be based on an evaluation of the additional reward or benefit that it offers with respect to uncertainty quantification. Modeling and simulation (M&S) play an important role in an integrated array of T&E capabilities. Test and experimentation capabilities should be considered with respect to the data they provide and how the data can be integrated into a strategy that includes M&S. Data from experimentation can help develop and improve M&S with the goal of creating a “virtual test bed” or “virtual T&E infrastructure.” This will reduce overall cost while expanding test capability, especially for T&E of systems, such as biological standoff detection systems, that can 8 A nephelometer is an instrument used to measure the light-scattering coefficient of aerosols. 9 A referee system, sometimes referred to as a ground truth system, is used to characterize an aerosol plume at a higher fidelity than the system under test.
OCR for page 6
Test and Evaluation of Biological Standoff Detection Systems: Abbreviated Version never be physically tested under realistic field conditions. The M&S capabilities developed to test nuclear weapons after nuclear tests were banned are an example of such work. RECOMMENDATION: The Department of Defense should assess the development and construction of new facilities that are suitable for whole-system testing of standoff detectors, for example, a large Biosafety Level 3 facility, in the context of an integrated approach to test and evaluation. It is possible that a large-scale Biosafety Level 3 (BSL-3) chamber will be required to demonstrate that a full-scale biological standoff detection system works reliably. However, given the current state of knowledge, building such a facility is premature. One critical issue that must be resolved before a BSL-3 facility is built is the impact of other effects on the detection signal.10 An integrated T&E protocol must be an essential part of decision making regarding a large-scale BSL-3 chamber for testing standoff detectors. The design and characteristics of a BSL-3 facility would depend on the understanding gained from other aspects of the integrated T&E approach and on the identified knowledge gaps that could not be filled by existing facilities. It is critical to understand how data from a BSL-3 facility will be integrated into a framework that can predict the response of a detection system to an open-air release of a biological threat under realistic field conditions. With the current level of understanding, it is not clear how data from a BSL-3 facility could be reliably interpreted. Laboratory and tunnel testing and M&S efforts should be pursued to enhance the state of knowledge needed to decide whether to build a large BSL-3 chamber. It is likely that additional facilities that enhance current T&E capabilities will be required. For example, scaled-down test beds with containment capabilities are likely to produce valuable data. Development of these capabilities should be pursued before other more risky and expensive options are considered. Building a large-scale BSL-3 facility is a high-risk endeavor given the technical difficulties and likely cost. Specifically, it is unclear whether the technical solutions proposed in the preliminary designs for the facility (see Chapter 4) would be sufficient to result in a facility that is capable of producing the requisite data. RECOMMENDATION: The test and evaluation community should place more emphasis on testing the capability of standoff detection systems to discriminate between human-made biological aerosols and natural biological aerosols. The rational behind this recommendation is discussed in the committee’s full report. RECOMMENDATION: Design of experiments should be used to efficiently explore the array of test conditions required to characterize a detection system’s performance fully in different environments and threat scenarios. In a given test modality, many factors affect a detection signal. Experimentation that tests one factor at a time is highly inefficient. Design of experiments affords an opportunity to choose combinations of factors that will yield results with the same information content as one-factor-at-a-time experimentation but at a much lower cost. A sound experimental design approach will also allow the T&E process to determine potential interactions between factors. And a sound experimental design approach can inform and guide research and development 10 These effects are discussed in the committee’s full report.
OCR for page 7
Test and Evaluation of Biological Standoff Detection Systems: Abbreviated Version efforts if the research and development and T&E processes are appropriately managed and coordinated. In many highly complex and expensive systems, adaptive design of experiments is a valuable tool. Adaptive design can be broadly classified as follows: “Given the uncertainty (as evidenced by test data, models, analysis of results, and so on) in a system at time t, what modality of testing, which factors, and what levels of those factors can maximize the reduction in uncertainty at time t + 1?” The term adaptive refers to the notion that the question is asked after each test is run. Adaptive designs are valuable, but they are also sequential; it takes substantial time to complete a fully adaptive approach to testing and evaluation. Thus, a hybrid approach that contains small-scale classical design of experiments and interim assessments of uncertainty would provide an opportunity to evaluate uncertainty periodically and to revise the test plan. Design of experiments should extend beyond laboratory, tunnel, and open-air concepts to include M&S experiments that are recommended in the integrated approach to T&E. The design-of-experiments protocol for the M&S process should be developed in light of the design-of-experiments approach for laboratory, tunnel, and open-air experiments. The M&S approach should be conducted according to principles of design of experiments and with the objective of linking laboratory, tunnel, and open-air results. RECOMMENDATION: The test and evaluation community should use the operational requirements for biological standoff detection systems to drive the development of testing and evaluation. The need for T&E capabilities is determined in part by the performance requirements of the standoff detection system under scrutiny. The likelihood that additional T&E capabilities are needed depends on whether the goal of the standoff detection system is to detect an aerosol plume, to detect a plume and determine whether it contains aerosol of biological origin or not, to detect a plume and determine whether its signature is biological and human-made, or to detect a plume and identify its signature as a BWA. Regardless of the biological standoff detector’s performance requirements, laboratory testing will be required to establish correlations between BWAs and surrogates. Data for the correlation can be collected in confined chambers. Once the correlation is established, if the goal of the standoff detection system is simply to detect an aerosol plume without regard to its composition, whole-system testing with simulants can be conducted to establish the detector’s performance under operationally relevant conditions. The correlation between BWAs and surrogates (in this case simulants) is required to ensure that the standoff detection system is not blind to aerosols that contain BWAs. If the goal of the standoff detection system is to detect plumes and distinguish between those that contain biological aerosols and those that do not, chemical characterization is needed in addition to physical and optical characterization. Chemical characterization by identification of the signatures associated with material of biological origin (such as amino acids) requires laboratory testing with surrogates, such as simulants, ALOs, or killed or inactivated BWAs. In addition to open-air testing with simulants, tunnel testing with killed ALOs may be used to reduce the uncertainties in the detector’s potential performance with BWAs. It may be possible to do some testing with risk group 2 (RG-2) organisms, such as live ALOs in the active standoff chamber or a facility similar to it if sufficient containment is demonstrated to further reduce the uncertainties.
OCR for page 8
Test and Evaluation of Biological Standoff Detection Systems: Abbreviated Version If the goal of the standoff detection system is to detect plumes and distinguish between plumes that contain human-made biological aerosol and plumes that contain naturally occurring biological aerosol, unique chemical characterization is required in addition to physical and optical characterization. Unique chemical characterization is required to detect evidence of a human “fingerprint” on the aerosol. Laboratory testing with such surrogates as simulants, ALOs, or killed or inactivated BWAs will be required. In addition to open-air testing with simulants, tunnel testing with killed ALOs may be necessary to predict the detection system’s performance on BWAs. As before, it may be possible that some testing with RG-2 organisms can be conducted in a facility to reduce the uncertainties in correlating surrogates with BWAs. These tests should also vary certain procedures to determine the effects of those procedures on the target signal. If the goal of the standoff detection system is to identify a specific BWA, controlled releases of live ALOs, killed or inactivated BWA, or even live BWA in a high-containment facility may be required. However, such testing is not necessary at the present time. RECOMMENDATION: The test and evaluation community must base its test and evaluation of biological standoff detection systems on a system’s measures of performance, measures of effectiveness, and concepts of operations. Materiel developers, combat developers, and T&E professionals must have a clear understanding of what information a candidate standoff system can provide and of the operational value of this information. In determining what information requirements a system can satisfy, developers and evaluators must consider the entire family of detector systems that commanders will have available to them and how detection functions may best be allocated to different systems. Identifying reasonable expectations for a technology allows applied development programs to focus on delivery of equipment designed to satisfy operational needs and allows T&E professionals to determine operationally relevant testing protocols. RECOMMENDATION: The Department of Defense should foster the development of a multidisciplinary biological testing community with increased interactions with the broader research community. The T&E community would benefit if staff and contractors were supported in getting results published in refereed literature. There is tension between protecting national security and maintaining open scientific exchange, but many aspects of the basic research carried out by members of the standoff detection group would benefit from peer review. Outside unbiased evaluation of experimental protocols and results would help guide T&E strategies. The annual Joint Conference on Standoff Detection for Chemical and Biological Defense held in Williamsburg, Virginia, publishes its proceedings, and the standoff detection community should present results at other conferences and publish in peer-reviewed literature. Because the field of biological standoff detection continues to evolve, the T&E community would benefit from a scientific advisory board composed of independent engineers and scientists to provide continuing direction and integration of the science and technology, development, and operational communities. The board could help develop measures of operational effectiveness and system performance and allow for review of emerging test results between phases of system definition and development. This advisory board might interact with the DOD Test Resource Management Center (TRMC). A T&E science and technology program
OCR for page 9
Test and Evaluation of Biological Standoff Detection Systems: Abbreviated Version exists in the DOD TRMC to develop new test technologies for enhancing T&E capabilities. It would be beneficial if the TRMC included biological T&E as a focus area. Finally, greater interaction between the biological point detection and standoff detection communities is urged. Analysis of the concept of operations for either kind of detection program suggests that exchange is crucial during T&E, inasmuch as information from the deployed systems must be exchanged to achieve detection for warning and protection.
OCR for page 10
Test and Evaluation of Biological Standoff Detection Systems: Abbreviated Version This page intentionally left blank.