4
Implications of BAULA for Detector Test and Evaluation

The benefits provided by the BAULA framework are dependent on its standard and consistent application to test and evaluation of detectors. As described in previous chapters, the recommended unit, BAULADae, is designed to accommodate advances in both detection technologies and our understanding of the effects BWA aerosol exposures have on humans. As use of the unit adjusts to new information, it will be important to be able to relate requirements and test results to historical data. Current test and evaluation (T&E) procedures will have to be adjusted to make best use of the new unit. This chapter describes several key components of the complex T&E process and discusses how they might be affected if the BAULA framework is implemented.

4.1
THE TEST AND EVALUATION PROCESS

Separate agencies within the Department of Defense (DOD) contain the functions of (1) determination of requirements, (2) research and development, (3) procurement, and (4) test and evaluation for chemical and biological warfare defense. These agencies interact to develop, evaluate, and ultimately procure systems for BW defense. In general, the DOD development and acquisition process is divided into four stages: (1) early research and concept development, (2) technology development, (3) systems development, and (4) production and deployment. Transitions between these stages are further designated by evaluation points referred to as Milestones A, B, and C. Development across these four broad stages is further divided by eight descriptive technology readiness levels (TRLs). The transition from a developmental system to a production and procurement process is denoted by Milestone C. At this point, an independent agency must certify that the system meets the specified performance requirements before the system enters initial production and procurement. Establishment of a universal framework for health risk evaluation of BWAs will be especially useful in coordinating and aligning the activities of the various agencies and offices involved in this lengthy process.

Experimental performance tests are an integral part of several steps in the system research and development process, from early feasibility demonstration to evaluating candidate technologies. Different test facilities are used at different stages. Prior to milestone C, (BW) detection systems, or key components, are often tested with various simulants instead of live agents for safety and cost reasons. This simulant testing is typically done at test chambers and aerosol wind tunnels either at developers’ facilities or sometimes at DOD-maintained test facilities (such as Dugway Proving Ground (DPG) in Utah, or Edgewood Chemical and Biological Center (ECBC) in Maryland) for critical program decision points. For more mature systems, carefully designed and controlled experiments are necessary to determine the basic performance measures: response time, probability of detection, probability of false alarm, and



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 47
4 Implications of BAULA for Detector Test and Evaluation The benefits provided by the BAULA framework are dependent on its standard and consistent application to test and evaluation of detectors. As described in previous chapters, the recommended unit, BAULADae, is designed to accommodate advances in both detection technologies and our understanding of the effects BWA aerosol exposures have on humans. As use of the unit adjusts to new information, it will be important to be able to relate requirements and test results to historical data. Current test and evaluation (T&E) procedures will have to be adjusted to make best use of the new unit. This chapter describes several key components of the complex T&E process and discusses how they might be affected if the BAULA framework is implemented. 4.1 THE TEST AND EVALUATION PROCESS Separate agencies within the Department of Defense (DOD) contain the functions of (1) determination of requirements, (2) research and development, (3) procurement, and (4) test and evaluation for chemical and biological warfare defense. These agencies interact to develop, evaluate, and ultimately procure systems for BW defense. In general, the DOD development and acquisition process is divided into four stages: (1) early research and concept development, (2) technology development, (3) systems development, and (4) production and deployment. Transitions between these stages are further designated by evaluation points referred to as Milestones A, B, and C. Development across these four broad stages is further divided by eight descriptive technology readiness levels (TRLs). The transition from a developmental system to a production and procurement process is denoted by Milestone C. At this point, an independent agency must certify that the system meets the specified performance requirements before the system enters initial production and procurement. Establishment of a universal framework for health risk evaluation of BWAs will be especially useful in coordinating and aligning the activities of the various agencies and offices involved in this lengthy process. Experimental performance tests are an integral part of several steps in the system research and development process, from early feasibility demonstration to evaluating candidate technologies. Different test facilities are used at different stages. Prior to milestone C, (BW) detection systems, or key components, are often tested with various simulants instead of live agents for safety and cost reasons. This simulant testing is typically done at test chambers and aerosol wind tunnels either at developers’ facilities or sometimes at DOD-maintained test facilities (such as Dugway Proving Ground (DPG) in Utah, or Edgewood Chemical and Biological Center (ECBC) in Maryland) for critical program decision points. For more mature systems, carefully designed and controlled experiments are necessary to determine the basic performance measures: response time, probability of detection, probability of false alarm, and 47

OCR for page 47
48 minimum concentration of detection. Data from field tests, ambient breeze tunnels, and open-air sites are then related back to component tests for overall system evaluation. To expand detector evaluation capabilities, the DOD is also planning a Whole System Live Agent Testing (WSLAT) chamber to conduct live agent tests of entire systems. A brief overview of testing in laboratories, ambient breeze tunnels, and open-air sites is provided in Box 4.1. BOX 4.1 Characteristics of Different Testing Facilities Laboratory Testing In the laboratory setting, aerosol wind tunnels and test chambers with different biosafety levels (BSL)(Centers for Disease Control and National Institutes of Health 2007) to accommodate simulants and live agents have been used. Laboratory experiments provide a test environment with well-controlled challenge aerosol, wind speed, wind direction, and background interferent aerosol. Depending on the sizes of the detector system and the test chamber, the whole detection system or individual components can be evaluated. A well-controlled test environment is especially useful in the initial stage of development, when laboratory tests can provide information to modify, redesign, and improve the system. Tests in aerosol wind tunnels provide information about sampling acquisition, including the aspiration efficiency and transmission efficiency as a function of particle size, wind speed, and sampling flow rate.(Cheng et al. 2004; Cheng and Chen 2001) BSL-2 or BSL-3 test chambers enable the use of live biological agents or their simulants and are used to assess the performance of biological agent detection. In current test chambers, biological agents are usually released as a well-mixed aerosol of a constant concentration level or as spikes of agent concentration.(Jensen et al. 1992; Li and Lin 1999; Xu et al. 2003; Semler, Roth, and Semler 2004; Kesavan 2005) By varying the biological agent concentration, one can determine the minimum concentration of detection as well as response time. One can also introduce interferents into the chamber to determine discrimination capability and estimate the probability of false positive detection. Open Air Testing Eventually operational tests are conducted on an open air range to evaluate detector performance under real-world conditions. In these tests, simulants and agent-like organisms are released in the open air test. Active agent is never released during these tests due to the health hazard these organisms pose to the population.(National Research Council 2005) Ideally these tests would further characterize the detector’s response time and false-positive detection rate in the presence of natural background. However, field tests can be time consuming and costly because of variable experimental conditions and environmental impact regulations. Highly variable environmental conditions (e.g., wind speed, wind direction, temperature, and relative humidity) affect the aerosol dispersion and introduce variability in the test data, which can require many replicate tests to average out. Breeze Tunnels Ambient Breeze Tunnels (ABT) provide an environment for controlled dispersion of challenge aerosols, while maintaining certain characteristics of open air conditions, including temperature, relative humidity, turbulent mixing, and even background aerosol. These tunnels are usually longer than 100 ft and have large cross-sectional areas. The wind is unidirectional, with speeds up to 5 mph. The tunnels— like a laboratory aerosol wind tunnel—have been designed to provide uniform aerosol concentration at the test section for the specified wind speed range. Because of the nature of remote detection, standoff detectors, such as LIDAR, can only be tested and evaluated in an ABT or an open air field. Both the open air field and ABT can provide performance test data on operational, full-scale detection systems.

OCR for page 47
49 4.2 THE CURRENT TEST AND EVALUATION PROCESS The committee has recommended the adoption of BAULADae as part of a health risk evaluation framework because it allows cross-comparison of diverse biological agents to a single, common quality: that of health hazard. The current unit, ACPLA, requires only measurement of how many agent-containing particles are found per liter of air. BAULADae requires that several additional characteristics be considered: how much total agent is present, how much of it is active, the size distribution of the agent-containing particles, and the probability of a negative health outcome (Figure 3.1). All of these factors have an important effect on the potential of the particular agent-containing aerosol to cause illness or death. Adopting the BAULADae unit, therefore, has several implications for the T&E system. Most importantly, referee systems—the part of the test system that characterizes the conditions experienced by the detector—will have to be able to measure all the characteristics included in the BAULADae framework. Only then will referee systems be able to evaluate detectors accurately. The remainder of this chapter is devoted to considering some of these implications and describing how current test and evaluation protocols can be adapted to provide evaluation of detectors in BAULADae units. Figure 4.1 shows a generalized test system layout. FIGURE 4.1 Sample of the agent itself or a surrogate is prepared, aerosolized, transported across the test site, and then sampled by various referee instruments. At each of these stages, the biological activity and particle size distribution may be altered. 4.2.1 Sample Preparation An early decision in the T&E procedure is the choice and preparation of the “test material,” which includes the BWA itself or a surrogate, possibly along with nonbiological particles and droplets, and interferents. The biological test material may consist of: • The BWA itself: Bacterial spores, vegetative bacterial cells, viruses, or toxins. • Agent-like organisms (ALOs): Organisms having physiological, physical, and chemical properties similar to those of a corresponding BWA while presenting a reduced risk of infection. ALOs are most often derived from a vaccine or attenuated strain of a BWA, or a non-viable or inactive form of a BWA. • Simulants: Nonpathogenic or nontoxic BWA surrogates that provide useful

OCR for page 47
50 evaluative information on the performace of a biodetection system and can sometimes be directly correlated with the BWA being simulated. For example, Bacillus subtilus var niger (BG) spores are frequently used as a simulant for bacterial spores. Erwinia herbicola (EH) has been used as a simulant of vegetative bacteria, MS2 bacteriophage as a simulant of virus, and ovalbumin as a simulant of toxins. The appropriateness and utility of certain organisms as simulants is an ongoing topic of debate. The choice of biological test material takes on a new importance if the standard of evaluation is changed from ACPLA to BAULADae. Detectors being evaluated for their sensitivity in ACPLA are not currently required to distinguish between active and inactive agents, while BAULADae specifically requires detection of active agent. Furthermore, BAULADae ideally incorporates information about differences in virulence among strains of similar agents by considering the relative LD50 values of the strains. Thus, tests using vaccine strains or simulant may not give a direct indication of the detector’s sensitivity in BAULADae units. Both cost and safety considerations affect the degree to which detectors can be tested against live, active agent—especially in an open air setting—so it is clear that future T&E systems will continue to use simulants. Over time, it is hoped that adoption of BAULADae will stimulate the development of detectors that distinguish strain differences and active from inactive agent. In the meantime, it will be important to bear in mind that detector technologies sensitive to inactive agent may have a high false positive rate when performance is measured against a referee system that measures active agent. Because BAULADae explicitly requires distinction between active and inactive agent, sample preparation techniques will need to be standardized so that the proportion of agent killed or inactivated during the preparation process is known. Variation in test sample preparation can lead to significant differences in BAULADae values. These considerations will be particularly important in the immediate future as the majority of detectors respond to inactive agent; calibrating their sensitivity against that of the referee system will require knowledge of what proportion of the agent they are being exposed to is active. 4.2.2 Aerosolization After the test material has been prepared, it must be aerosolized to form a “test cloud.” There are both wet and dry dissemination techniques, with their own advantages and disadvantages. For the purposes of implementing BAULADae, the important factors that need to be considered with respect to aerosolization are: first, the degree to which agent is inactivated during aerosolization, and, second, the particle size distribution produced by the aerosolizer. For a more detailed discussion of currently available dissemination techniques see Box 4.2. Often test materials will work only with certain types of dissemination systems. The information on dry dissemination of biological test material is quite limited. The effects of the aerosol generation technique on the viability of the test material are largely unknown, and aerosols are generated without specific attention to how particle size distribution relates to their behavior within the respiratory tract. A biological agent or simulant may not survive the aerosolization processes because of desiccation, shear forces, or impaction of primary droplets onto the baffle design to break up large drops. Implementation of BAULADae will require that these effects be evaluated.

OCR for page 47
51 Box 4.2 Dissemination Techniques Wet Dissemination Techniques Several methods can be used to convert bulk liquid—containing biological test material—to small droplets, a process also referred to as aerosolization, atomization, nebulization, or spraying. These methods include pneumatic, or air blast, nebulization, and ultrasonic atomization. • Air blast nebulizers(Mercer, Tillery, and Chow 1968) use compressed air to draw bulk liquid from a reservoir. The high-velocity air breaks up the liquid into droplets and then suspends the droplets as part of the aerosol. Droplets produced from this method have a volume median diameter (VMD) of 1-10 µm and a geometric standard deviation (σg) of 1.4-2.5.(Cheng and Chen 2001) Collison nebulizers (BGI Inc., Waltham, MA) and Hudson nebulizers (CAN Medical, Rockwall, TX) generate particles ranging in size from 0.1 to 5 µm, depending on the amounts of the test material in the aerosol suspension and operational conditions. Neither Collison nebulizers nor Hudson nebulizers are able to produce large droplets or large amount of aerosols. Agriculture sprayers (Micro Spray Ltd., Bromyar, UK) have been used successfully to produce biological simulants in open air field tests that require a high dissemination rate of droplets. The Micronair spray is vehicle mounted and uses a rotary atomization technology for droplet generation. • Ultrasonic nebulizers atomize liquid into droplets by the mechanical energy provided from a vibrating piezoelectric crystal driven by a variable-frequency electrical oscillator. The ultrasonic nebulizer does not require a pressurized air source. Some of the problems associated with pneumatic nebulizers, including concentration and cooling effects resulting from evaporation, are not issues in ultrasonic nebulization. It is possible to produce test aerosols of different particle sizes primarily by adjusting the concentration of the suspended material in the liquid but also by adjusting the nozzle size, driving frequency of the crystal, and the liquid feed rate. Dry Dissemination Techniques A dry dissemination generator usually consists of a mechanism to store, transport, and deliver the test material (a powder), as well as a mechanism to disperse the powder. Powder handling systems may use a gravity feed (hopper), screw feed, rotating disks, conveyer belt or chain, brushes, or compressed cylindrical packs in which powder is delivered by scraping off the top layer.(Moss and Cheng 1989; Cheng and Chen 2001; Hinds 1999) The powder delivery rate is usually adjustable and is a key factor influencing the rate of aerosol generation and, therefore, the aerosol concentration. Two major factors influence the stability of dry powder dissemination: the feed mechanism and flow property of the powder.(Moss and Cheng 1989) Resulting aerosol size distributions depend critically on the material properties of the powder itself as well as the generation method. Commercial and laboratory powder generators listed in Table 4.1 consist of a combination of delivery and dispersion mechanisms. These dispersion systems are usually used in the laboratory setting. For outdoor field tests, a knapsack agriculture spray, (Micro Spray Ltd., Bromyar, UK), which uses a vortex nozzle to disperse powder, has been used to disperse powder simulants.

OCR for page 47
52 TABLE 4.1 Dry Dissemination Methods Particle Size Generation Feeding (μm) Rate (mg/min) Product Mechanism Wright Dust Scraping the 1-100 0.1-1000 Feed packed plug Chain Fluidized Bed 1-100 0.5-10 conveyor Small Scale Powder Rotating disk 0.5-50 0.05-2.0 Disperser Jet-O-Mizer Screw feed 2-50 0.1-3 /Screw Feed RBG 1000 Rotating brush 0.1-100 0.5-9000 GRIMM 7840 <200 15-9000 Belt conveyer Rotating Vilnius Aerosol 1-50 __ /Vibrating Generator Turbine 4.2.3 Transport Other potential environmental stresses on the sample are the desiccation of the bioaerosol material in transport and exposure to sunlight (UV radiation). The change in viability of the test agent between the time it leaves the aerosolizer and when it reaches the referee instruments and detector under evaluation will have to be considered. 4.2.4 Referee Instruments The referee system measures and quantifies the amount of BWA aerosol in the aerosol cloud (indoors or outdoors) during a test trial. The referee system is the performance standard to which a device under test is being compared. In order to provide a reference in BAULADae units, it should provide representative and accurate information of the aerosol characteristics of the test material, including, at minimum, aerosol concentration of the active or viable agents, total aerosol concentration of viable and nonviable test material, and particle size distribution. This information is then analyzed to arrive at a standard evaluation of risk. Current referee systems have been designed to provide an accurate assessment of the ACPLA or CFU/PFU concentrations in test clouds. It should be noted that the instruments within the referee systems are not currently standardized across the T&E community. In order for these systems to provide accurate BAULADae concentration assessments, the effect of a number of steps in the evaluation process on the viability and size distribution of the particles will have to be taken into account.

OCR for page 47
53 The referee system may constitute a suite of instruments including: • sample-collection devices for quantification of biological aerosol concentration and viability, and • time-resolved measurement equipment for monitoring aerosol concentrations and particle sizes. As detailed in Box 4.3, there are several different sampling technologies, some of which give information in near real time, and others that require laboratory follow-up to assess the biological activity of the sample. There are three types of sampling methods. Filtration methods allow collection of different size fractions, can sample large volumes, and can attain 100 percent gas aspiration efficiency; however these require an extraction process that decreases aerosol collection efficiency and can affect sample viability. Impaction methods direct particles directly onto agar plates or other culture media substrates, and the resultant colonies can be counted. Samplers are available that can provide information about the quantity of colony-forming units per unit time, as well as quantity of colony-forming units per volume of air. Samplers that can measure the aerodynamic size distribution of culturable aerosol are also available. Liquid capture methods (or impingers) direct particles into liquid which can be used for PCR (polymerase chain reaction), ECL, or other immunological or nucleotide-sequence-based assays with minimal extraction. Impingers are very commonly used because they are easy to use, readily available, have good collection efficiency, and are relatively low cost. However, particle size information about the sample is lost. BOX 4.3 Referee System Sampling Techniques Isokinetic Sampling Diagram of an isokinetic filter sampler. Courtesy of Yung Sung Cheng. Filtration Filtration methods with adequate pore size will collect most particles larger than a minimum penetration size (typically between 0.1 to 1 µm in diameter), whereas the impaction and impingement methods will collect particles above a cutoff diameter. Filter samplers have been used to collect test aerosol on a filter substrate to determine representative concentration of the test aerosol in the test section. There are numerous types of filtration material, but the main collection mechanism is by impaction, diffusion, interception, and electrostatic collection. The filter sample provides a time-averaged aerosol concentration. For most analytical methods,

OCR for page 47
54 the filter must go through an extraction process to release the particulate, which will cause a decrease in the overall collection efficiency and an increase in interferents. In aerosol wind tunnel tests, an isokinetic filter sampler, which has 100 percent gas aspiration efficiency, is used to provide the measure of representative aerosol concentration.(Ambient Air Monitoring Reference and Equivalent Methods, Test Procedure: Full wind tunnel test. 40 CFR, 53.62, Federal Code of Regulations 1997) The isokinetic sampler shown above has an inlet sampling velocity matching the wind speed at the test section. The filter substrate, such as glass fiber filter, has near 100 percent particle removal. After the test, the filter substrate is removed for analysis to determine amounts of test material collected on the filter substrate. Usually, gravimetric analysis and analysis of fluorescent material are used to determine mass concentration (mass per volume of air) of nonbiological test material. Biological test material can be extracted into liquid medium and subsequently assayed for culturable counts and other information. Dry filtration allows for high volume sampling which can provide for a lower detection limit. Dry filtration is the most versatile and easiest to use in a field setting, but dry filtration causes desiccation of the biological organism and, in general, is not used for methods for assessing viability. Impaction Slit (or slit-to-agar) samplers (New Brunswick Scientific Co, Inc., Edison, NJ) use an agar plate as the collector. The aerosol is sampled through the inlet and impacts on the agar plate through a rectangular slit nozzle. The sampling flow rate is 15 to 52 L/min controlled by a mass flow controller. The collection plate is placed on a rotating disk, which rotates at a speed from 2.0 to 99.9 minutes in a cycle. The exposed plate can be incubated, allowing viable biological particles to grow into visible colonies that can be counted. The colonies are then counted. The plate is segmented into equal time sectors, providing the information of colony-forming units as a function of sampling time. The sample volume of slit samplers is more than impingers but less than dry filtration methods. Another sampler is the six-stage Andersen microbial impactor (Andersen Instruments, Symrna, GA). This impactor has hundreds of nozzles per stage and agar plates as collection surface. It is operated at a flow rate of 28.3 L/min. Particles are accelerated in the nozzle and those that cannot follow the flow streamline around a sharp turn make contact with the collection surface. Large particles impact on the first stage and smaller particles follow the flow until accelerated sufficiently to either impact at a later stage or pass through the system uncollected. The Andersen impactor provides bioaerosol concentration in terms of colony-forming units per unit volume air as well as aerodynamic size distribution of culturable aerosol. Liquid Impingers A liquid impinger may be used for biological test material that cannot be grown directly on agar plates. The liquid impinger has been widely used in test and evaluation programs to sample bacteria, spores, toxins, and viruses. A liquid impinger samples air through a nozzle and then accelerates and impinges the air jet into a collection liquid. Particles either impact directly into the liquid or are taken up by the liquid after impaction onto the bottom surface. The collected liquid sample can be used for PCR, ECL, or other established bio-assays with minimal extraction to determine the level of the specific biological agent of interest. Several liquid impingers have been used to collect biological aerosols. An all-glass impinger draws air at 12.5/min through a 1.1-mm capillary jet. The distance between the jet orifice and the bottom of the vessel is 4 mm or 30 mm. Particle bounce, re-aerosolization, as well as liquid evaporation, have been found to have significant effects on the collection efficiency.(Grinshpun et al. 1997) A BioSampler (SKC Inc., Eighty Four, PA) is used to draw air at 12.5 L/min through three 0.75 mm nozzles. The nozzles are directed at an angle to the inner sampler wall, inducing a swirling air motion. This design minimizes particle losses after collection and has been shown to provide better collection efficiency.(Oberdorster 2000) Both impingers use about 20 mL of collection fluid. Impingers are the most common method for quantifying viable BWA, because they are easy to use, readily available, have good collection efficiencies, and are relatively low cost. Moderate volumes can be collected, but in general they are an order of magnitude lower than the dry filtration. Impingers require monitoring of the liquid level, especially when collecting for long periods of time and in a dry climate as evaporation and re-aerosolization can become sources of error.

OCR for page 47
55 The assessment of biological activity that follows sampling may take hours or days; referee systems also include a time-resolved, continuous-operation component to monitor characteristics of the aerosol cloud. These near real-time measurements can be part of a feedback loop to control test aerosol concentration. Several technologies with different capabilities are available for temporal monitoring. Optical counters and aerodynamic particle-sizing instruments can measure particle concentration and size distribution (for detail on how these instruments work, their collection efficiency, and particle size detection limits, see Box 4.4). The fluorescence aerosol particle sensors use fluorescence to identify particles that may have biological origins. Some of these instruments also provide information about particle size distribution and the proportion of particles of possible biological origin in each size category. Fluorescence aerosol particle sensors can sample only a small fraction of the actual sample flow. BOX 4.4 Time-resolved, Continuous Measurement Systems Referee systems that collect samples provide information on biological aerosol concentration and size distribution after time-consuming laboratory assays. On the other hand, real-time measurements provide continuous monitoring of aerosol cloud and may be used to monitor aerosol in the chamber with a feedback loop to control the test aerosol concentration. It is especially useful in the laboratory chamber test to maintain a constant level of aerosol concentration or pre-set concentration profiles. Optical counters, aerodynamic particle sizers, and fluorescence aerosol particle sensors have been used in the instrument development community for rapid time-resolved measurement. An optical particle counter (manufacturers include Climet Instruments Company, Redlands CA; Particle Measurement Systems, Boulder, CO; Grimm Technologies, Inc., Douglasville, GA) is based on the principle of light scattering. Aerosol is sampled into the instrument, typically at flow rates on the order of 1L/min. Some optical particle counters designed for measurement of relatively large (tens to hundreds of microns) particles sample at much higher flow rates. Particles are aerodynamically focused into a focal volume where they are illuminated; the light that is scattered from individual particles into a defined solid angle is measured with a photodetector. The peak pulse intensity measured with the photodetector is used to estimate particle size. That estimation requires assumptions about the particle shape and refractive index, and for the same type of particle will differ from instrument to instrument, depending on the light source employed (typically a monochromatic laser or a white light source) and the viewing angle. Particle counts are accumulated into intensity bins. The particle size distribution is inferred from the distribution of counts in those bins. Aerodynamic particle-sizing instruments infer the aerodynamic diameters of individual aerosol particles by measuring their response to rapid changes in the velocity at which they are carried. Typically the air flow is accelerated through a nozzle to produce a high-velocity jet. Particles smaller than some threshold size will exit the nozzle at, or very close to, the gas velocity. Due to their inertia, larger particles will lag behind the gas and exit at lower velocities. The particle velocity, determined by measuring the time required for the particle to pass between two laser beams separated by a known distance, can be directly related to the aerodynamic diameter of the particle. Since aerodynamic diameter is sensitive to particle shape and orientation, sometimes correction factors may be applied to specific particle types. For example, liquid particle size measurements may be biased due to aerodynamic distortions of the droplet shape from a perfect sphere. The most common method for measurement of this kind is the aerodynamic particle sizer (APS) (TSI Inc., St. Paul, MN) that is based on a time-of-flight principle.(Baron, Mazumder, and Cheng 2001) The APS provides high resolution of size measurement between 0.7 and 20 µm. The fluorescence aerosol particle sensors use fluorescence to identify particles that may have biological origins. This method, which has its roots in flow cytometry, has been applied in a number of configurations. Different instruments use different excitation wavelengths and probe different fluorescent emissions. These fluorescence measurements provide information about the possible biological content or the presence of other materials with known fluorescence characteristics. Some instruments combine fluorescence with other measurements to provide additional information about the particles. The fluorescence aerodynamic particle sizer (FLAPS) (TSI Inc., St. Paul,

OCR for page 47
56 MN) integrates measurement of the fluorescence in the long and short wavelength bands that is induced by a violet (405 nm) laser diode.(Hairston, Ho, and Quant 1997) It is a single particle sensor that provides three real-time measurements of individual particles into the aerodynamic particle sizer. These measurements are the fluorescence emission intensity in two wavelengths and the scattered light intensity. The two fluorescence measurements are based on excitation illumination using a pulsed violet laser diode (405 nm) and fluorescence emission in the short and long visible wavelength bands. Due to the limited repetition rate of the laser, fluorescence is measured only for a fraction of the particles. The instrument thus reports the number distribution of particles with respect to aerodynamic diameter and the fraction of particles in the different size bins that fluoresce. The FLAPS samples aerosol at 1 L/min and detects and sizes particles in the 0.5-to-20 µm size range. 4.2.5 Validation and Calibration of the Referee System Of critical importance to the T&E system is ensuring that the referee system can provide an accurate measure of the actual conditions at the time of the test. Currently, no instruments are available that can measure directly in BAULADae units. However, as described above, a suite of instruments can be used to measure all the factors required to evaluate BAULADae. Validation and calibration of referee system performance will require taking into account earlier steps in the testing process that may have an impact on some of the qualities that use of BAULADae units require. Both the size distribution of the particles and the percentage of viable agents in the test cloud will depend on the techniques used to prepare the stock suspension, method of aerosol generation, and transport of the aerosol cloud during the test and evaluation. Considerable effort will have to be devoted to calibrating referee systems to ensure that they are measuring conditions accurately and that the measurements are comparable, especially if different test environments use different suites of instruments. For example, during the calibration phase, it will be necessary to verify such measures as: • the proportion of sample that is active depending on different sample preparation techniques; • the amount of sample that remains active following aerosolization; • the size distribution of the particles leaving the entering the air to be transported; • the amount of agent inactivated during transport from aerosol generator to the referee instruments; and • the response function of the different sampling instruments to active and inactive agent. These determinations will not have to be made for every test, but will have to be established for each set of instruments and for each biological agent. Careful attention to this calibration process will ensure that the BAULADae determination made by the referee system is accurate—a critical factor if it is to be used for evaluating detector performance, comparing the performance of different detectors, and comparing results from test to test and from site to site. Sample and analysis methods for BWA materials are less mature than methods for other particulate matter, and a standard method is not currently available. BWA sampling and analysis

OCR for page 47
57 instruments have generally been used to assess the presence of a biological threat and are considered only semiquantitative. They are also used to evaluate the performance of laboratory and fieldable instruments that detect or identify a biological aerosol. There are three basic types of sampling methods: filtration, impaction onto surface, and liquid capture (impingers). In order for referee systems to provide accurate assessments of BAULADae, it is likely that some combination of these sampling technologies will need to be used: filtration to assess the total quantity of agent present, and impaction or impingement to assess the quantity per unit air of viable or active agent. 4.3 LIMITATIONS OF THE CURRENT SYSTEM FOR IMPLEMENTING BAULADae The previous sections summarize the variety of methods and instruments that are currently available to assess biological aerosols. There are limited standard methods to generate and characterize biological test aerosol in the test and evaluation community. The lack of standard methods and procedures already results in difficulty of translation of test results from one test bed to another and in quality control of test data. If the BAULA standard is adopted, this problem will become more severe, since more characteristics of the aerosol will need to be accurately assessed. It will be important to establish and make available consistent test protocols in order to compare the performance of different kinds of instruments. Such standard protocols will also assist in the development and improvement of bioaerosol detectors. A means to obtain consistent and available protocols is to submit the protocols through a standard development organization such as ASTM or ISO. To implement BAULADae, future tests are likely to require test aerosols with several defined-size distributions to determine the detector performance as a function of particle size. Current test aerosols do not have specified particle size distributions. As discussed in the previous section, live agents can be dispersed into aerosols with a range of particle size, depending on the formulation and dissemination technique. Current methods to provide a measure of test aerosol have several disadvantages: 1. Filters, slit-to-agar samplers, and AGI impingers (Box 4.3) do not classify aerosol into particle size fractions and, therefore, do not provide concentration information as a function of particle size. This represents a major functional hurdle for the integration of size selective components into existing detectors. 2. Current methods provide measures of concentration of active biological material in terms of ACPLA, CFU/m3, PFU/m3, etc., but no information on inactive biological material. As discussed earlier, it is unlikely that test aerosol will consist only of active biological material. Most test aerosols have both active and inactive materials. Certain types of biological devices (e.g., LIDAR, fluorescence sensor, and PCR analyzer) will respond to both active and inactive biological material, but do not report the proportion of active material. 3. Referee instruments have not been well characterized in terms of their sampling efficiencies as a function of particle size, wind speed, and orientation with wind direction. Therefore, it is difficult to assess their measurement accuracy. A standard referee instrument should have near 100 percent gas sampling efficiency, as attained by the isokinetic filter samplers used for the EPA certification of particulate matter samplers or, at least, known sampling efficiencies as a function of particle size, wind speed, and wind direction. This

OCR for page 47
58 information is critical when particles of different sizes are included in bioaerosol sampler testing. The variations and limitations detailed above make direct comparison of testing results from different locations or years difficult, if not impossible. Effective use of BAULADae will require a greater degree of standardization of equipment and protocols across the testing community as the unit is inherently linked to the physical properties of the aerosol cloud and particles. The committee recommends the following: • The method for implementing the BAULA framework should be documented, externally peer reviewed, and published. Revisions and updates should follow similar vetting processes so that calibration, referee instruments, and testing reagents are standardized and variation is identified. • Aerosol challenges need to be well characterized including Dae, BAULADae., BAPLADae, and the loss of biological activity during aerosol dissemination/transport. • Biodetectors should be challenged with aerosols of defined size distributions. At least three size distributions should be used in chamber or component testing of detectors. These should be chosen to represent deposition in the three regions of the respiratory tract to reflect health risk measurements. 4.4 IMPLEMENTING BAULADAE WITH CURRENT TESTING AND REFEREE SYSTEMS A feature of the new unit is a focus on the activity, as predicted by viability or other biological activity of interest, of the aerosol particles, and total agent contained in the particles. Currently the principal detector characteristic that is evaluated is minimum detectable concentration. Detectors are tested against lower and lower concentrations of agent until they no longer respond. In addition to determining the detector’s minimum detection threshold, its response time, probability of detection, and probability of false alarm must be characterized or at least specified as part of the evaluation. One of the first objectives of a modified test procedure will be to determine the responsivity of the candidate system to one or more BW agents as reported in the appropriate unit, BAULADae. Evaluating detectors for the ability to determine whether a sample is viable or nonviable, as required for measurement in the BAULADae unit, presents an interesting challenge. In practice, it is unrealistic to expect that a pure agent material (consisting of 100 percent active agent and no other material) can be created and presented to the detection system. At least some of the agent material will be rendered nonviable and, therefore, inactive by the sample preparation, aerosolization, and transport processes. Current candidate detection systems detect both viable and nonviable agent material, but to determine a detector’s sensitivity in BAULADae units, evaluators must determine the detector’s limit of detection for biologically active units. In order to compare the performance of the detector to the ground truth of the referee system (which will be measuring active agent concentrations), it will be important to estimate contributions from viable agent and from previously viable agent material. Some instruments respond to material that may be present in the absence of viable as well as from previously viable target agent. For instance, optical-based detection based on aromatic amino acid (e.g., tryptophan) signatures will respond to cellular material from other species and not just from the target agent

OCR for page 47
59 or surrogate. These systems could indicate the presence of agent and trigger an alarm even when the referee system indicates an accurate BAULADae value of zero. Figure 4.3 gives a very simple example of how loss of activity during testing affects the assessment of detector sensitivity. More detailed examples, with suggestions on how testing in BAULADae could be implemented with current equipment despite these complications, are given in Appendix A. FIGURE 4.2 Testing requirement for minimum sensitivity. A simple example of the sensitivity bias that can be introduced if the response of a given detector to both active and inactive agent is not taken into account. The candidate detector will continue to signal presence of the agent even when the agent is inactive. The referee system will not signal the presence of agent as it is sensitive only to active agent. 4.5 ADDITIONAL CONSIDERATIONS Over time it is likely to be desirable to evaluate the performance of deployed detectors and their history of signaling, either correctly or incorrectly, the presence of agent. It will be important that the test and evaluation community have enough information to modify procedures and protocols appropriately. The committee recommends that DOD maintain the ability to learn from unanticipated events by archiving both T&E and deployed systems results (data). Test parameters, methods, measurements, and test conditions during testing and evaluation of detectors should be recorded and archived to compare performance of deployed with tested systems. In addition, DOD should consider archiving raw data collected by deployed systems to guide future development of detectors.

OCR for page 47