Engineering Aviation Security Environments—Reduction of False Alarms in Computed Tomography-Based Screening of Checked Baggage (2013)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

2 Overview of Deployed Explosive Detection System Technologies This chapter discusses computed tomography (CT) technology as it is applied to explosives detection (image reconstruction and the automated threat recognition [ATR] algorithm), ways in which the physics of CT limit its applicability for explosives detection (including image artifacts and material density), the processes of testing and evaluating these systems at the Transportation Security Laboratory (TSL), and the implementation of explosive detection systems (EDSs), including the screening process, in an airport setting. The differences between CT as used for explosives detection and medical CT are covered in Chapter 5 of this report. As pointed out in Chapter 1, the equipment described in this report does not detect the presence of explosives. Instead, as used by the Transportation Security Administration (TSA) and this committee, the term “explosive detection system” refers to a CT-based device for interrogating a bag, coupled with an ATR algorithm for evaluating the results of the interrogation. The purpose of the algorithm is to identify materials within checked bags that possess certain known properties of explosives, in order to determine whether or not an alarm should be given. All EDSs must be certified by the TSL, as discussed in the subsection below entitled “Testing at the Transportation Security Laboratory.” OVERVIEW OF A COMPUTED TOMOGRAPHY SCANNER A typical CT scanner (Figure 2-1) consists of a support frame and five key subsystems: (1) a high-voltage power supply (HVPS), (2) an x-ray tube, (3) a detector, (4) a gantry, and (5) a data acquisition system (DAS). Bags are fed into the scanner by means of a conveyer belt. The HVPS produces voltages necessary to power the x-ray tube. The potential of the HVPS generally falls between 140 and 180 kilovolts, with power in the range of 500 to 5,000 watts. Some systems use a direct current waveform. Other systems add an alternating current component as one means of collecting dual-energy information (see the subsection below entitled “Dual-Energy Scanning”). The x-ray tube produces a Bremsstrahlung spectrum of x-rays from 0 kiloelectronvolts to the peak potential of the HVPS. One or more rows of detectors, outwardly aligned in a cone shape from the x-ray tube, first convert the x-ray photons into light photons, and then convert the light photons to an electrical charge. The output of the detector is then digitized by the DAS into either fan- or cone-beam projections. These projections are related to the line-integrals of the x-ray attenuation coefficient of the bag along the paths from the x-ray tube to the detectors and are sampled at approximately 1 kilohertz so that projections are obtained at various angular positions around the bag. 17 

FIGURE 2-1 Photograph of the inside of a computed tomography scanner, showing (A) x-ray detectors, (B) gantry rotation, (C) x-ray beam, and (D) x-ray tube. The data acquisition system is not shown. SOURCE: Adapted from a Wikimedia Commons image available at http://en.wikipedia.org/wiki/File:Ct-internals.jpg. The x-ray tube, power supply, detectors, and DAS are mounted on a gantry. The rotation speed of the gantry ranges from 60 to 180 revolutions per minute. 1 As the gantry rotates around the bag, the conveyor belt on which the bag rests may be stationary or moving. If the conveyer is stationary during scanning, the scanner is considered to be a “step-and-shoot” variety. Conversely, if the conveyer is moving, the scanner is helical or spiral. Image Reconstruction and Correction Following the scan, the outputs of the DAS are sent to a reconstruction computer (see Figure 2-2) to be converted into cross-sectional images, which are then sent to another computer on which the automated threat-recognition algorithm is performed. Most scanners use a process called filtered back-projection (FBP) to reconstruct the cross- sectional images; 2 the algorithms used in image reconstruction were developed for medical imaging and have not been optimized for use in security applications, which is one potential source of error that can lead to false alarms. 3 In the reconstruction process, the output of the DAS is corrected to account for imperfections in the machine hardware and to generate the line-integral data. Additional steps are used to compensate for the cone shape of the x-ray beam and for helical scanning (where the scanner, itself, moves). These steps are approximate, and artifacts are created in images due to these approximations. The steps in the reconstruction process are shown in Figure 2-2. 1 Or 0.33-1.0 second per rotation. In general, newer models are faster. See, for example, Ge Wang and Hengyong Yu, An outlook on x-ray CT research and development, Medical Physics 35(3):1051-1064, 2008. 2 A.C. Kak and M. Slaney, Principles of Computerized Tomographic Imaging, IEEE Press, New York, N.Y., 1988. 3 See, for example, Lawrence Livermore National Laboratory, Improved Aviation Security via Technology Advancements, available at http://www-eng.llnl.gov/ndc_aviation.html, accessed April 7, 2011, which describes the shortcomings of these medical CT-based reconstruction algorithms and the work being done at Lawrence Livermore National Laboratory to improve them. 18 

FIGURE 2-2 Steps in the computed tomography image reconstruction process. TABLE 2-1 Operations Performed During the Correction Steps in Image Reconstruction Step Synopsis Offset The electronics (photodiode and amplifiers in the data acquisition system [DAS]) have dark currents. The dark currents are measured with the x-ray tube turned off and then subtracted. Temperature drift of the offset has to be considered. Reference The current supplied by the high-voltage power supply to the x-ray tube may vary. A reference detector measures the incident x-ray flux. Beam The x-ray tube produces a polychromatic spectrum. The x-ray attenuation coefficient is a hardening function of the photon energy, with lower-energy photons being preferentially removed. A polynomial correction is applied. Unfortunately the different materials require the use of different polynomials, and so artifacts will remain. Spectral Each detector has its own spectral response to polychromatic x-rays. This response is known as response the detector’s transfer function. The difference of the transfer function for each detector with respect to the mean of the functions for all the detectors is corrected in order to prevent the insertion of concentric rings and bands in images. One method of performing this correction is to apply a polynomial correction, which is specific to each detector and whose coefficients are determined during a calibration step. Afterglow The detector and DAS have finite impulse responses leading to a temporal blur of the projections. The impulse responses may be de-convolved. Scatter Scattered x-ray photons may reach the detector. Some scattered photons may be eliminated with removal antiscatter plates placed in the septa between detectors. Additional algorithmic correction can be used to remove scatter based on measurements from auxiliary detectors or using the projections themselves. Clamping The DAS has a finite dynamic range, which is determined in part by the electronic noise in the DAS. The number of x-ray photons reaching the detector may be on the level of the electronic noise. The number of photons is clamped to a positive number. However, artifacts will still be generated in images when this condition occurs. Gain Each detector has its own gain. The gain is measured by scanning only air. The values of the air measurement readings are used to scale the readings through a bag. The gains may be a function of the angular position of the gantry. Logarithm The DAS/detector combination integrates energy. In order to generate the line integrals required by filtered back-projection, the natural logarithm of the reading is taken. Note that this step is based on the physics of detecting x-rays and is not a correction step. However, it is included in the correction steps of reconstruction because of implementation considerations. Re-binning The cone-beam projections are processed to form fan-beam or parallel-beam projections. If the projections were acquired using helical scanning, then the movement of the bag during data acquisition is removed using interpolation. 19 

Imprecision in image reconstruction is one source of error that can lead to false alarms (see the subsection below on “Image Artifacts”). If the steps in correction cannot completely account for the underlying physical effects of scanning, images will be degraded. Another source of artifacts is due to the approximations made in the reconstruction algorithm for compensating for cone-beam divergence of the x-ray beam and for helical scanning. Any artifacts can lead to inaccurate measurements of an object’s linear attenuation coefficient, density, and atomic number. Table 2-1 provides an overview of the operations that are performed during correction. Noise in the image may also create problems in the automated image segmentation process, and make it difficult to distinguish between explosives and other material. While filters, image smoothing, and other enhancement techniques have been developed for medical CT applications, these measures can be difficult to extend to baggage, because of the large number of objects present in baggage and because—unlike tumors—a single object in a baggage scan may present with wide number of grey levels in the image. 4 Automated Threat Recognition The automated threat-recognition process (outlined in Figure 2-3) segments the cross-sectional images into individual objects and then classifies each object as either a threat or a non-threat. Specifics of each vendor’s ATR algorithm are proprietary; the process described here is general. As part of its analysis, the ATR algorithm may also compensate for imperfect correction in the CT reconstruction step and extracts features such as density, atomic number, and feature size. Once this information has been extracted, it is compared to the density and properties of known explosives. If the information derived from an object falls within the specified range and the object’s mass is above a TSA-specified value for that range, then the object is declared a potential threat. FIGURE 2-3 Simplified, representational outline of the automated threat-recognition process. Note that each explosive detection system vendor’s process is different. 4 Sameer Singh and Maneesha Singh, Explosives detection systems for aviation security: A review, Signal Processing 83(1):31-55, 2003. 20 

Alarm No Alarm Present True Detection Missed Detection Threat Present False Positive True Negative Threat No FIGURE 2-4 A contingency table of the potential results of an interrogation of checked baggage by a computed tomography-based explosive detection system. Some vendors may also use the filtered back-projection data and the images from the digital radiography line scanner in their ATR algorithm. They may also have separate methods (or “paths”) for the identification of potential sheet explosives and the identification of potential bulk explosives. An alarm may be either a “true positive detection” (meaning that the ATR algorithm signals an alarm and a threat is present in the scanned bag) or a “false alarm” (sometimes called a “false positive,” which means that the ATR algorithm signals an alarm but no threat is present). Bags that do not cause the EDS to alarm may be “true negative” (the ATR algorithm does not signal an alarm, and there is no threat present) or a “missed detection” (meaning that the ATR algorithm failed to report the presence of a threat). These possibilities are detailed in Figure 2-4. The ATR algorithms have been developed and refined over many years to alert on threat amounts of materials that fall within a specified density and mass range (“detection window”). However, owing to the nature and composition of many non-threat objects, the criteria cannot be made specific enough to include only threat materials, and innocuous materials may fall within or near the detection windows and may be mistaken for threat materials. Consequently, there will always a trade-off between false alarms and missed detections. When applied only to the information available from the CT scan, no algorithm will always identify a threat item while never misidentifying an innocuous item as a threat. Narrowing the detection windows in order to eliminate the misidentification of non-threat materials carries with it the risk of decreasing the detection rate and missing a true threat. Expanding the detection window to ensure the capture of all threat materials will result in capturing non-threat materials and increasing the false alarm rate. A plot of threat identification (probability of detection, or PD) versus misidentification of an innocuous item as a threat (probability of false alarm, or PFA) is known as the receiver operating characteristic (ROC) curve, as shown in Figure 2-5. A method of reducing the false alarm rate is to increase the area under the ROC curve; this can be done for the EDS by adjusting the signal-to-noise ratio (either increasing the strength of the signal or reducing the amount of noise), but the maximum area under the curve is fundamentally limited by the technology. 5 5 David Heeger, “Signal Detection Theory.” New York University, 1997, available at http://www.cns.nyu.edu/ ~david/handouts/sdt-advanced.pdf, accessed June 14, 2011. 21 

FIGURE 2-5 An example of a receiver operating characteristic curve. The Transportation Security Administration has chosen to maintain a high level of detection. The agency thus has been forced to accept a concomitantly high level of false alarms. Vendors who spoke at the committee’s meetings (see the section entitled “Study Process” in Chapter 1) indicated that more recent hardware and software updates have the potential to deliver a lower false alarm rate while maintaining probability of detection, but they indicated that in many cases these technical improvements are not followed up on by the TSA or the airports. 6 For example, the committee was told that General Electric (GE) Security’s CTX 9400 model 7 demonstrated a 45 percent reduction in “shield alarms” (that is, alarms that occur when the CT machine cannot penetrate an area of a bag) and a 10 percent reduction in other false alarms while concurrently demonstrating better detection and a slightly lower throughput (fewer bags per hour). 8 However, the cost to upgrade each of these machines is more than $100,000, which has—up to this point—been regarded as prohibitively costly for the TSA or airports. L-3 Communications also indicated that it had developed TSL-certified software to reduce false alarms, but that, as of the committee’s meeting in 2009, this software had not been purchased. FUNDAMENTAL LIMITATIONS OF COMPUTED TOMOGRAPHY-BASED EXPLOSIVE DETECTION SYSTEMS BASED ON THE PHYSICS OF THE TECHNOLOGY As stated above, CT-based explosive detection systems are not systems that detect explosives, but rather systems that can identify materials that have specific properties. In this section the committee expands on the limitations of the use of this form CT for the purposes of detecting explosives. 6 David Heeger, “Signal Detection Theory.” New York University, 1997, available at http://www.cns.nyu.edu/ ~david/handouts/sdt-advanced.pdf, accessed June 14, 2011. 7 Now produced by Morpho Detection, Incorporated. 8 Matthew Merzbacher, General Electric, “Overview of Detection Algorithms,” presentation to the committee. February 12, 2009, San Francisco, Calif. 22 

Image Artifacts The correction step in CT image reconstruction attempts to compensate for imperfections in the projection data acquired during scanning, but these corrections are not perfect, and image artifacts will always remain. These artifacts produce signals that can lead to uncertainty in the measurement and evaluation of the objects that are being scanned. This uncertainty manifests itself in the threat-detection process in various ways—for example, in mis-estimation of object mass, a widening of density and atomic number windows, and inaccurate region building (which leads to over-aggregation of different objects). This issue, which is also relevant to medical CT, is discussed in Chapter 4. Image artifacts contribute to false alarms mainly by causing uncertainty in the shapes and sizes of the individual objects within a bag. Although metal—which leads to streak artifacts in CT images—is one of the primary sources of image artifacts, there are other contributing factors, such as photon starvation (common in large, densely packed bags or bags that include one or more heavy metal objects), beam hardening (generally caused by long, very straight objects that run across the width of a bag), partial volume (caused by very thin objects), bag motion artifacts, and approximations in the reconstruction algorithm to compensate for cone-beam divergence and helical scanning. The larger and more cluttered a bag is, the more likely there are to be errors in image reconstruction. Image artifacts caused by imperfections in both the CT hardware and the software reconstruction lower confidence in the estimated characteristics of an object within a bag, forcing the threat-defining windows to be widened, which results in a concurrent increase in false alarms. Thus, improvements in the image reconstruction and correction process would enable the more accurate measurement of objects and could lead to a lower false alarm rate. Baggage Contents Variations in the attitudes and practices of the flying public can also lead to variations in the false alarm rate over time. EDS vendors and screeners at San Francisco International Airport told the committee that different airline policies and TSA policies have had an impact on the way that passengers pack their bags and, consequently, on the false alarm rate. For example, when more airlines began to charge for checked baggage customers responded by packing their suitcases more densely, which, as noted above, can also lead to a higher false alarm rate. Seasonal changes can also precipitate changes in passengers’ packing habits and can make it difficult to know where to set the parameters for the ATR algorithm. For example, passengers traveling in the summer or to a tropical destination and those traveling in the winter or to a snowy destination are likely to pack differently. Material Density A non-threat material that falls within the same density and atomic number range as that of a given threat material will result in a false alarm. The difficulty in isolating threat materials from non- threat materials can be seen in Figure 2-6, which shows typical density ranges for some threat and non- threat materials. This figure shows that while clothes are clearly distinguishable as a single-valued function of density, other non-threat materials commonly found in passenger bags overlap in material density with some threat materials. It is inevitable then that relying solely on density to indentify threat material will lead to some misidentification and false alarms. 23 

FIGURE 2-6 Notional distribution of threats and non-threats in computed tomography (CT) density space. Clothes are clearly distinguishable as single-valued function of density, but other non-threat materials commonly found in passenger bags show some overlap in material density with some material density with some threat materials. Setting the limits for declaring a material a potential threat is a difficult balance of science and policy. For example, a lower density limit of 1,100 kilograms per cubic meter would lead to missing the detection of some commercial explosives, whereas many innocuous materials would still be identified as potential threats. Lowering the limit to about 1,000 kilograms per cubic meter would allow capturing more commercial explosives, but it would also result in misidentifying a much larger number of innocuous materials. Adding the atomic number may reduce the overlap in two-dimensional space by providing conditional data to distinguish between threat and non-threat materials, as discussed in the next subsection. Dual-Energy Scanning Because errors in indentifying non-threat materials as potential threat materials occur when the non-threat materials have a density similar to that of threat materials, adding atomic number to the screening criteria could improve the ATR algorithm’s ability to distinguish between threat and non-threat materials and lower the probability that the EDS would give a false alarm. Although it seems reasonable that adding an extra dimension to the measurement would improve false alarm rates, anecdotal information presented to the committee indicates that the false alarm rates for dual-energy CT machines in an airport setting are not appreciably different from the false alarm rates for single-energy machines. Nevertheless, the committee does believe that the technology deserves further exploration so that there can be a full understanding of its advantages and limitations. Reveal Imaging has conducted a DHS sponsored study to gain a better understanding of limits of its CT-80 machine and false alarm images from two different airports. 9 This is a very important first step to allowing researchers and the TSA to evaluate more effectively the potential improvements and results of their efforts. TESTING AT THE TRANSPORTATION SECURITY LABORATORY Certification testing of EDSs and their subsequent performance testing in an airport setting are one way to gain a better understanding of EDS performance and of the causes of false alarms. To be 9 Elan Scheinman, Reveal Imaging Technologies, Inc., presentation to the committee, April 29, 2009, Washington, D.C. 24 

certified, a machine must demonstrate the ability to detect a number of categories of explosives, with each category having a specific detection threshold (i.e., level of detection that must be met.) The machine must also meet an average detection threshold across all categories of explosives and not exceed a maximum false alarm rate (which is tested separately from the detection). This certification testing is performed at the Transportation Security Laboratory (TSL), located at the William J. Hughes Technical Center at the Atlantic City International Airport in New Jersey. Originally, established in 1992 as part of the U.S. Department of Transportation, the laboratory is now under the umbrella of the U.S. Department of Homeland Security Science and Technology Directorate. There is a limitation in being able to predict the performance of machines in an airport setting; however, the use of bags more representative of those seen in an airport setting would be enormously complicated, given the variations in bags and contents. Additionally, the use of a specific set of test bags allows all manufactures to be tested against a common standard. Other tests besides that for certification are performed at the TSL. Certification readiness testing and pre-certification tests qualify a system to enter the path to certification. Post-certification tests are performed to ascertain whether certain configurations or locations of explosives go undetected. Conclusion: Certification testing at the Transportation Security Laboratory fills a specific and useful role. However, it should not be used as the sole basis for predictions of performance in an airport setting. IMPLEMENTATION WITHIN AN AIRPORT SETTING Shifting Emphasis In the airport setting a shift in emphasis occurs—from a focus on detection to a focus on reducing the costs of screening by minimizing the number of secondary inspections and the number of manual bag inspections. Because on-screen alarm resolution is the link between the automated alarm of the EDS and manual bag inspection, there can be pressure to clear bags in order to make delivery deadlines. 10 Additionally, because the bags and objects scanned in the airport are more varied than those in the TSL- certified test set, the probability of detection established for an EDS at the TSL may not be maintained an airport setting. Beyond these limitations, additional shortcomings in the effectiveness of an EDS system may develop over time, including, for example, the lack of a feedback system by which false alarms are analyzed and fed back into the ATR software development (discussed in greater detail in Chapter 6). Recommendation: The TSA should develop procedures for periodic verification to ensure that fielded EDSs meet detection-performance-level standards that correspond to the requirements for EDS certification. In addition to monitoring detection capability directly (e.g., using standard bag sets and red- team testing), these procedures should include the frequent monitoring of critical system parameters (e.g., voltages and currents) and imaging parameters (e.g., image resolution and image noise) to detect system problems as soon as they arise. For purposes of monitoring EDS performance, the TSA and EDS vendors should develop specification limits for all critical system parameters (and their tolerances) that could be monitored frequently and recorded to track changes in performance during normal operations or to verify performance after maintenance or upgrading. 10 See, for example, Sara Kraemer, Pascale Carayon, and Thomas F. Sanquist, Human and organizational factors in security screening and inspection systems: Conceptual framework and key research needs, Cognition, Technology, and Work 11:29-41, 2009. 25 

Screening Process Airports can incorporate explosive detection systems into their screening processes in a variety of ways, depending on such factors as the space available, flight schedules, and typical flight destinations. Here the committee describes a “typical” screening scenario as an aid to understanding the overall checked-baggage screening system, including alarm resolution by human screeners. This scenario is not meant to represent a preferred approach to screening or to limit the variation in checked-baggage screening equipment in airports. EDSs are deployed in two basic configurations: (1) in-line: the bags are fed into an EDS by the baggage-handling system, 11 and (2) stand-alone: the bags are fed into an EDS manually. The stand-alone EDSs are usually in the check-in lobby or behind the check-in counter. Figure 2-7 represents the CT-based EDS screening process. It should be noted that most of the expenses associated with clearing false alarms occur at the baggage-viewing station and in the baggage- inspection room (BIR). Clearing alarms in both of these areas requires human intervention in the process. When the ATR algorithm determines that an object or objects within a scanned bag meet the established threat criteria, a human screener must resolve the alarm. Information is presented on a display to a human screener at a baggage-viewing station. The information includes cross-sectional images of the bag and specifies of any suspicious objects generated by the ATR algorithm. The screener uses this information to decide either (1) that the alarm was caused by a non-threat item (whereupon the bag can be cleared to go on the airplane) or (2) that the screener is unable to determine—based on the on-screen alarm-resolution protocol (OSARP)—that the object indentified by the machine is not a threat, in which case the bag is sent to the baggage inspection room or other local area where it is opened and examined manually. As part of its analysis, the ATR algorithm evaluates whether there are any areas of the bag that the x-rays cannot penetrate. If there are any such areas, the system signals a shield alarm. Because these “shielded” areas could conceal potential threats, any bag with a shield alarm is sent directly to the baggage-inspection room for further screening, bypassing the option of on-screen resolution. In addition to shield alarms, “exceptions” that result in a bag’s being sent directly for further screening include “mis- tracking” (the baggage-handling system loses track of the bag), operator time-out errors ( the operator fails to clear the bag within a time limit), jamming of the bag in scanner, and scanner failures. In discussions with the committee, screeners indicated that these exceptions are counted as part of the overall false alarm rate, but specific data related to the percentage of the overall rate that they represent were not available. The on-screen alarm-resolution protocol serves as the link between the ATR algorithm and the baggage-inspection room. Adjusting the operating point on the ROC curve of the ATR algorithm and introducing variability in the EDS’s performance may improve the overall performance of the system, which includes the decision by both the EDS and the screener. This variability could be driven by intelligence-adjusting the algorithm to be more sensitive toward specific types of explosives while not searching for those that are less likely to be used. Or it could be driven by the introduction of other data to create a passenger- specific, risk-based screening approach. Such variability might also provide might also provide a deterrence value, as it would make it more difficult for any adversary of the system to predict the EDS’s capabilities. It will remain necessary to include the screener-in-the-loop when making any modifications to the overall screening system’s operations. Any changes may also require changes to the on-screen alarm- resolution protocol to ensure that the link between the ATR algorithm and the baggage-inspection room functions effectively. At risk assessment approach, such as the quantitative risk assessment process 11 The baggage-handling system consists of a set of conveyor belts, diverting mechanisms, and a tracking system. The conveyor belt moves bags in and out of the EDS, to the baggage-inspection room or other local area, and to the airplane. Diverting mechanisms transfer the bags between the different sections of the conveyor belts. 26 

FIGURE 2-7 Diagram of an in-line explosive detection system (EDS) consisting of (A) the computed tomography (CT) scanner, (B) the automated threat recognition (ATR) algorithm, (C) the baggage-viewing station and the on- screen alarm-resolution protocol (OSARP), and (D) the control computer. The EDS is integrated with (E) the baggage handling system, (F) the baggage-inspection room and/ or area, and (G) the ordinance disposal team. Shaded boxes are components of the EDS, white boxes are subsystems used in conjunction with the EDS, solid connecting lines show the flow of bags and/ or images of the bags, and dashed connecting lines show the flow of the control and information. described in Chapter 6 and Appendix B in this report, may be one way to evaluate these changes. Because the probability of false alarm can be measured in an airport setting, and probability of detection is rarely measured except in Transportation Security Laboratory testing, it is difficult to determine the simultaneous PD/PFA performance of EDSs in an airport setting. The committee believes that it is likely that the PD measured at the TSL is not maintained in an airport setting owing to a combination of the use of non-representative bags to measure PD at the TSL and the possibility that screeners may clear too many bags on screen that should be inspected by hand due to the low occurrence of true positives. Paradoxically, forcing EDSs to operate at their highest PD, and simultaneously their highest PFA, creates a situation in which screeners expect that every alarm is a false alarm and in which bags are cleared that should be sent for additional screening, lowering the detection capability of the entire system. It is counterintuitive that lowering the probability of detection of the EDS could lead to increased probability of detection of the overall system, but the committee believes that the TSA should consider evaluating this possibility. When a bag is sent to the baggage-inspection room, screeners may open it to visually inspect the objects indentified as potential threats and—depending on the object indentified—may also employ explosive trace detection to attempt to clear the bag. If the airport’s integration of the baggage-inspection room with the rest of the baggage-screening system is robust enough to permit it, this inspection may be guided and informed by other data related to the bags being inspected including CT slices and the outputs of the ATR algorithm-although it is possible that the threat indentified by the ATR algorithm will not be found by the transportation security officer (TSO) screening the bag or that another item will be mistaken for the threat. If the screener is able to clear all potential threats in the bag, it is sent to the airplane. Bags that cannot be cleared are handled according to local regulations for potential explosive threats. Finding: The low prevalence of true positives may make it nearly impossible to measure probability of detection with humans-in-the-loop without forcing true positives via red-team testing. Recommendation: The Transportation Security Administration, through the Transportation Security Laboratory, should support human-factor studies to assess the impact on overall system performance, that is, the EDS plus the screener resolution, when the operating point on the explosive 27 

detection system’s receiver operating characteristic curve is adjusted so that both the probability of detection and probability of false alarm are lowered. If the results of such studies determine that screener attention is degraded by the expectation that every alarm is a false alarm, the TSA should consider implementing adjustments to the operating point on the receiver operating characteristic curve and allowing vendors to reduce probability of detection in an airport setting to the minimum rate required for certification. DISCUSSION, WITH RELATED FINDING AND RECOMMENDATION To reduce the costs associated with screening baggage in airports, it will be necessary to lower both the number of automated alarms from EDSs and the number of bags opened manually and the number of bags that have to be traced. The automated threat-recognition algorithm will be able to make more correct decisions when it is provided with more accurate information about the contents of bags, including both the materials properties and object sizes. Improving object segmentation so that adjacent but unrelated objects in the bag are correctly separated in the image could also reduce false alarm rates by allowing for a more accurate estimation of an object’s density and mass. Hardware improvements aimed at more accurate estimates of materials properties could include dual- or multi-energy approaches or other methods to differentiate materials within a bag. Modifying an EDS’s operation to emphasize image quality over operational requirements (such as throughput) by such means as slowing the scan speed, improving reconstruction algorithms, or changing parameter settings within ATR algorithms based on threat level could also achieve the same aim—that is, it could improve the estimate of materials properties or object segmentation. Augmenting the CT scanner data with additional screening data could provide additional means of distinguishing between threat and non-threat materials. 12 Results of inspections in the baggage inspection room would be useful in identifying the causes of false alarms, but in the committee’s tour of the BIR at San Francisco International airport the committee saw no mechanism for collecting data on the results of such inspections. The committee believes that San Francisco International Airport is representative of airports through the Unites States in this respect. However, as with the data on exceptions, there is no requirement of either the TSA or the EDS vendors to mine or analyze collected data. This topic is discussed more fully in Chapter 3. Finding: Based on the information available at this time about the performance characteristics of these approaches and available data on the actual sources of false alarms raised by today’s explosives detection systems, it is not possible to establish which are most promising or merit significant investment. Recommendation: The TSA should not fund an overall replacement of fielded explosive detection systems, because replacing all the units in service with currently available technology would not allow for learning in an airport setting to inform future performance improvements. Instead, the TSA should plan its capital spending for explosives detection improvements over a period of time sufficient to allow several generations of technology to be fielded on a limited basis, revaluated, and iteratively improved—thus leading to a gradual improvement in the overall field performance of CT-based explosives detection systems. 12 A full discussion of this process can be found in National Research Council, Fusion of Security System Data to Improve Airport Security, The National Academies Press, Washington, D.C., 2007. 28