Engineering Aviation Security Environments—Reduction of False Alarms in Computed Tomography-Based Screening of Checked Baggage (2013)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

1 Introduction BACKGROUND AND REQUEST FOR STUDY With the enactment of the Aviation and Transportation Security Act (Public Law 170-71) on November 19, 2001, the Transportation Security Administration (TSA) was created as a separate entity within the U.S. Department of Transportation. 1 The act also mandated that as of December 31, 2002, all checked baggage on U.S. flights be scanned by explosive detection systems (EDSs) for the presence of potential explosives threats. As a result of the need to deploy EDS equipment quickly and universally, the procurement and installation of certified systems were emphasized by the TSA, as was the development of alternate equipment and procedures to provide screening where certified equipment was not yet available. Although any TSA-certified method of detecting explosives will meet the requirements of the Aviation and Transportation Security Act, in most airports this is provided by computed tomography (CT)-based systems. Although the system is called an explosive detection system, CT does not have the ability to detect explosives. Computed tomography was originally developed as a medical diagnostic tool that uses x-rays transmitted through the scanned area to provide a three-dimensional image of the human body. For aviation security applications, the transmitted x-rays are reconstructed into a three-dimensional image of the scanned bag, and software algorithms (see the subsection entitled “Automated Threat Recognition” in Chapter 2 of this report) are applied to that image to estimate the material properties of the items within the bag and to compare those estimated properties to a set of criteria defined by the TSA for explosive threat items. These criteria have been developed over many years to detect specified materials at levels that would pose a threat, but they are not perfect. No algorithm can always find a threat item and never misidentify an innocuous item as a threat. Baggage screening is always a trade-off between false alarms and missed detections (also called false negatives). The TSA has put a high priority on maintaining a high level of detection, and as a result it must deal with a concomitantly high level of false alarms. To gain a better understanding of and to be better able to address the issue of false alarms in EDSs, the Department of Homeland Security Science and Technology Directorate (DHS S&T) through the TSA requested a study from the National Research Council (NRC) of the National Academies. As a result of this request, the NRC established the Committee on Engineering Aviation Security Environments—False Positives from Explosives Detection Systems. The committee’s full statement of task is as follows: An ad hoc committee will examine the technology of current aviation-security explosive-detection systems (EDSs) and the false positives produced by this equipment. In assessing methods to reduce the EDS false-alarm rate, the committee will: 1. Examine and evaluate the causes of false positives in aviation explosive-detection systems, including considering the role of equipment design standards that rely on the fusion of explosive density measurement, total mass, and shape effects. 2. Assess the impact false positive resolution has on personnel and resource allocation. 1 The TSA was later moved to the Department of Homeland Security after its creation on November 25, 2002. 12 

3. Make recommendations on mitigating false positives without increasing false negatives, considering both technology and personnel approaches and related short- and long-term research. The committee recommendations will also bear in mind any risk of increased missed detection. Although the TSA recognized that false alarm rates for CT-based EDSs in an airport setting could be substantially higher than the false alarm rates measured in certification testing (see the section entitled “Testing at the Transportation Security Laboratory” in Chapter 2), the agency and policy makers believed that it was nevertheless important to field the equipment and provide screening of all checked baggage. Now that CT-Based EDSs have been employed for more than 10 years, the DHS S&T is focusing on better detection algorithms and more reliable equipment in order to reduce the number of false alarms and thereby reduce the costs of screening checked baggage. The Transportation Security Administration estimates that each percentage point of the current false alarm rate costs the government millions of dollars per year. The main element of these costs for resolving false alarms is personnel time, because every bag that causes an alarm must be further inspected. However, there are other elements that contribute to the cost of resolving these alarms— including the infrastructure for segregating bags for manual inspection, controlled areas for opening bags, and tracking bags to the baggage-inspection room (BIR) and back to their designated flight. In addition to the added expensed that it generates, the process of resolving false alarms may increase the risk to the air transport infrastructure because the time and personnel allocated to resolving false alarms may take away from security efforts that could detect or deter other threats. Moreover, some studies suggest that the current high false alarm rate may in fact reduce the likelihood of identifying an actual threat, as screeners have come to expect that the cause of an alarm is a non-threat. 2 Adding to the risks described above associated with personnel being diverted from detecting other threats and screeners expecting non-threats is the fact that as currently deployed, CT-based EDSs are tuned only to detect certain explosives. Improving the algorithm and image-processing systems and reducing the false alarm rate to an acceptable level without increasing the rate of false negatives would clear the way for the work necessary to detect additional explosive materials. 3 In discussions with the committee, the Transportation Security Administration and the sponsor at DHS S&T indicated that its primary interest was in the contributions to the false alarm rate made by the EDS—that is, the wrong decision by the machine to send a bag for further screening by the human screener and the lack of sufficient or appropriate information needed to resolve the alarm correctly using the TSA-established on-screen alarm-resolution protocol (OSARP). The TSA also indicated a desire to leverage what has been learned in the field of medical CT imaging and interpretation and to understand how this knowledge might apply to CT for explosives detection. In addition to the areas of focus described by the sponsor, the committee considered factors that may lead screeners to make a correct decision more often. The specific areas of focus addressed in the committee’s review are identified in the section entitled “Screening Process” in Chapter 2. Throughout the report the committee considers the trade-offs between false alarms and missed detections, but processing time, or throughout, is a third performance dimension. Indeed, with unlimited time to screen a bag, the estimations of mass and density would be near-perfect. The committee made the assumption that the amount of time currently required for bag screening is acceptable and focused on how to lower false alarms and maintain detection without explicitly paying a penalty in terms of much longer screening times. In implementing any changes intended to reduce false alarms, the TSA will have to determine if any increase in bag-screening time can be justified by a decrease in the number of false alarms. 2 See, for example, Mathias S. Fleck and Stephen R. Mitroff, Rare targets are rarely missed in correctable search, Psychological Science 18:943-947, 2007; and Anina N. Rich, Melina A. Kunar, Michael J. Van Wert, Barbara Hidalgo-Sotelo, Todd S. Horowiths, and Jeremy M. Wolfe, Why do we miss rare targets? Exploring the boundaries of the low prevalence effect, Journal of Vision 8:1-17, 2008. 3 Studies addressing the detection of novel and liquid explosives are discussed in the subsection entitled “Dual- Energy Scanning” in Chapter 2 of this report. 13 

DISTINCTIONS IN TERMS AND THE NEED FOR DATA Through discussions with the sponsor and site visits (see the section below entitled “Committee Meetings”), the committee learned that there are many mechanisms in the existing TSA system for recording data from EDSs—including the data on false alarms and their causes—but these data are not collected or analyzed in any comprehensive way. For this reason, although the causes of false alarms can be broadly identified, the committee is unable to assess the impact of any particular cause on either the overall false alarm rate or the system. 4 In particular, this lack of data also limits the committee’s ability to assess what impact the false alarm rate has on personnel allocation or costs. During the course of this study, the committee determined that there is inconsistent usage of the terms “false alarm rate” and “probability of false alarm.” “False alarm rate” is a function of the number of bags scanned and the likelihood of non-threat items that can be confused with threat items. “Probability of false alarm” is a function of the specific contents of a scanned bag and the technology and decision algorithm of the EDS. The committee has tried to be consistent in using “false alarm rate” when comparing the number of false alarms to the total number of bags scanned, and “probability of false alarm” when discussing the chance that an individual bag will cause the EDS to alarm. Within the EDS community, the terms tend to be used interchangeably, but maintaining a distinction in the technical meanings of these terms is important in order to help the TSA take advantage of the large body of literature dealing with classification, detection, and statistical decision making. The overall false alarm rate includes two distinct “populations” of bags, each of which would require a different approach in order to reduce false alarm rates: • The first population includes bags for which the EDS cannot make a decision—so-called “exceptions,” such as bags containing solid objects that cannot be penetrated by the EDS x-rays, mis- tracked bags, and bags that are poorly positioned in the EDS in such a way that the EDS cannot interrogate the entire bag (“cut bags”). These exceptions are sent directly to the bag inspection room (BIR) without the opportunity for a screener to evaluate the image and clear the bag. • The second population includes bags whose contents include items that are misidentified by the EDS as potential threat items—for example, when the item’s properties fall within the window defined for threat items, or multiple items are mistakenly agglomerated in the image and are treated as a single item that meets the criteria for a potential threat item. This second population is the one evaluated to determine the probability of false alarm. The false alarm rate of the first population cannot be improved by improving the performance of the EDS because of limitations of the CT approach or of the baggage-handling system. The second population can be addressed by improving the ability of the EDS to interrogate the bag—for example, by improving the ability to distinguish threat materials from non-threat materials or by improving the ability of the image analysis algorithms to segment objects within a bag correctly. Although both of these populations contribute to the cost of evaluating false alarms, the committee focused on the second population, in which improvements to EDS technology can have an impact. The committee heard estimates of the false alarm rate from many sources—the DHS, the TSA, airport operators, and equipment vendors. However, the committee was not able to find hard data for the entire system beyond estimates of the overall false alarm rate. Even such fundamental data as the percentage of bags scanned that were identified as “exceptions” were not readily available. In the absence of such hard data, the committee directed its focus primarily on machine-driven false alarms that lead to a bag’s being sent to the BIR, as the additional time, personnel, and space necessary to resolve these alarms are likely a large component of the associated cost. 4 For example, false alarms caused by hardware problems or errors in image reconstruction (see the subsection entitled “Image Reconstruction and Correction” in Chapter 2) cannot currently be distinguished from those caused by the automated threat-recognition algorithm (see the subsection entitled “Material Density” in Chapter 2). 14 

STUDY PROCESS Over the course of its study, the committee held four meetings. At its first meeting, on October 23 and 24, 2008, in Washington, D.C., the committee discussed the statement of task for the study and heard the perspectives of the TSA from its representatives as well as the perspectives of others regarding issues related to EDSs and false alarms. The committee’s second meeting was held February 11 to 13, 2009, in San Francisco, California. As part of this meeting, the committee visited San Francisco International Airport where it observed baggage-handling and baggage-screening operations from bag check-in through scanning, the OSARP and inspection processes, and, finally, the clearance of bags for loading onto an airplane. The committee spent the 2 days after the site visit in information gathering. On February 12, it heard from the DHS S&T on its view of the statement of task, and then from employees of L-3 Communications and General Electric (GE) Security, who spoke about their approaches to CT algorithm development and false alarm reduction. 5 On February 13, the committee heard from two experts in human-machine interaction in the medical CT field. At its third meeting, held on April 27 and 28, 2009, in Washington, D.C., the committee heard from employees of the DHS and the TSA about the TSA’s approach to deploying new technology. The committee then heard from the National Safe Skies Alliance 6 about its research into false alarms for carry-on baggage. Following this presentation, as part of its efforts to understand how researchers in the realm of EDS CT can learn from advances in medical CT, the committee heard from an employee of the U.S. Food and Drug Administration (FDA), who discussed the FDA’s process of ongoing quality assurance for medical diagnostics. On the second day of this meeting, the committee heard from AAI Corporation (a Department of Defense contractor) regarding the use of performance-based incentive contracts. A speaker from Reveal Imaging then discussed that company’s approach to false alarm reduction. After the meeting, two members of the committee participated on April 29, 2009, in a visit to the TSA Systems Integration Facility at Ronald Reagan Washington National Airport outside Washington, D.C. The committee held its final meeting on June 15 and 16, 2009, in Woods Hole, Massachusetts, to draft the final report and reach a consensus on its conclusions and recommendation. REPORT STRUCTURE Following the report Summary—which includes key conclusions, findings, and recommendations from the chapters of the report—and the background provided in this introductory chapter, Chapter 2 presents an overview of CT-based EDSs and their integration in the airport setting. Chapter 3 discusses possible alternative approaches to false alarm reduction in the field: the use of multiple CT scans to improve the probability of detection, the use of mass spectrometry, x-ray diffraction technology, and the effective use of orthogonal technologies. In Chapter 4, other approaches to complement improvements in technology—namely, possible adjustments to the contractual structure that the TSA uses with equipment manufacturers—are addressed with respect to decreasing false alarms in an airport setting. 5 A representative from Reveal Imaging, another large vendor of CT-based EDSs, was unable to attend this meeting but did speak at the committee’s third meeting. 6 As described on its website (http://www.sskies.org, accessed December 3, 2009), “The National Safe Skies Alliance is a 501c 3 non-profit organization, formed in 1997 to support testing of aviation security technologies and processes.” It is funded through the Federal Aviation Administration, and receives coordination support from the TSA. It conducts tests operationally at airports and pre-operationally at its facility in Alcoa, Tennessee. 15 

In response to the sponsor’s desire to capitalize on lessons from the medical CT realm, Chapter 5 provides an overview of medical CT systems and an analysis of approaches that may and may not be appropriate to incorporate into CT-based EDSs. Finally, to provide the sponsor with guidance on how to make better use of the already-existing data in order to gain a better understanding of, and to be better able to address, the causes of false alarms, Chapter 6 focuses on issues related to data collection and analysis. The appendixes of the report provide the following information. Appendix A presents biographical sketches of the members of the committee. Appendixes B, C, and D, by individual committee members, are independently authored papers with the endorsement of the rest of the committee. Appendix B outlines an approach to quantifying the risk and causes of false alarm scenarios associated with the airport screening of checked baggage. Appendix C discusses chemistry-based alternatives to computed tomography-based explosives detection. Appendix D presents a statistical approach to reducing the probability of false alarms while improving the probability of detection. Appendix E presents the committee’s statement of task. Finally, Appendix F provides a list of the acronyms used in the report and their definitions, as well as a brief glossary. 16