Engineering Aviation Security Environments—Reduction of False Alarms in Computed Tomography-Based Screening of Checked Baggage (2013)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

B Quantifying the Risk of False Alarms with Airport Screening of Checked Baggage The false alarm rate for checked baggage is high, and these bags must all be inspected by hand, adding a great deal to the overall processing cost for baggage inspection. The Transportation Security Agency and the Department of Homeland Security are seeking recommendations from the National Research Council for actions that will reduce the rate of false alarms, while not unduly compromising the throughput rate of baggage being screened or the probability of detecting explosives. Probability of detection, false alarm rate, and throughput are interconnected, and any solution proposed will result in trade-offs. For example, some actions to decrease the rate of false alarms will lessen the probability of detecting explosives. Thus, there are important constraints on reducing false alarms that must be taken into consideration when making any recommendations for their reduction. This appendix outlines an approach to quantifying the risk of false alarm scenarios associated with the airport screening of checked baggage and their causes. Studies have been performed on the causes of false alarms and other factors associated with the screening performance. 1 These studies have been based on sampled data from screening operations and revealed the contribution to false positives of different categories of articles such as cosmetics (e.g., creams, gels, powders, lotions). A rigorous analysis based on the principles of contemporary quantitative risk assessment (QRA) will provide value-added insights for taking corrective actions to reduce the frequency of false alarms. This appendix outlines a systematic process based on QRA principles for a rigorous analysis of the causes of false alarms. The QRA approach outlined here could be extended to allow an informed assessment of the trade-offs in decisions that could reduce baggage-handling costs. The QRA model proposed tracks baggage through the entire screening process and quantifies the alarm rate at each screening point in a manner that shows the interaction of all components of the screening system. A full-scope QRA would include detailed analyses of the causes of false alarms, which could include hardware, software and algorithms, screening operators, and baggage items and baggage packing procedures. The results of implementing a QRA of the type proposed would be as follows: • A quantification 2 of the frequency of occurrence of various screening scenarios. • A quantification of the frequency of different outcomes of the scenarios in terms of the final disposition of the baggage, such as calling in a bomb squad or allowing the baggage to be loaded on the airplane, and the potential consequences of the different outcomes. • A quantification of the false alarm scenarios burdening the screening system and their causes, thereby enabling the development a roadmap for taking corrective actions to reduce their frequency. NOTE: This appendix was independently authored by John Garrick, committee member, with the endorsement of the rest of the committee. 1 Frannie Hamrick, “Field Data for Carry-on and Check Points,” presentation to the committee on April 27, 2009, Washington, D.C. 2 Quantification is taken to mean a full disclosure of what is known about a parameter, including its uncertainty and the supporting evidence. It does not mean absolute certainty, but it implies the quantification of uncertainties. 64 

• A quantification of the uncertainties associated with false alarm rates. The result should be a clearer path forward for a data collection and processing system that more directly exposes false alarms as well as guidance on improved machine algorithms. The QRA method 3 has been used extensively to enhance the safety and operational performance of complex systems in the nuclear, chemical, aerospace, defense, transportation, and environmental fields. The discussion below is general and omits detailed analysis, because the intent is to describe the QRA approach, not perform an actual QRA. THE SYSTEM The quality of a QRA is determined by the extent to which it represents the system being analyzed. In this case the system is a generic screening process for checked baggage typical of U.S. airports. The main components are presented in Figure B-1. The computed tomography (CT) scanner produces cross-sectional images of the bags. The images are either (1) a set of contiguous slices, known as three-dimensional or volumetric data, or (2) a variable number of slices at varying slice spacing, known as selective slices. The CT scanner may be combined with an x-ray line scanner that is a threat image, projection-ready x-ray scanner, where the images from the scanner are used to determine where to acquire the selective slices. The automated threat recognition (ATR) system algorithm processes the images produced by the CT scanner to identify the locations of potential threats within bags. Cleared bags (bags with no identified threats) are sent to the airplane. The ATR algorithm is characterized by its probability of detection (PD) and its probability of false alarm (PFA). The ATR algorithm may run on computers in the CT portion of the explosive detection system (EDS) or on the baggage viewing station. The ATR algorithm also analyzes the images of the bags to determine whether a threat could be shielded from the x-rays used in the EDS. If shielded regions are found in the bag, the bag and its images are sent directly to the baggage-inspection room BIR. If in the course of the baggage handling there is loss of identification of a bag (mistracking), that bag is also sent directly to the BIR. A computer monitor at the on-screen resolution (OSR) station displays images of bags that the ATR identified as containing potential threats. A transportation security officer (TSO) may clear the decision of ATR using available protocols. Cleared bags are sent to the airplane. If the ATR identifies multiple potential threats, the TSO may clear some or all of the threats. The BIR receives bags that have not been cleared during OSR. TSOs in the BIR visually inspect the threats or apply explosive trace detection (ETD). If the TSO clears the threats, the bag is sent to the airplane. Bags with remaining threats are handled by an ordnance disposal team (ODT). Thus, false alarms of threats are driven by both machines and humans. In a machine-driven false alarm, the screening algorithm signals an alarm when there is no threat. Common causes of machine driven false alarms are non-threat substances mistaken for a threat substance or items that aggregate several non-threat items into single items that meet the screening criteria for a threat. A human-driven false alarm involves a TSO. When prompted by the ATR to investigate a specific item or area of a bag, the TSO may mistake a non-threat substance for a threat. In particular, a search for causes of false alarms must investigate the machine, including the data being processed and the ATR algorithm, and the decision-making process of the TSO. 3 B.J. Garrick and Robert F. Christie, Quantifying and Controlling Catastrophic Risks, Elsevier, Amsterdam, 2008, pp. 17-31. 65 

FIGURE B-1 Diagram of an in-line EDS consisting of (A) a CT scanner;(B) an automated threat recognition (ATR) algorithm, (C) a baggage viewing station and the on-screen alarm-resolution protocol (OSARP), and (D) a control computer. This is integrated with (E) the baggage handling system, (F) the baggage inspection room and/or area, and (G) the ordnance disposal team. Shaded boxes are components of EDS. White boxes are subsystems used in conjunction with the EDS. Solid connecting lines show flow of bags and/or images of the bags. Dashed connecting lines show the flow of control and information. THE MEASURE OF RISK Generally risk is assessed with respect to a threat to human health (injuries and fatalities), damage to a facility, a transportation accident, an environmental impact, a catastrophic event, or other such situations. In this illustration the committee focuses on the risk of false alarms from EDSs, and in the process exposes their causes to guide corrective actions for their reduction. The parameter of the model is the frequency of false alarms and, more particularly, the frequency with which alarms lead to different action states such as extra screening or even the need for an ODT. Of course, there will be variability and uncertainty in the frequency, and that is something that must be a part of the quantification process. We account for uncertainties using the language of probability. DEVELOPMENT OF THE RISK SCENARIOS The cornerstone of the QRA approach advocated here is the triplet definition of risk. 4 That is, when we ask “what is the risk of something?” we are really asking three questions, • What can go wrong? • How likely is it to go wrong? And, • What would be the consequences? The main task of the risk triplet framework is developing the “what can go wrong?” scenarios. These scenarios can be structured in a variety of ways; the one that should be used is the one that works best for the analyst and the system being analyzed. One framework for structuring scenarios has had a great deal of success: the event tree. Basically, an event tree is an inductive reasoning logic diagram that traces the response of a system to different stimuli, that is, to different initiating and intervening events 4 Ibid., pp. 18 and 19. 66 

FIGURE B-2 Event-sequence diagram for airport checked baggage. until the response is terminated in the system either when the system corrects itself through automated or administrative action or when it goes into a particular damage state or undesirable state. Of course, there can be many paths through an event tree. Each path is considered a scenario. Figure B-2 is an event sequence diagram that outlines the bag-inspection process flow and process logic. Each path through the diagram corresponds to a “what can go wrong?” scenario. The event tree in Figure B-3 provides the structured set of scenarios desired. It communicates the logic of the system including the branch points and the interaction details of the subsystems A, B, C, and D. Figure B-3 can serve as a roadmap for identifying the various scenarios leading to false alarms. Each path through is a scenario. These paths are readily identified in the figure, where one can see where a scenario originates, which path it takes, and its end point in terms of impeding the flow of baggage to the airplane. For most systems, there are several initiating events and thus several event trees and in some cases thousands or even millions of scenarios. The good news is that the physics of the process usually leads one to a reasonably small set of scenarios that dominate the overall risk. A comprehensive risk assessment of false alarms would most likely involve segregating the causes and developing separate event trees for each cause set. Candidate cause sets for baggage inspection include cosmetics, foodstuffs, metals/electronics, paper (including books), shoes, bag parts, hardware characteristics, software characteristics, and traveler packing practices. It is obvious that once the logic is laid out and it becomes clear what is needed to quantify the scenarios, the data can be analyzed to quantifying the probabilities associated with each scenario. With knowledge of the scenarios and the attendant logic, the data analysis can be very efficient because it is clear just what information is needed. QUANTIFICATION OF THE SCENARIOS The branch points of the event tree are called “split fractions.” In order to track the fraction of alarms that are indeed false, it is necessary to know how that fraction varies in order to represent the variability and uncertainty involved. When data exist on false alarms at the branch point and a researcher knows the variability of those data, then the split fraction distribution can be obtained directly from the data. Often such data are limited, and it is necessary to perform some probability analyses to obtain the 67 

split fraction distribution that reflects both the variability and the uncertainty. In other words, the available data, other quantifiable information, and probability methods are the basis for assigning distributions to each of the split fractions. Rates and split fractions vary by such factors as airport, plane destination, time of day, and season of the year. It will be necessary to do separate analyses for given levels of these factors in order to assess—for example, the potential consequences of process or practice changes at a particular class of inspection facilities. For other questions, the scenario distribution can reflect the combination of variability and uncertainty in the occurrence of the scenario. For example, human factors could be captured by probability distributions describing operator responses over a range of image types or bag features. That is, for a given set of bag features, there is a particular probability that the operator clears the bag. When the data are strong, the scenario distributions would represent only the variability in the frequency of the split fraction. If the data are not strong, then Bayesian methods or other probability methods can be used to process the available data and other information into probability distributions reflecting available knowledge about the split fractions. Because there are only limited data available on the detection of actual threats, these other methods would be needed to assess scenarios in an expanded QRA that assesses the risk of both false positives and false negatives. FAULT TREE ANALYSIS To develop the split fraction distributions so as to reveal false alarm causes, the committee introduces another risk assessment tool known as the “fault tree.” Whereas the event tree is basically an application of inductive logic and thus the framework for structuring event sequences or scenarios, the fault tree is based on deductive logic and is useful for quantifying split fractions. The fault tree starts with the undesired event—for example, a false alarm—and works backwards decomposing the logic to basic causes or events. Even if it is possible to obtain the split fraction frequencies and their variability directly from field data, fault-tree-type analyses are necessary to reveal the basic events triggering the false alarm. The power of fault tree analysis is the ability to trace undesired events to such basic causes as equipment components and parts, software and algorithms, or human reliability. Figure B-3 is a simple fault tree to illustrate some of the key logic gates used in structuring fault trees. Extensive software exists for processing fault trees. Complex systems require a more comprehensive set of logic gates than is shown in Figure B-3. Examples of other logic gates include “conditional,” “inhibit,” and “external event” gates. But Figure B-3 presents the general idea. For example, the diamond-shaped box indicates that the logic of the event remains to be developed. Thus, if equipment were the cause of the alarm the logic would be developed to expose the equipment basic event(s) that caused the false alarm, whether they be hardware failure or a fundamental limitation of the equipment. The task then is to seek the supporting evidence for assigning a probability distribution to that basic event. The fault-tree logic is the equation path for then calculating the probability distribution of the split fraction. QUANTIFICATION OF EVENT FREQUENCY Once the split fractions are quantified, the logic of the event tree can be implemented. For example, with respect to the six scenarios in Figures B-3 and B-4 involving the top events A, B, C, and D, and the initiating event, I, the Boolean expressions for scenario, Si i = 1,6 are given in Table B-1, where the bar above the letter denotes a threat, i.e., an alarm. 68 

FIGURE B-3 Generic fault tree illustration. FIGURE B-4 Event-tree diagram for airport screening of checked baggage. TABLE B-1 Boolean Expressions for Scenarios 1 Through 6 Scenario Event Description Frequency Scenario Cost S1 = I ABCD Bag cleared by EDS Φ(S1) C1 𝐶̅ D S2 = I ĀBCD Bag cleared by OSR Φ(S2) C2 S4 = I 𝐴 B 𝐶 𝐷 S3 = I ĀB Bag cleared by BIR Φ(S3) S5 = I 𝐴̅ � C � 𝐵 𝐷 Threat declared by BIR Φ(S4) C4 S6 = I 𝐴̅ � C � 𝐵 𝐷 Bag cleared by BIR after shield alarm/exception Φ(S5) C5 Threat declared by BIR after shield alarm/exception Φ(S6) C6 69 

Ultimately we want to quantify the frequency with which each scenario will occur. Therefore we need to transform the Boolean equations into frequency equations. Of course, the frequencies of the frequency is denoted by I (bags per unit time), the frequencies of the various scenarios, denoted by 𝜑 (Si), scenarios and the end states are linked to the throughput rate of the baggage. If the baggage throughput i = 1,6, are given by the following equations, where the ⨍ (X | Y) s are the split fractions (conditional densities) of X at any given branch point, conditional on the path history in Y: As mentioned above, if the data are strong (as might be expected for S1, S2, S3, and S5, because corresponding 𝜑 (Si) might be adequately represented by a single number. Otherwise the uncertainty is these events occur frequently), there will be a little uncertainty in the scenario frequency, and the quantified with a probability distribution. Suppose that we wanted to know how frequently false alarms resulted in involvement of the ODT. To obtain this result we would have to quantify the frequencies of both S4 and S6. For example, to quantify the frequency of S6, we need to convolve the distribution of the complete scenario. Figure B-5 sampling methods. The result is a distribution quantifying the uncertainty in 𝜑 (S6). illustrates the process whereby the probability arithmetic is usually performed using Monte Carlo S6 = I𝐴̅ � C𝐷 𝐵 � FIGURE B-5 Bayesian convolution of split fraction uncertainties. 70 

ASSEMBLING THE RESULTS under the curve represents a probability of 1. The area under the curve between any two values of 𝜑 is the Each scenario has a probability density curve like that illustrated in Figure B-6. The total area probability that 𝜑 (S6) is between those values. FIGURE B-6 Probability density for frequency 𝜑. There are several ways to communicate uncertainty in the risk when the uncertainty is quantified probability (the area between 𝜑1 and 𝜑2 of Figure B-7) is 90 percent of the area under the curve. The way by a probability distribution. One approach is to compute a 90 percent probability interval such that the to read this result is we are 90 percent confident that the false alarm rate is between 𝜑1 and 𝜑2. FIGURE B-7 Probability density function. Probability distributions similar to Figure B-7 can be computed for any given single scenario or for combinations of scenarios leading to a single consequence. For example, the event S4 or S6 depicted in Figure B-4 would have to be used to convolve 𝜑 (S4) and 𝜑 (S6) to obtain a probability corresponds to the event that a bag will result in the need to call the ODT. A process similar to that distribution to describe the variability and uncertainty in the frequency with which the ODT needs to be called. USE OF QRA TO REDUCE RISK For some decisions the expected cost would be a useful metric for assessing the benefits or the effect of making changes to the baggage-inspection system. For the example in this section, the expected cost could be computed as 71 

where f(𝜙i) is the probability density function of 𝜑 (Si). OTHER USEFUL INFORMATION FROM A QRA Although risk measures such as those mentioned earlier in this appendix are useful for decision making, they are not necessarily the most important output of the risk assessment. Often the most important outputs is the exposure of the detailed causes of the risks—a critical requirement for effective risk management and process improvement. The contributors to risk are buried in the results assembled to generate the curve in Figure B-6. And in particular, understanding the root causes of risk behind the split fraction analyses illustrated earlier using the fault tree methodology can provide insight to help determine the parts of the system that can be modified to produce important improvements—for example, a modification that will reduce the probability of a false positive without reducing the probability of detecting explosives. INTERPRETING THE RESULTS As indicated earlier, the link between the proposed model and the actual reduction of false alarms is the quantification of the total screening process in such a manner that part of the output is the exposure of detailed causes of the risks—in this case, the causes of false alarms. Knowing the causes is the prerequisite for taking corrective actions to reduce their occurrence. The simplicity of the screening system and the experience base provides an opportunity to collect high-quality data on false alarms. Thus, the most important contribution of applying QRA principles may be to define and fine tune the data management system that will best reveal the causes of false alarms. It is not apparent, however, that any such focused data management system exists that has the scope to quantify false alarm rates and their causes in relation to the total screening system. While the screening process as a system is indeed simple, the individual components of the system are state of the art and are being pushed to their performance limits, primarily because of the throughput demands of airport screening. Experience indicates that applying QRA methods to the analysis of complex equipment has excelled in exposing the fundamental causes of undesired performance outcomes. EXTENSION OF QRA TO THE ASSESSMENT OF A BOMB THREAT There are a number of other ways in which QRA might be applied to study and improve the baggage-handling processes. For example, the approach described above was entirely conditional on having no real threats in baggage (the usual situation). The QRA model could be extended to include a parallel set of scenarios conditional on there being real explosive and other components of a bomb in the inspected baggage. The more difficult parts of this extension are the quantification in the absence of any meaningful data of the costs of the scenarios that involve on-board explosives and of the probability of some of the elementary events. This extended QRA would be valuable to assessing combinations of changes to the baggage-inspections process. For example, there might be some changes that would decrease PFA substantially with only a small increase Pd, in combination with other changes that would increase PFA marginally but decrease PD substantially, resulting in a net improvement in PFA without an overall decrease in PD. 72