4
Baseline Characteristics of Explosives-Detection Systems Based on X-ray-Computed Tomography
A legitimate comparison of x-ray CT-based EDSs and PFNTS-based explosives-detection techniques must be based on their baseline performance. Laboratory test results for PFNTS were presented in Chapter 3. In this chapter, both the laboratory performance and the operational performance of the FAA-certified CTX-5000 SP are reviewed. To date, most of the performance data on deployed explosives-detection equipment has been generated from tests conducted at the FAA Technical Center. However, operational (field) test data on false alarm rates are also reviewed in this chapter.
Test Data from the FAA Technical Center
Most of the performance data available for x-ray CT-based EDSs are from certification tests of the InVision CTX-5000 SP in 1996. Data from the certification tests of the recently certified InVision CTX-5500 DS and L3 Communications 3DX-6000 are not available yet but should be included in future analyses. The InVision CTX-5500 DS has been certified for two different inspection modes: SURE98 Mode, which has a lower false alarm rate than the CTX-5000 SP but a lower throughput rate, and CERT98 Mode, which has a similar false alarm rate to the CTX-5000 SP but a much higher throughput rate. The panel's analysis of performance data focuses on the CTX-5000 SP, although performance data for the CTX-5500 DS are also presented for reference. Table 4-1 shows the performance factors for the CTX-5000 SP and CTX-5500 DS, including the Pd, Pfd, and the bag throughput rate. Because the actual Pd and Pfa numbers are classified,1 Pd is given as a percentage of the overall Pd required for certification (X), and Pfa is given as a percentage of the Pfa required for certification (Y).
Operational Demonstration
In 1995, the FAA initiated the Airport Operational Demonstration Project (FAA, 1995) to determine the operational performance of the InVision CTX-5000 SP in the field as compared to its performance in certification tests. Three test sites were selected for the project: San Francisco International Airport (United Airlines); Atlanta Hartsfield International Airport (Delta Airlines); and Manila International Airport (Northwest Airlines).
TABLE 4-1 Performance Test Results for the InVision CTX-5000 SP and CTX-5500 DS
Equipment |
Detection Rate |
False Alarm Rate |
Throughput (bags/hour) |
||
|
Sample Size |
Pd(% X)a |
Sample Size |
Pfa(% Y)a |
|
CTX-5000 SP |
600b |
106 |
1,000c |
90 |
245 |
CTX-5500 DS/SURE98 mode |
600b |
106 |
1,000c |
60 |
264 |
CTX-5500 DS/CERT98 mode |
600b |
100 |
1,000c |
95 |
362 |
a Percentage of the classified value required for certification. b A total of 150 bags were used containing improvised explosive devices (25 samples of six explosives, detonators, timers, and wires) and four different orientations to get a sample size of 600. c The 1,000 bags used to acquire the Pfa differed from the 600 bags used to acquire the Pd. The 1,000-bag set was also used to obtain throughput rates. |
Two InVision CTX-5000s were installed in Atlanta and one each in San Francisco and Manila. The demonstration project included four open tests and one blind (red team) test using improvised explosive devices to determine Pd. The Pfa was acquired routinely throughout the project on real passenger bags. Only the data from the San Francisco and Atlanta operational deployments have been documented in final reports (FAA, 1997a, 1997b, 1997c).
Some of the performance data for the CTX-5000 SP installed at San Francisco International Airport are given in Table 4-2. The automated explosives-detection capability of the CTX-5000 SP during open testing was about the same as the capability measured at the FAA Technical Center. However, operator intervention to resolve alarms measurably reduced Pd. This tendency was also observed during blind testing at San Francisco and Atlanta (FAA, 1997a, 1997b). During the operational demonstration project at San Francisco, the automated Pfa was 113 percent to 150 percent higher than the certification standard. Operational data reviewed by the inspector general of the U.S. Department of Transportation suggested that false alarm rates were as much as 169 percent higher than the certification standard. Data from the first of the four open tests show an average of 50 seconds for alarm resolution using the CTX-5000 SP. Although the resolution time was lower during subsequent tests, the combination of a high Pfa and a long alarm resolution time can have a significant impact on throughput rates. In fact, it was determined to be the limiting factor for throughput rate.
Although the CTX-5000 SP exhibited a significantly higher automated (i.e., without operator intervention) Pfa in operation at airports than during certification testing, the
TABLE 4-2 Summary of Open Testing of CTX-5000 SP at San Francisco International Airport
Tests |
Sample Size |
Pd(% X) |
Pfa(% Y)a |
Machine (automated) |
131 |
102 |
150 |
Machine + operator |
131 |
89 |
5 |
aPfa was obtained by measuring the Pfa for regular passenger baggage. |
image produced by the system is used by trained operators to assist in resolving the alarms. Thus, the actual Pfa (with operator intervention) is much lower than the automated Pfa measured during certification testing. Nevertheless, the time required for alarm resolution is a factor that must be taken into consideration when evaluating the performance of this EDS.
The FAA's Security Equipment Integrated Product Team (SEIPT) has deployed more than 40 InVision CTX-5000 SP EDSs since 1997. Thus, the FAA now has an outstanding opportunity to collect performance data (e.g., Pd and Pfa) as well as operational performance data (e.g., down time, mean time between failure, and mean time to repair). Unfortunately, this information was not available to the panel, which substantially compromised the panel's ability to assess the performance of deployed equipment and to make comparisons between deployed equipment and alternative technologies, such as PFNTS. Credible future assessments of the performance of deployed explosives-detection equipment will require both performance and operational data.