3
Laboratory Tests of Pulsed Fast Neutron Transmission Spectroscopy
PFNTS technology was tested for its capability to meet both the Pd (probability of detection) and the probability of a false alarm (Pfa) specified in the EDS certification standard for the range of explosive classes. The FAA conducted blind tests on two PFNTS-based explosives-detection devices, one developed by the University of Oregon and the other by Tensor Technology. The blind tests measured detection performance (coupled Pd and Pfa) but not bag throughput. The results described below can be used to set the demonstrated PFNTS detection performance level.
University of Oregon Blind Tests
The FAA conducted a series of blind tests (Chmelik et al., 1997) at the University of Oregon in September 1996 using the PFNTS B-matrix approach (Lefevre and Overley, 1998) and two spatial correlation algorithms, a contiguous-pixel test and a shape test. The software used to implement the B-matrix detection algorithm along with the blind test database have been archived (Lefevre, 1998). The University of Oregon detection system examined 3.2 x 3.2 cm2 (1.3 x 1.3 in.2) pixels of a suitcase. The test involved 134 suitcases and eight different nitrogen-based explosives. The FAA tester placed explosives in 75 of the bags. In six instances, two or more different explosives were placed in a single bag. Attempts were made to elicit false alarms by placing a variety of available materials in the bags. No attempt was made to combine materials along the neutron path to reproduce the elemental densities or ratios expected of explosives in cluttered bags. Attempts were made to conceal explosives by placing them in iron or aluminum pipes, behind books, wood, and other objects, and in radios or video cassettes.
Attempts to conceal explosives were mostly unsuccessful. In the initial blind test, using the automated contiguous-pixel test, 67 of the 75 explosive-containing bags were correctly identified, for a Pd of 89.3 percent. Eight of the 59 bags that did not contain explosives did set off an alarm, for a Pfa of 13.6 percent. Using the automated shape test, the presence of explosives in 70 of the 75 explosive-containing bags was detected, for a Pd of 93 percent; seven of the 59 bags that did not contain explosives set off alarms, for a Pfa of 11.8 percent. With operator intervention, the shape test operator, relying primarily on past experience, correctly identified 71 explosive-containing bags out of 75. However, in this test only 46 of the 59 bags that did not contain explosives were correctly identified. These results correspond to a Pd of 94.7 percent and a Pfa of 22 percent. Blind tests conducted at the University of Oregon, unlike EDS certification tests, distinguished between "true detections" and "false detections," that is, false alarms were registered if the explosive was not located in the portion of the bag identified by the algorithm. None of the 59 benign bags registered a false alarm. Most of the false alarms were produced by large areas of only slightly elevated explosive probabilities, which corresponded to the location of large books or blocks of wood. Table 3-1 summarizes the detection performance for the various algorithms used during the blind tests.
In four of the six cases where multiple explosives were placed in the bag, the system was able to detect the presence of each explosive. In the other two cases involving multiple explosives, the explosives were in close proximity and were identified as only one explosive. In 38 of the 75 explosives-
TABLE 3-1 Performance of the University of Oregon Explosives-Detection Algorithm in Blind Tests
Detection Algorithm |
Pda |
Pfa |
Contiguous pixel |
89.3% |
13.6% |
Shape test |
93% |
11.8% |
Operator intervention |
94.7% |
22% |
Post-test algorithm adjustment |
93.3% |
4.5% |
a Pd indicates a "true detection," the correct identification of the region of the bag containing the explosive. |
TABLE 3-2 Performance of the Tensor Explosives-Detection Algorithm in Blind Tests
Detection algorithm |
Explosives Classa |
Pd |
Pfa |
Operator assisted |
B |
94.3% |
25% |
Operator assisted |
A |
51.8% |
25% |
Operator assisted |
A-1 |
83.3% |
25% |
Operator assisted |
A-2 |
43.2% |
25% |
Automated neural net (with 10 scans eliminated because of possible interference) |
all classes |
88% |
24% |
Automated neural net detection on loose cargo when trained on bags |
all classes |
70% |
see note |
Automated neural net detection on loose cargo when trained on bags |
A |
40% |
see note |
a Explosive classes A-1 and A-2 are subclasses of explosive class A. Note: Although no false alarms were recorded, only 4 of the 60 cargo scans did not contain an explosive. Therefore, Pfa is not statistically meaningful in this case. |
containing bags, the PFNTS detection algorithm not only indicated the presence of an explosive but also successfully identified the explosives class. In another 16 cases, the actual test explosive corresponded with the second most likely explosives class predicted by the detection algorithm.
When the blind test pixel data were used to modify the B-matrix method, the error rate was significantly reduced. This post-processed blind test analysis missed five out of the 75 explosives and produced no false alarms in the 59 benign bags. If the six cases where the wrong region of the bag was identified as containing an explosive (when in fact there was an explosive in a different region of the bag) were considered to be false alarms, the Pd was 93.3 percent, and the Pfa was 4.5 percent (Algorithm 4 in Table 3-1). The performance for this algorithm must be treated as an indicator of the "potential" detection performance (Pd coupled with Pfa) for the PFNTS and not as a valid blind test result because post-processing of blind test data using detection algorithms optimized to the test data can produce misleading results.
The explosives the University of Oregon expected would be tested, which were specified in the "FNTS Developmental Test Plan," had a defined configuration. However, different than expected explosive configurations were used in the actual blind tests. The test final report (Overley, 1998) indicates that seven out of the eight missed explosives involved smaller amounts of explosive than the University of Oregon researchers expected to detect.
Tensor Technology Blind Tests
In blind tests at Tensor Technology, Inc., in late September 1997, 150 suitcases were scanned at two angles, 0° (broadside) and 60° in azimuth (Gibson et al., 1997; Tensor Technology, 1998a). The initial determination was performed by an operator and was "somewhat subjective" but was "based on a combination of explosive size, general level of attenuation, atomic number density analysis, and the neural net analysis." Table 3-2 shows the Tensor detection performance. The Pd was 94.3 percent for Class B explosives and 51.8 percent for Class A explosives (83.33 percent for Class A-1 and 43.18 percent for Class A-2). The Pfa was 25 percent. There was no significant difference in detection performance between the two scan angles.
An automated explosive detection algorithm is under development using the regression neural net analysis. A preliminary version incorporating the neutron attenuation magnitude and the nitrogen content was used to analyze the blind test scans. The details of the blind test results from this automated algorithm are not clear from Tensor's final report to the FAA (Tensor Technology, 1998b). However, the report suggests that the operator response was somewhat better than the initial automated response. Some questions have been raised about 10 test scans in which, because of attempts to limit interference from a table used to hold the test articles, the explosives may not have been in the detector array's field of view. If these 10 scans are omitted, the automated response algorithm had an overall Pd (for Class A and Class B) of 88 percent and a Pfa of 24 percent.
The Tensor blind tests included tests of the system's potential for screening loose cargo consisting of boxes with varied contents. Thirty loose cargo items were scanned twice each for a total of 60 scans. Explosives appeared 56 times during the testing, including four cases in which two explosives were detected in a single cargo item. A neural network detection algorithm trained on data acquired from luggage (not loose cargo) was used. The overall Pd was 70 percent. The Pd for Class A explosives was only 40 percent. No false alarms were recorded during the cargo scanning.
Some of the loose cargo containers were fairly dense. Six of the 17 missed explosives were in containers with a neutron transmission fraction of less than 0.003, which is comparable to noise levels of background scattered neutrons during screening. In other words, transmitted neutrons could not be differentiated from background noise caused by scattered neutrons at this low transmission level.
Detection of Class a Explosives
The PFNTS method has problems detecting Class A explosives (required by the FAA for certification). This may
be an intrinsic limitation for explosives-detection approaches that use a large pixel size (> 0.2 cm [> 0.08 in.]). Figures 3-1 and 3-2 show the resolution available to the detection algorithms used in the blind tests. Except for Class A explosives, the PFNTS detection performance (Pfa and Pd) has the potential to meet the FAA's EDS certification requirements. Because the Tensor and University of Oregon tests used very different detection algorithms and because both provided good detection levels for Class B explosives, the PFNTS elemental densities seem to provide a very robust set of measurements for explosives detection. Refinements in the detection algorithm or coupling with other technologies may be necessary, however, to overcome the limitations of PFNTS for the detection of Class A explosives. Once this limitation has been better quantified, it should be possible to determine the potential role for PFNTS in commercial aviation security.
The unreliable detection of Class A explosives is a serious deficiency of PFNTS, particularly for Class A-1 explosives. However, according to reports by the University of Oregon (Overley, 1998), some subclasses of Class A explosives can be reliably detected. Approaches to improving the capability of PFNTS to detect Class A explosives are listed below:
- reducing the pixel dimensions at the bag location
- using multiple scans at different angles and a tomographic analysis of the data
- improving spatial correlation algorithms so that large areas with a low probability of being an explosive are identified as an explosive
Determining the efficacy of these approaches will require further laboratory testing. Some of these approaches (e.g., tomographic analysis) will reduce the bag throughput rate compared to a single-view radiographic method.
Assessment of Detection Performance
The post-processing of blind test data using detection algorithms optimized to the test data can produce misleading results. For example, if the detection algorithm is optimized to specific test data, it is possible that when new data are analyzed (i.e., from a new set of baggage), the algorithm will not perform as well. Although the 4.5 percent Pfa and 93.3 percent Pd described above for post-processed data from the University of Oregon PFNTS tests indicate a detection performance level that may exceed the explosives-detection potential of x-ray-based CT approaches, these data must be treated with a healthy skepticism. Furthermore, the low Pfa attained during the University of Oregon tests—with a 16-element linear array detector—may not hold for a full two-dimensional array. The University of Oregon acknowledges that changing to a two-dimensional array detector could degrade the performance of PFNTS for the following reasons (Lefevre, 1998):
- As the detector geometry is expanded, the background noise level becomes much more severe (i.e., more scattered neutrons enter the scanning area).
- More collimator area is visible to each detector, which affects the line shapes for monoenergetic neutrons by increasing long flight-time tails.
- Neutron in-scattering from luggage items increases.
- Detector cross talk increases.
For all of these reasons, the panel cannot confidently state that the PFNTS system has the potential for a very low false alarm rate under conditions that meet the FAA's required throughput rate. The problems listed above will have to be addressed through more laboratory experimentation or detailed radiation transport modeling. The Tensor two-dimensional 99-element array detector exhibited a much higher false alarm rate during blind testing (compared to the University of Oregon tests), which may be associated with the background from a different neutron source configuration or the use of a different explosives-detection algorithm rather than intrinsic characteristics of the two-dimensional detector array.