Appendix F Test Protocol for Bulk Explosives-Detection Equipment

Daily Performance Verification

Equipment of the sophistication of bulk explosives-detection equipment will likely have diagnostic procedures designed by the manufacturer that are resident in the instrument. Typically these diagnostic procedures determine if certain parameters (voltage, signal, frequency, etc.) are within certain boundaries. In conjunction with test articles, these diagnostic procedures can test functional capabilities of the equipment. If the test articles and parameter values are defined and supplied by the vendor and are tested during certification and if performance-verification testing are shown to correlate with performance, they could be used as a daily calibration and check of explosives-detection equipment.

An example test is shown in Figure F-1, which is composed of secondary standard materials embedded in a matrix that simulates clothes, books, plastic, etc. The panel is not proposing that the explosives-detection equipment manufacturers or the FAA fabricate or use the test article exactly as shown in Figure F-1. Rather, the test article shown in Figure F-1 is a conceptual example that more closely represents the threat than does a test article that merely determines certain performance metrics of explosives-detection equipment. The main point of this conceptual example is that the development of a standard test article—one that is independent of the equipment being tested within a specific technology area (e.g., x-ray computed tomography [CT])—representative of the threat to aviation security is of critical importance. A collaborative effort between the FAA and explosives-detection equipment manufacturers is needed to develop a standard test article that includes materials representative of explosives and common nonexplosive materials (i.e., items typically found in passenger baggage). Any test article proposed as a standard (for a particular technology area) for performance verification of explosives-detection systems (EDSs) should be validated against the threat materials it is intended to represent using EDSs based on the same technology (e.g., x-ray CT) produced by at least two different manufacturers.

For the case of equipment based on CT (e.g., CTX-5000-SP) one CT slice would be taken at every level (as shown in the side view in Figure F-1) such that the following system performance parameters could be measured while testing the test article: spatial resolution, contrast sensitivity, noise level, or perhaps the probability of detection and the probability of false alarm. Daily testing of bulk explosives-detection equipment, as recommended by the panel, will involve using a test article similar to that described in Figure F-1 at each personnel shift change. Given the condition that baseline parameter values will be known, subsequent test results can be referenced to these values as a measure of system performance. This daily test could be automated such that the operator could simply put the test article in the machine, push a button, and be presented with a "go" or "no-go" message.

Comprehensive Performance Verification

The process recommended by the panel for comprehensive performance verification in the field requires that the FAA establish a testing approach (a protocol) and baseline performance measures that are used to verify the performance of each device or system at the manufacturing site or at an airport. To date only one manufacturer has produced an FAA-certified EDS (i.e., InVision). The EDS is a bulk explosives-detection system (as opposed to a trace-type system). For this reason, the panel has focused on approaches for performance verification of FAA-certified bulk EDS. Much of what follows, however, could be applied to performance verification of noncertified bulk explosives-detection equipment.

For comprehensive performance verification of deployed bulk equipment the panel recommends using a secondary standard bag set to obtain estimates of the probability of detection



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 64
--> Appendix F Test Protocol for Bulk Explosives-Detection Equipment Daily Performance Verification Equipment of the sophistication of bulk explosives-detection equipment will likely have diagnostic procedures designed by the manufacturer that are resident in the instrument. Typically these diagnostic procedures determine if certain parameters (voltage, signal, frequency, etc.) are within certain boundaries. In conjunction with test articles, these diagnostic procedures can test functional capabilities of the equipment. If the test articles and parameter values are defined and supplied by the vendor and are tested during certification and if performance-verification testing are shown to correlate with performance, they could be used as a daily calibration and check of explosives-detection equipment. An example test is shown in Figure F-1, which is composed of secondary standard materials embedded in a matrix that simulates clothes, books, plastic, etc. The panel is not proposing that the explosives-detection equipment manufacturers or the FAA fabricate or use the test article exactly as shown in Figure F-1. Rather, the test article shown in Figure F-1 is a conceptual example that more closely represents the threat than does a test article that merely determines certain performance metrics of explosives-detection equipment. The main point of this conceptual example is that the development of a standard test article—one that is independent of the equipment being tested within a specific technology area (e.g., x-ray computed tomography [CT])—representative of the threat to aviation security is of critical importance. A collaborative effort between the FAA and explosives-detection equipment manufacturers is needed to develop a standard test article that includes materials representative of explosives and common nonexplosive materials (i.e., items typically found in passenger baggage). Any test article proposed as a standard (for a particular technology area) for performance verification of explosives-detection systems (EDSs) should be validated against the threat materials it is intended to represent using EDSs based on the same technology (e.g., x-ray CT) produced by at least two different manufacturers. For the case of equipment based on CT (e.g., CTX-5000-SP) one CT slice would be taken at every level (as shown in the side view in Figure F-1) such that the following system performance parameters could be measured while testing the test article: spatial resolution, contrast sensitivity, noise level, or perhaps the probability of detection and the probability of false alarm. Daily testing of bulk explosives-detection equipment, as recommended by the panel, will involve using a test article similar to that described in Figure F-1 at each personnel shift change. Given the condition that baseline parameter values will be known, subsequent test results can be referenced to these values as a measure of system performance. This daily test could be automated such that the operator could simply put the test article in the machine, push a button, and be presented with a "go" or "no-go" message. Comprehensive Performance Verification The process recommended by the panel for comprehensive performance verification in the field requires that the FAA establish a testing approach (a protocol) and baseline performance measures that are used to verify the performance of each device or system at the manufacturing site or at an airport. To date only one manufacturer has produced an FAA-certified EDS (i.e., InVision). The EDS is a bulk explosives-detection system (as opposed to a trace-type system). For this reason, the panel has focused on approaches for performance verification of FAA-certified bulk EDS. Much of what follows, however, could be applied to performance verification of noncertified bulk explosives-detection equipment. For comprehensive performance verification of deployed bulk equipment the panel recommends using a secondary standard bag set to obtain estimates of the probability of detection

OCR for page 64
--> Figure F-1 Example standard test article for daily performance  verification of bulk explosives-detection equipment.  Note: In this test article, N different simulated  explosives are surrounded by different background  materials that simulate common materials found in  luggage (e.g., food, clothes, plastic, aluminum, etc.). (PD) and the probability of false alarm (PFA) in addition to the PD and PFA values obtained using the primary standard bag set used in the certification process. This secondary-standard bag set is then used to test and obtain estimates of the probabilities (PD, PFA) for the deployed explosives-detection equipment. If statistically significant agreement exists between the test with the secondary-standard bag set at the FAA's Technical Center (at the time of certification) and the one at the airport, then the performance of the explosives-detection equipment is said to be verified. Secondary Standard Bag Set A secondary standard bag set should be developed by the FAA to consist of a number of representative international passenger bags (that do not contain explosives or simulated explosives) and a number of representative international passenger bags that do contain simulated 1 explosives. These simulated explosives should be validated by the FAA (see Annex II, NRC, 1993) and mimic the real explosives used in the primary standard bag set. This secondary standard bag set should be developed and controlled by the FAA personnel responsible for conducting the certification test and utilized in the conduct of all comprehensive verification testing. All data generated by the use of this bag set should be collected, analyzed, reported, and maintained by the FAA personnel. As the final phase of certification testing, the FAA should test explosives-detection equipment against the secondary standard bag set to obtain estimates of PD and PFA. The bag set and the data collected would be retained by the FAA as the baseline performance-verification data base. The FAA should make this secondary standard bag set large enough to yield statistically meaningful data, yet small enough to be manageable in terms of transporting the bags from the FAA Technical Center to the various sites. Furthermore, it is desirable to be able to conduct the test in a reasonable amount of time so as not to interfere with routine airport and airline operations. For example, a secondary standard bag set that is 20 percent of the size of the primary standard bag set (which totals some 2,150 bags) would result in a secondary standard bag set consisting of 430 bags. This would yield estimates of PD and PFA with a standard error 51/2 (2.236) times larger than the error in estimated PD and PFA obtained using the primary standard bag set. In general, if the secondary standard bag set is 1/N of the size of the primary standard bag set, the standard error of the estimate will be N1/2 times larger. 1   The panel believes that the FAA, in cooperation with the EDS manufacturers, should develop such simulants for the various technologies being considered or used for bulk EDSs. These simulants should be made available to the developers of the EDS so that they can be used in the process of early determination of the detection capabilities of the technology.

OCR for page 64
--> Statistical Analysis After the physical testing has been completed (i.e., the secondary standard bag set has been tested by the explosives-detection equipment), the following statistical tests should be performed: Test that the performance probabilities for the to-be-deployed explosives-detection equipment are statistically equal to or better than the baseline values of the equipment. Test that a "significant" number of the bags in the secondary standard bag set are correctly classified by both the explosives-detection equipment that underwent certification testing and the explosives-detection equipment being tested in the field. The former test will be used in the following example. Testing the hypothesis that the false-alarm rate of the explosives-detection equipment to be deployed is less than or equal to that of the equipment certified by the FAA (with M bags in the secondary standard bag set used for estimating PFA- pFA) with an error of the first kind of 5 percent, the value of the Y statistic is given by the following equation: where PFA is the false-alarm rate observed at certification and pFA is the false-alarm rate observed on the to-be-deployed equipment. If PFA- pFA is greater than or equal to Y, then one would have no reason to reject the hypothesis that the "population" value of the false-alarm rate of the to-be-deployed equipment is less than or equal to the "population" value of the false-alarm rate of the equipment at certification. This statement is made at the 95 percent level of confidence. As a guide to determining the size of the secondary standard bag set, M must be sufficiently large to allow one to assume normality in the distribution of PFA- pFA. Conducting a similar test using the simulated explosive threat bags, if it can be determined that there is no reason to reject the hypothesis that the population value of the detection rate of the to-be-deployed EDS is greater than or equal to the population value of the detection rate of the EDS at certification (again at the 95 percent level of confidence), then the performance of the to-be-deployed EDS is verified. Although the above refers to the to-be-deployed equipment, the same approach can be used to verify that the performance of deployed equipment is still at the level (or better) as when the equipment was certified at the FAA. Reference NRC (National Research Council). 1993. Detection of Explosives for Commercial Aviation Security. National Materials Advisory Board. Washington, D.C.: National Academy Press.