APPENDIX A
A GENERAL TESTING PROTOCOL FOR BULK EXPLOSIVE DETECTION SYSTEMS

Developed in Consultation With The Committee on Commercial Aviation Security January 1993



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 37
Detection of Explosives for Commercial Aviation Security APPENDIX A A GENERAL TESTING PROTOCOL FOR BULK EXPLOSIVE DETECTION SYSTEMS Developed in Consultation With The Committee on Commercial Aviation Security January 1993

OCR for page 37
Detection of Explosives for Commercial Aviation Security This page in the original is blank.

OCR for page 37
Detection of Explosives for Commercial Aviation Security CONTENTS     A1. INTRODUCTION   41     A2. GENERAL REQUIREMENTS   45     A. EXPLOSIVE CHARACTERISTICS TO BE MEASURED   45     B. IDENTIFICATION OF THE SET OF THREAT EXPLOSIVES   46     C. IDENTIFICATION OF POTENTIAL BAG POPULATIONS   46     D. SYSTEM CALIBRATION AND THRESHOLD SETTINGS   47     E. MANUFACTURER/CONTRACTOR PARTICIPATION   47     F. TEST SITES   47     G. A STANDARD SET OF BAGS AND THREATS   48     A3. SPECIFIC REQUIREMENTS   49     A. IDENTIFICATION OF DISTINCT BAG POPULATION   49     B. IDENTIFICATION AND SELECTION OF THE THREAT   51     C. BAG PROCESSING RATES   52     D. PRE-TESTING   53     A4. DEVELOPMENT OF THE TEST   55     A. GENERAL FACTORS   55     B. DATA ANALYSIS PLAN   56     C. SELECTION OF TEST BAGS   56     D. NUMBER OF TEST BAGS REQUIRED   58     E. SELECTION OF THE NUMBER OF THREAT ARTICLES   59     A5. TEST EXECUTION   61     A6. ANALYSIS AND EVALUATION TEST DATA   65     ANNEXES   67 I.   STANDARD SET OF BAGS AND THREATS FOR TESTING BULK EXPLOSIVE DEVICES OR SYSTEMS AT AN FAA DEDICATED TEST SITE   67 II.   VALIDATION OF SIMULANTS   71 III.   MODEL TEST SCENARIO   79

OCR for page 37
Detection of Explosives for Commercial Aviation Security This page in the original is blank.

OCR for page 37
Detection of Explosives for Commercial Aviation Security A1 INTRODUCTION In March 1990, the FAA Technical Center established a task force of independent consultants1 to undertake testing and evaluation of the Thermal Neutron Analysis (TNA) technology as implemented by SAIC at Kennedy International Airport in New York (JFK). Based on the results of that work, the task force was then asked to prepare a protocol for the conduct of operational testing of explosive detection devices (EDD) or systems (EDS) for checked or carry-on airline baggage (bags, containers, etc). The FAA also intended to use the protocol in the certification of bulk explosive detection systems. This version of the protocol has been developed in consultation with the National Research Council's Committee on Commercial Aviation Security. The committee has differentiated between those testing aspects that pertain to certification and those that pertain to verification testing; added to the discussion regarding the use and composition of a standard set of baggage for certification testing; clearly recommends a FAA dedicated test site for certification testing; requires that the rationale for deviations from the protocol be documented; and provides for revisions to the protocol as additional testing experience is gained, added some discussion on testing with countermeasures, required analysis of false alarm data for verification testing, and required the documentation of deviations from the protocol. This protocol is specific to testing of production hardware, as opposed to developmental brass/bread board models or prototype versions of the equipment. This protocol is applicable to: the testing of devices or systems that are automated (i.e., no human intervention used for the detection process); test methods that do not change the characteristics of the item as a result of the test; test methods that detect explosives and explosive devices via bulk properties (e.g., vapor detection devices are excluded). This protocol does not provide sufficiently detailed plans and procedures to allow the testing and evaluation of a specific hardware device or system. However, the protocol provides the guidelines and framework for planning more detailed test procedures. 1   Dr. Joseph A. Navarro (Chairman); Mr. Donald A. Becker; Dr. Bernard T. Kenna; and Dr. Carl F. Kossack.

OCR for page 37
Detection of Explosives for Commercial Aviation Security As experience is gained in the testing of equipment, this protocol should be updated annually to incorporate additional or modified guidance. For each application, the FAA will establish the specific threat package (including size, shape, amount and type of explosive) to be detected. Although there will only be one overall threat package, one could envision (in a long range plan) that technologies could be appropriate for, or apply to, a subset of the threat package but not the total package. The FAA will also provide a description of any likely countermeasures that the equipment should be tested against. This protocol addresses two types of testing: Pass/Fail testing, required for certification testing; and parametric testing, used to obtain statistically-valid verification performance data. Table A1 summarizes the primary differences between these two types of tests. TABLE A1. Certification Versus Verification Operational Testing   Test Outcome Type Of Equipment Test Location Threat Package Bag Population Test Time EDS Certification Testing Pass/Fail Low rate or full-scale production units FAA Dedicated Site Live Explosives, types and quantities specified in the FAA's EDS Requirements Specification FAA Standard Set Limited Duration EDD Performance Verification Testing Parametric Data on Functional Characteristics Low rate or full-scale production units FAA Dedicated Site, or Airport Environment Live Explosives, or Simulants (at Airport Sites) FAA Standard Set, or Actual Passenger Bags Limited Duration, or Extended Duration (at Airport Site) In order for the FAA to make a decision on the operational functional characteristics of the device for systems, the FAA must consider: estimated probability of detecting explosives p(d), (as observed in the testing of the EDD/EDS. estimated probability of false alarm, p(fa), (as observed in the testing of the EDD/EDS). estimated processing rate of the bags, r. The FAA may also be interested in the trade-off between the two probabilities, p(d) and p(fa), especially for those approaches that can readily adjust detection thresholds (thus affecting these fractions). Other factors that should be considered by the FAA, in determining the characteristics of the detection equipment, include: reliability/maintainability/availability, cost, initial and recurring; size; and weight, significant operational constraints (environment, manpower, etc.), bag processing time distribution as appropriate, false alarm types and causes.

OCR for page 37
Detection of Explosives for Commercial Aviation Security The FAA will: provide the test team (which will be responsible for generating the specific test plan preparing and executing the test, analyzing and evaluating the test data, and preparing the report on the findings of the test), and establishing where and when the test will take place. The test team should have a test director and be composed of experts in the technology being tested, test and evaluation planners, and analysts who can design the statistical plan and conduct the evaluation of the test results. An independent observer should also be a member of the test team. This observer should comment on all activities associated with the testing and evaluation of the EDD/EDS, including: adherence to the test plan, adequacy of the plan, perceived testing limitations, and potential sources of test bias. All test baggage and test articles, the threat package (explosives or simulants), and personnel will be provided by the FAA. Chapter A2 of this protocol provides some general requirements associated with the operational testing process. Chapter A3 addresses a set of issues that must be considered and specific requirements that must be fulfilled prior to the development of a detailed test and evaluation plan. In Chapter A4, specific aspects of the detailed plan are discussed. Chapter A5 deals with issues related to the conduct of the test, while Chapter A6 discusses the data analyses and evaluations of the test data. Finally, Annex I discusses how the FAA could establish a standard set of bags and explosives to conduct operational tests or certification tests at a dedicated FAA test site. Annex II contains a suggested approach for validating explosive simulants and Annex III contains a model testing scenario.

OCR for page 37
Detection of Explosives for Commercial Aviation Security This page in the original is blank.

OCR for page 37
Detection of Explosives for Commercial Aviation Security A2 GENERAL REQUIREMENTS In order to develop a specific test plan, the test team must consider all of the factors that may: influence the conduct of the test; bias the measurements related to the detection equipment under test; and affect the results obtained from the test and/or the reliable interpretation of those results. The test team must establish the conditions under which the test is to be conducted and the characteristics (i.e., attributes or variables) of any bag as it is processed. There are a number of steps or topics that must be considered before development of a final specific test plan for the equipment under test. These may be separated into two groups: general topics and specific topics. The general topics are discussed in this chapter, and the specific topics are covered in the next chapter. However, since this is a generic protocol for a variety of explosive detection devices or systems (using different technologies), there may be some additional factors that may need to be considered, and the discussion that follows should not preclude additional factors from being included in the final test plan if those factors are considered to be relevant by the test team. It is assumed that prior to this point, the following activities have already occurred: The specific device or system presented to the FAA for testing would first have been tested by the manufacturer using, to the extent possible, the same protocols and performance criteria that the FAA will use. These procedures and results would then, in most cases, have been reviewed by the FAA for adequacy before the candidate device or system would be accepted for testing. The FAA would have appointed an independent test director and a test team to prepare a detailed test procedure from this general test protocol. The test team would be expected to conduct the test in accordance with the detailed test procedure. An agreement would be finalized with the equipment supplier regarding what results will be provided by the FAA after the testing is completed. A. Explosive Characteristics to be Measured After the specific equipment to be tested has been identified, the set of characteristics of the explosives that the equipment will measure for detection must be identified and specified. The physical principles employed

OCR for page 37
Detection of Explosives for Commercial Aviation Security by the instrument for detection will determine which physicial characteristics of the explosive material are of interest. This determination is especially important when the use of explosives—such as at airport facilities—will be prohibited during the test of the equipment, so that the use of a simulant should be considered. For example, one of the primary characteristics of explosives measured by TNA devices is the nitrogen content. The test team must determine if simulants can be identified which will exactly mimic the characteristic nitrogen content for the equipment under test. In addition, any countermeasure techniques to be included in the testing should be identified prior to the test initiation by the FAA, or by the test team in consultation with the FAA. Some technologies are relatively easy to countermeasure while others may be more difficult. B. Identification of the Set of Threat Explosives The set of threat explosives (type, shape, and weight) to be used in the testing must be specified by the FAA. For example, the FAA may require that testing must include x pounds of sheet explosive of RDX/PETN base (Semtex). The FAA must also specify the relative frequency of expected occurrence for each item in the set of threat explosives. The FAA should identify the placement location for the explosives in the containers. During the course of testing, it may be observed that the instruments response is sensitive to the placement location of the explosive in the bag. If this occurs, the FAA should consider additional testing with the threat in most disadvantageous locations. C. Identification of Potential Bag Populations The test data base should contain observations for the test bags of all the major characteristics that will be measured by the detection system being tested. This database will be used by the test team to select representative groups of test bags. The actual set of bags used for testing could be: (1) actual passenger bags; (2) fabricated by the FAA appointed team; or, (3) selected from the set of FAA ''lost'' bags. Bag selection is a crucial topic in designing a test plan that is fair and effective. For certification testing of an EDS, the testing should be performed on a standard set of bags to provide a fair and consistent comparison against the EDS Standard. Annex I discusses how a standard set of bags could be established. For non-certification testing, the goal will be the generation of parametric performance data. The test team must determine the characteristics of bags typical of those that will be processed at a designated airport. If the testing itself will not be conducted at the airport, data on actual passenger bags being processed at those facilities could be collected. If these data were collected over a sufficiently long period of time, the effect of seasonal and bag-destination differences on bag characteristics could be determined.

OCR for page 37
Detection of Explosives for Commercial Aviation Security Additional information on bag selection techniques is contained in Chapter A3. D. System Calibration and Threshold Settings This protocol applies to devices or systems that are totally automatic in their response; i.e. operator independent. The manufacturer will not be allowed to change or modify the settings of the equipment once the test for a given bag population has been initiated. Thus prior to the start of the test, the manufacturer should be allowed to have access to the set of bag populations that will be used for the testing so that they can determine the associated response of the equipment to the characteristics, and thus calibrate and establish the threshold settings of the equipment. The manufacturer should provide the FAA with the complete calibration protocol. The manufacturer should not be provided details regarding the relative frequency of threat occurrence and location of the threats in the bags, since the tests must be as blind as possible. While the manufacturer is establishing the threshold settings, they should also be required to provide the FAA test team with the p(fa) versus p(d) relationship as a function of the threshold setting. This data should be made available prior to the testing unless the equipment can store the basic data so that, after the fact, the p(fa) versus p(d) curves for the different threshold settings could be reconstructed. E. Manufacturer/Contractor Participation Although the test team may be required to rely heavily on manufacturer or contractor personnel for support in conducting the testing, procedures should be established to minimize the possibility that they could influence the test results. Toward this end, the manufacturer may be required to train FAA chosen personnel to operate the equipment during the test. These personnel should have the same general skill levels as those who will be expected to operate the equipment in the airport environment. F. Test Sites For EDS certification testing, an FAA-dedicated test site is required that can be controlled and characterized, particularly with respect to background contamination. Explosives representing the standard threat set must be used. The standard bag population also must be used. Similar considerations apply to EDD verification testing at the FAA test site although a bag population representative of a specific airport may be used. For testing conducted at airports, available passenger bag populations could be used. If tests are conducted over an extended period of time, the effect that various seasons have on the characteristics of the bags can be determined. In most cases, the use of explosives will not be acceptable and simulants will be required. (Simulants would also be needed to monitor the performance of the devices or systems while they are operational at airports.)

OCR for page 37
Detection of Explosives for Commercial Aviation Security This page in the original is blank.

OCR for page 37
Detection of Explosives for Commercial Aviation Security ANNEX II VALIDATION OF SIMULANTS Annex II contains a detailed statistical analysis procedure for validation simulants. 1. Test Procedures. Explosives will be placed in bags representative of the total set of population of bags being tested. The bags are then inspected by the equipment, and the resultant values of each of the characteristics are recorded. Then, using the same set of bags but replacing the explosives in the same location with explosive simulant, repeat the test. Note: if possible, the selection of the location should be such as to provide high signal-to-noise ratio in order to facilitate the explosive-simulant comparison. All bags should be sampled at least ten times with the explosive, with the corresponding simulant and with neither. By comparing the average and the variances of the reading of the characteristics with explosives in the bag, with the average and the variances of the readings of the characteristics with the simulated explosive in the same bag, the validity of each simulant can be determined. For example, in order to test simulants of ten explosive types identified by FAA, fifteen bags, covering the population range of the characteristics being measured, are recommended for use. Five empty bags (containing neither explosive nor simulant, but only the normal contents of the bag) would be randomly placed among the ten bags containing either explosives or simulants on every one of the major runs that are used to obtain data on the explosives and the simulants. One purpose of using the same empty bags on each run is to collect data to evaluate if the equipment has ''memory,'' in that if higher values of characteristics are being measured on a given run (as a result of ten bags containing explosives or simulants) then the equipment might be reading higher on all bags, including the empties. A second purpose is to collect large quantities of data of a selected number of bags to estimate the variability of the measurements from the selected bag set. This should be used to reconfirm the bag selection process for validation testing and to help guide the test team in the bag selection process for testing at an airport terminal. The test team must monitor all aspects of the validation testing, record data for each bag processed, verify the sequencing of bags through the detection equipment, place all explosives and simulants in the appropriate bags, and observe the verification tests of each of the samples of explosives used for comparison against the simulants. 2. Test Results. The detailed data collected on each bag will be maintained by the FAA. The mean value and standard deviation of the measurements of the characteristics for empty bags, the bags with explosive and the same bags with the simulated explosive should be recorded and compared. For each

OCR for page 37
Detection of Explosives for Commercial Aviation Security characteristic, its mean value for the bags with an explosive is compared to its mean value for the bags with the simulated explosive using conventional statistical analyses, such as F-tests and t-tests. A priori, the test team should select the critical region for both the F and t tests (i.e. , reject the hypothesis that the simulant is representative of the explosive with respect to the characteristics being considered). This region is usually selected to be at the a=5 percent level. Based on this data, simulants are accepted or rejected using the "t test." The rejected simulants may be reworked (if possible) and rerun through the equipment. This process may be continued in an attempt to validate as many simulants as possible. For overall confirmation, analysis of variance can be performed to determine if differences of the mean value of the characteristic of the bags with real explosives and the mean value of the characteristic of the bags with simulated explosives were zero (using the F-test, with the associated test of homogeneity). The detailed statistical approach that can be used is described in any statistical analysis book. The validated simulants should be suitably marked and immediately placed in the custody of the test team who will then deliver them to the test site at the appropriate time. If the simulant characteristics can change over time, the simulants should be revalidated after appropriate time periods. The decision to be made is whether or not a given simulant gives a similar enough response to that given by the real explosive as far as equipment operation is concerned, namely detect or no detect. The following testing is recommended for the validation process. Start with a representative group of J+Q bags where J equals the number of different explosive types being simulated and Q represents a small number of additional bags. For the j-th explosive type, there are m(j) simulants which need to be validated. Here the m(j) depends on the test design. These J+Q bags are run through N times for each of the three conditions, (1) with the explosive, (2) with the matching simulants, and (3) with neither (to check on additivity of the measurements and to ensure that the bag characteristics have not changed over the testing period). The explosive and its matching simulant are also placed in the same bag in the same position, that position being where the measurements of the characteristics are considered to be optimal for measurement purposes. The set of characteristic measurements of the bag are the determining operational variables used in the test. Thus the primary data can be considered as three matrices: where (1) is the observed value of the i-th characteristic in the k-th bag, (2) is the observed value of the i-th characteristic with explosive type E(j) in the k-th bag, and (3) is the observed value of the i-th characteristic with

OCR for page 37
Detection of Explosives for Commercial Aviation Security the explosive replaced by the m-th simulant of the j-th type explosive, S(j, m), in the k-th bag. Here, The following process is recommended: Run K+Q bags through N=10 times without explosives or simulants. This data is needed to ensure that the bags being used are representative of the bag population they are thought to be representing and to check for additivity of the measurements; Place the J explosives in the K bags, selecting the assignment at random, all in the same location, that being where the a{} readings are expected to be least variable. If appropriate, use the flip/twist move procedure, described above, to sample the a{} at different relative locations in the equipment. Run the K+Q bags through N=10 times; Replace each of the J explosives with a simulant of that explosive, S(j,m), and repeat (ii), continuing this until all simulants have been tested in bags which contained the appropriate explosive; At the beginning of each new day of testing and at the end of the testing, repeat (i). 3. Analyses for the Validation of Simulants Let A{}=Sa{}/10 be the mean value of the respective reading, taken over the 10 observations on the k-th bag, and v()=S[a{}-A{}]2 /9 be the sample variance. In order to validate a simulant, we are testing the hypothesis that the population mean as estimated by A{i,E(j),k} is equal to the population mean as estimated by A{i,S(j,m),k}, given the populations are normally distributed with equal variances. Three statistical tests can be used to determine if these samples came from the same distribution, so as to justify the validation of the m-th simulant of the j-th type explosive. a. The Significance of the Difference Between Two Sample Means. In this approach only one bag is used for the testing of the explosive and the simulant of that explosive. That bag (say the k-th bag) should be randomly assigned for a given simulant and explosive pairing.

OCR for page 37
Detection of Explosives for Commercial Aviation Security For a simulant of the j-th type explosive, one now tests that A{i,S(j,m),k}=A{i,E(j),k}, for all m=1,. . .,m(j) and for all characteristics i=1,. . .,I. If, for each of the I characteristics, the two sample means are not significantly different at the 1-a level of significance using the two-sided alternative, we will declare the m-th simulant to be validated, at the 1-(1-a)I level of significance. To make such a statistical test, one must first determine if the two observable random variables, for each of the I characteristics, have equal variances or not. To test this one uses the Snedecor F statistic computing F(9,9)=V(i,j,m)/v(i,j,m), where by convention V() represents the larger of the two variances, and v() the smaller, and (9,9) are the respective degrees of freedom. If the F-value proves to be not significantly different than 1, using once again the 1-a level of significance, one can pool the two variances into a single variance v(i,j,m)=[9v(i,E(j))+9v(i,S(j,m))/18 Then the student t-test statistic is t(18)=[A{i,E(j),k}-A{i,S(j,m),k}]/[v(i,j,m)*2/10].5 If |t(18)|<t(18|1-a/2), for each of the I characteristics, the simulant will be declared to be validated at the 1-(1-a)I. If, however, the F test of the two variances shows them to be significantly different, then the simulant does not exhibit the same properties as the explosive and should be rejected. In this validating approach each of the m simulants are considered independently even though some of them are simulating the same type of explosive. b. The Paired Difference Approach. Since the values of a{} come from the same bag with the explosive and its matching simulant in the same position in the bag, it is natural to consider the paired difference approach to test the correspondence between a simulant and the explosive. In this approach, one uses the differences d{i,j,m}=a{i,E(j),k}-a{i,S(j,m),k}, where the pairing over the 10 observations are randomly assigned. The population of such differences has an expected value, E(d{i,j,m})=0 if indeed the m-th simulant faithfully represented the j-th explosive for the i-th characteristic. An appropriate null hypothesis to test would be

OCR for page 37
Detection of Explosives for Commercial Aviation Security H(0): E(d{i,j,m})=0, against the two-sided alternative, H(1): E(d{i,j,m})=|= 0. Once again the m-th simulant could be accepted as being validated if the null hypothesis is not discarded at the a level of significance for all of the I characteristics. The test statistic to use in making this test is: t(9)=D(i,j,m)/[v(d(i,j,m].5 where D(i,j,m)=Sd(i,j,m)/10, taken over the 10 samples, and v(i,j)=S[d(i,j,m)-D(i,j,m)]2/9. Note that in using this test we have lost 9 degrees of freedom and as a result this test would be preferred only when there is a relatively high correlation between a{i,E(j),k} and a{i,S(j,m),k}. Here again, each simulant would be subject to its own validating decision. c. The Analysis of Variance Approach. It should be noted that d(i,j,m) as defined above, can be considered in an Analysis of Variance for each explosive type j. The Analysis of Variance approach provides a statistical test of the composite null hypothesis: H(0): E[D(i,j,m)]=C, for all i and m against the alternative, H(1): E[D(i,j,m)]=|=C, for some i or m. The Analysis of Variance table would simply be Analysis of Variance Source of Variation Sum of Squares Degrees of Freedom Mean Squares Among simulants SS (Simulants) m(j)-1 SM Within simulant (Experimental Error) SSE m(j)*(I-1) SP Total SST I*m(j)-1  

OCR for page 37
Detection of Explosives for Commercial Aviation Security Here SM=I*S(D(j,m)-D(j)2/(m(j)-1) SP=S(S(D(i,j,m)-D(j,m))2/m(j)*(I-1) and D(j,m)=SD(i,j,m)/I, D(j)=SSD(i,j,m)/m(j)*I We can then test the above hypothesis by selecting the level of significance a and assuming homogeneous variance, establish the critical region (reject H(0)) as F>F1-a(m(j)-1, m(j)*(I-1)) where F=SM/SP This test can then be applied for every one of the J explosives. Tests such as this are available on most computer's statistical packages. If the null hypothesis is accepted, then one needs to test the hypothesis that the constant C=0. One approach to this test is the use of the t-test. In using this test if the composite null hypothesis is not discarded all simulants of the j-th type explosive would be declared to be validated. However, if the null hypothesis is discarded, none of the simulants would be validated. There is an approach that allows one to also examine a wide variety of possible differences, including those suggested by the data itself. This test is based on the range of the m(j) observed means (we are looking at the difference between the smallest and the largest observed values). The reader is referred to Introduction to Statistical Analysis, Dixon & Massey, Second Edition, pages 152 to 155. d. Measurement Additivity. As was mentioned above only one bag is used to validate the simulants of a given explosive. This is acceptable if the measurement system is additive, that is, if the measurement of the characteristic of the explosive plus the measurement of the characteristic of the bag statistically equals the measurement of the explosive in the bag. If this is not the case, then the validation of the simulants will require testing over a representative set of bags, for each explosive threat. Hence one must first test for additivity of the measurements. This is done by taking measurements of the explosive in the absence of any background which would interfere with the measurement of the characteristics of the explosive. These measurements are repeated 10 times (as in the above). One then compares the sum of average reading of the explosive and the average reading of the bag (which is associated with the testing of that explosive, but not containing the explosive) with the average

OCR for page 37
Detection of Explosives for Commercial Aviation Security reading of the bag with the explosive. The above Analysis of Variance approach is suggested for testing for additivity of the measurements. 4. A Recommended Approach for Validating Simulants Repeated use of the Analysis of Variance approach appears to be appropriate. If in its initial use, none of the simulants are validated, individual t-tests should be run and the simulant which differed most significantly from its null hypothesis would be declared to be invalidated and dropped from the group of simulants being tested. The Analysis of Variance approach would then be repeated using the reduced group. This process would be repeated until a final reduced group of simulants would be declared to be validated. This is then repeated for all of the J explosives.

OCR for page 37
Detection of Explosives for Commercial Aviation Security This page in the original is blank.

OCR for page 37
Detection of Explosives for Commercial Aviation Security ANNEX III MODEL TESTING SCENARIO The following testing scenario is a model that could be used for the actual testing. For explanatory purpose only, it will be assumed that 20 threats are available to conduct the test. If passenger bags cannot be used, then it is suggested that for each of the bag populations, six groups of twenty bags be selected as representative of that population. If it is not possible to "reproduce" the distribution because an adequate number of bags is not available, a statistical weighting process could be used to "match" the distribution. After having selected the six groups of twenty bags, randomly order the bags in each group with the first group, called A1, A2, . . ., A20, the second group called B1, B2, . . ., B20, etc. To accomplish this, one can use a random number generator or a table. Next, the first 10 B bags are randomly intermingled with the first 10 A bags and the second 10 B bags with the second 10 A bags. Twenty threats are then randomly assigned to the 20 A bags. Each newly defined group of ten A and ten B bags are then processed through the equipment. Each time the group of twenty bags is processed, the threat location in the bag is changed by flipping and/or twisting the bag. Figure A2 shows the 15 possible locations in the bag and 9 locations that should be used since they represent all 15 locations. Figure A2. Threat Placement Locations

OCR for page 37
Detection of Explosives for Commercial Aviation Security The threats are then transferred to the B bags in the group and again processed through the equipment. In this manner, one is able to obtain measurements on A bags with threats and B bags without and on B bags with threats and A bags without, obtaining 9 observations on each bag. Then testing proceeds with C & D bags (using the same process-replace A with C and B with D) obtaining 9 observations on each combination. Finally testing is done with E & F bags (using the same process—replacing C with E and D with F) obtaining 9 observations on each combination. The total number of observations on empty bags is 1080 while the number of observations on bags with threats is 1080, with each threat being tested in 6 different bags, and in 9 locations in each bag. Before running the six groups of twenty bags through the equipment, two control bags should be processed 10 times, one bag without a threat and the second bag contained one of the threats. This set of two bags should be processed in a similar manner at various times during the tests and also at the end of each day. Comparing these measurements assure the test team that the equipment readings are not varying over the duration of the tests.