Click for next page ( 4

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 3
Summary Containerized cargo entering the United States at sea ports and land-border crossings for trucks is currently screened for radiation using detectors, called radiation portal monitors (RPMs) made from a plastic scintillator, called PVT, 1 in conjunction with handheld radioisotope identifiers (RIIDs). The Department of Homeland Security (DHS) is seeking to deploy new radiation detectors, called advanced spectroscopic portals (ASPs), to replace the PVT and RIID combination, which has known deficiencies. Title IV of division E of the Consolidated Appropriations Act, 2008 (Public Law 110-161) requires the Secretary of Homeland Security to submit to Congress a report certifying that a “significant increase in operational effectiveness” over continued use of the existing screening devices will be achieved with the ASP before “funds appropriated under this heading shall be obligated for full-scale procurement of Advanced Spectroscopic Portal Monitors.” DHS is testing and evaluating the ASPs to inform the Secretary’s certification decision. If the Secretary certifies the ASPs, DHS may purchase more than one billion dollars worth of ASPs. The net lifecycle cost of these ASPs could be more than twice that figure. The U.S. Congress directed DHS to request that the National Research Council of the National Academies conduct a study prior to certification to: (1) evaluate the adequacy of the past testing and analyses of the ASP systems performed by DHS’s Domestic Nuclear Detection Office (DNDO); (2) evaluate the scientific rigor and robustness of DNDO's current testing and analysis approach; and (3) evaluate DNDO's cost-benefit analysis of ASP technology. Due to delays in the test and evaluation program, the Academies and DHS agreed that the study committee would issue an interim report that provides the committee’s evaluation of testing plans and execution it has seen, and advice on how DNDO can complete and make more rigorous its ASP evaluation for the Secretary and the nation. This interim report is based on testing done before 2008, plans for and preliminary results from tests done in 2008, and the agency’s draft cost-benefit analysis as of October 2008. The committee received briefings on the performance test results and analysis and on the cost-benefit analysis, but the committee did not receive written reports on those topics by February 2009, when the interim report entered the Academy peer review process. The committee addresses each element of the study task below. PAST PERFORMANCE TESTING Performance tests prior to 2008 had serious flaws that were identified by the Government Accountability Office and the Secretary’s ASP Independent Review Team. All truck-conveyed containers at ports and border crossings pass through a PVT portal which constitutes primary screening, and those trucks that trigger an alarm are sent to secondary screening, which is conducted with a PVT portal and RIID. The tests prior to 2008 did not adequately assess the capabilities of the ASP systems in primary and secondary screening compared with the currently deployed PVT and RIID screening systems, nor whether the ASP systems met criteria for procurement. DNDO utilized the same sources in performance testing that were used to set up and calibrate this testing. The number of sources available was small, but this is not sufficient 1 PVT stands for polyvinyl toluene. 3

OCR for page 3
4 EVALUATING TESTING, COSTS, & BENEFITS OF ASPs: INTERIM REPORT reason to use the same sources for both set up and testing. Device setup and any calibration must use separate sources from those used for testing. A component of the standard operating procedures for the RIIDs in secondary screening was not followed in the performance tests, which disadvantaged the RIID in comparisons with ASPs. 2008 PERFORMANCE TESTING In describing and discussing the tests with the committee, DNDO staff acknowledged several pre-2008 deficiencies. According to the 2008 test plan and briefings to the committee in Washington, D.C., and at the Nevada Test Site, these deficiencies were corrected. This is consistent with the committee’s observations of tests and questioning of test personnel. Because they have large detectors and because of their configuration, ASPs would be expected to improve isotope identification, and provide greater consistency in screening each container, greater coverage of each container, and increased speed of screening over that of the PVT/RIID combination when used in secondary screening. Consequently, tests of ASPs in secondary screening focused on confirming and quantifying that advantage for several threat objects, cargos, and configurations. When used for primary screening, an ASP system must be compared to the existing combined primary and secondary screening system (both PVT and RIID) because of differences in standard operating procedures for primary screening (ASPs in primary have an identification function). DNDO’s preliminary analysis did account for this difference. The 2008 performance tests were an improvement over previous tests. DNDO physically tested some of the limits of the systems. However, the following shortcomings remain. (1) Without modeling to complement the physical experiments, the selected test configurations are too limited; (2) the sample sizes are small and limit the confidence that can be placed in comparisons among the results; and (3) in its analysis, some of the performance metrics are not the correct ones for comparing operational performance of screening systems. These shortcomings are described in greater detail within the report. For these reasons, DHS cannot conclude definitively whether ASPs will consistently outperform the current PVT-RIID systems in routine practice until the shortcomings are addressed. Better measurement and characterization are a necessary first step but may not be sufficient to enable DHS to conclude that the ASPs meet the criteria DHS has defined for achieving a “significant increase in operational effectiveness.” The committee recommends modifications to the current DHS approach to the evaluation procedure. These modifications would influence subsequent procurement steps. RECOMMENDED APPROACH FOR TESTING AND EVALUATION To make the testing and evaluation more scientifically rigorous, the committee recommends an iterative approach with modeling and physical testing complementing each other. The threat space—that is, the set of possible threat objects, configurations, surrounding cargoes, and conditions of transport—is so large and multidimensional that DNDO needs an analytical basis for understanding the capabilities of detectors for screening cargo. DNDO’s current approach is to physically test small portions of the threat space and to use other experimental data to interpolate and extrapolate throughout the threat space to test the identification algorithms in the detector systems.

OCR for page 3
SUMMARY 5 For a more rigorous approach, DNDO should use theory and models of threat objects, radiation transport, and detector response to simulate performance and predict outcomes. Then DNDO can use physical experiments to validate the predictions and allow a critique of the models’ fidelity to reality. This would enable developers to refine the models iteratively. With validated models, DNDO can evaluate the performance of the ASP systems over a larger, more meaningful range of cases and threat space than is feasible with physical tests alone. This kind of interaction between computer models and physical tests is standard for the development of some high-technology equipment and is essential for building scientific confidence. The performance tests conducted in 2008, and even prior to 2008, can be used to help refine and validate models. RECOMMENDED APPROACH FOR THE PROCUREMENT PROCESS The idea of an iterative approach extends to deployment, too. The committee noted that DHS’s testing philosophy is oriented toward a one-time certification decision in the near future. However, the mandate for passive radiation screening of cargo at ports of entry is expected to continue indefinitely. Rather than focusing on the single decision about the deployment of ASPs, the current testing should be viewed as a first step in a continuous process of improvement and adaptation of the systems. The threat environment, the composition of container cargo, technological and analytical capabilities, and the nature of commerce at the ports of entry have changed significantly over the last decade and can be expected to evolve in both predictable and unpredictable ways in the coming years. DHS should develop a process for incremental deployment and continuous improvement, with experience leading to refinements in both technologies and operations over time, rather than a single product purchase to replace current screening technology. The process should be developed to address and exploit changes. This would result in a system that can be adapted and updated continuously so that it would not be outdated by the time all of the ASPs are deployed. As the first step in this process DHS should deploy its currently unused low-rate initial production ASPs for primary and secondary inspection at various sites as extended operational testing. Such deployment, even on this limited scale, would provide additional data concerning their operation, reliability, and performance, and allow DHS to better assess their capabilities in multiple environments without investing in a much larger acquisition at the outset. The development of the hardware for radiation detection and the software for analyzing the signals from the detectors is separable. It has been useful to have a competitive approach for the combined systems and to see the results. However, as DHS moves forward, it should match the best hardware to the best software (particularly the algorithms), drawing on tools developed for the competition and elsewhere, such as the national laboratories. ASPs will not eliminate the need for handheld detectors with spectroscopic capabilities. Because some of the improvement in isotope identification offered by the ASPs over the RIIDs is a result of software improvements, the best software package also should be incorporated into improved handheld detectors. Newer RIIDs with better software might significantly improve their performance and expand the range of deployment options available to CBP for cargo screening. By separating these elements and engaging the broader science and engineering community, DHS would have increased confidence in its procurement of the best product available with current technology, and simultaneously could advance the state of the art.

OCR for page 3
6 EVALUATING TESTING, COSTS, & BENEFITS OF ASPs: INTERIM REPORT RECOMMENDED APPROACH FOR COST-BENEFIT ANALYSIS The preliminary analysis presented to the committee suggests that benefits of deploying the ASPs may not be clearly and undeniably greater than the costs. Because DNDO’s preliminary estimates indicate that the cost increases from replacing the PVT/RIID combination with ASPs outweigh the cost reductions from operational efficiencies, it is important to consider carefully the conditions under which the benefits of deploying ASPs justify the program costs. A cost-benefit analysis (CBA) can provide a structure for evaluating whether a proposed program (such as the ASP program) is reasonable and justified. The Secretary’s decision on ASP certification is to rely, at least in part, on whether the ASPs meet the objectives in DHS’ definition of “significant increase in operational effectiveness” (SIOE); however, other factors relating to the costs and benefits of the proposed ASP program will also need to be taken into account. DHS’ definition of a SIOE is a modest set of goals: As noted above, the increases in operational efficiency do not by themselves appear to outweigh the cost increases from replacing the PVT/RIID combination with ASPs, based on DNDO’s preliminary estimates, and the criteria do not require significantly improved ability to detect SNM in primary screening (see Sidebar 3.1). If the ASPs meet the defined criteria and are able to detect the minimum quantities of nuclear threat material that DOE recommends (the “DOE guidance”), DHS still will not know whether the benefits of the ASPs outweigh the additional costs associated with them, or whether the funds are more effectively spent on other elements of the Global Architecture. A CBA can provide insight about the effects of alternative decisions, whether the benefits of a given program exceed its costs, and which choices are most cost-effective. To do this, the cost-benefit analysis needs to include three key elements: (1) a clear statement of the objectives of the screening program; (2) an assessment of meaningful alternatives to deploying ASPs; and (3) a comprehensive, credible and transparent analysis of in-scope benefits and costs. The CBA should begin by stating clearly what operational problem the ASPs are intended to address. This statement will define the role that the system plays in providing a layer in the defense against the importation of a nuclear or radiological device. It should include a narrative that clarifies how the task of improving detection for containers at ports of entry to the United States fits into a larger effort to implement or improve detection capabilities, in recognition of the many ways that materials could be brought into the United States through ports of entry that are not already screened, or across uncontrolled stretches of border. Furthermore, to be useful in a procurement decision, a CBA will need to address whether funds are better spent to replace the currently deployed equipment rather than to expand coverage to other pathways that currently have no radiation screening. This is needed in the ASP CBA because it is not evident that it has been done elsewhere. The CBA needs to account for meaningful alternatives (including non-ASP programs) to reveal the scale of the benefits of ASPs for radiation screening and determine whether these benefits outweigh the additional costs. The complexity of the container screening task provides opportunities for many different options worthy of consideration. These options include variations on deployment configuration and operational processes, and application of technologies beyond the PVT/RIID and ASP detectors such as improved versions of existing handheld passive detectors (deploying handhelds with state-of-the-art software) and advanced

OCR for page 3
SUMMARY 7 methods for detecting nuclear materials. Considerations should include active interrogation, improved imaging systems, and integration of existing technologies. These alternatives need to be compared to a baseline that reflects as realistically as possible the screening capability that DHS currently has in place. Thus, the baseline should reflect the number and placement of PVT and RIID detectors, sensitivity of the sensors based on how they are operated at each port, and performance of existing handheld detectors in the manner they are used in the field. Such an analysis would indicate what capability an investment in ASPs will provide beyond the existing systems as they are currently deployed and operated or beyond alternative technologies that could be developed and deployed for radiation detection. In comparing these alternatives, it is important that the cost-benefit analysis treat benefits and costs in a comprehensive, credible, and transparent manner. The benefit assessment should show how this program contributes to improving security with respect to prevention of the detonation of a nuclear device or radiological weapon in the United States. Because this is the primary objective of the ASP program, a cost-benefit analysis that is silent on this subject would be incomplete. Such an assessment is difficult and no assessment of such benefits will be definitive or unassailable, however it remains important to consider these factors. The cost assessment should cover all phases of the acquisition life cycle in a manner that is independent of contractor or program office biases and assess the risk of cost escalation associated with the estimate. The committee recommends that DHS not proceed with further procurement until it has addressed the findings and recommendations in this report and the ASP is shown to be a favored option in the cost-benefit analysis.