Attachment 2
Evaluating Testing, Costs, and Benefits of Advanced Spectroscopic Portals for Screening Cargo at Ports of Entry INTERIM REPORT
Executive Summary
To improve screening of containerized cargo for nuclear and radiological material that might be entering the United States, the Department of Homeland Security (DHS) is seeking to deploy new radiation detectors, called advanced spectroscopic portals (ASPs). The ASPs are intended to replace some or all of the current system of radiation portal monitors (called PVT RPMs) used in conjunction with handheld radioisotope identifiers (RIIDs) to detect and identify radioactive material in cargo. The U.S. Congress required the Secretary of Homeland Security to certify that ASPs will provide a “significant increase in operational effectiveness” over continued use of the existing screening devices before DHS can proceed with full-scale procurement of ASPs for deployment. Congress also directed DHS to request this National Research Council study to advise the Secretary of Homeland Security about testing, analysis, costs, and benefits of the ASPs prior to the certification decision. The objectives of this study are to: (1) evaluate the adequacy of the past testing and analyses of the ASP systems; (2) evaluate the scientific rigor and robustness of the testing and analysis approach; and (3) evaluate the cost-benefit analysis of ASP technology. Each of these is discussed below. This interim report is based on testing done before 2008; on plans for, observations of, and preliminary results from tests done in 2008; and on the agency’s draft cost-benefit analysis as of October 2008. The report provides advice on how DHS’ Domestic Nuclear Detection Office (DNDO) can complete and make more rigorous its ASP evaluation for the Secretary and the nation.
Testing: The committee finds that past testing had serious flaws. DNDO has acknowledged and addressed a number of those flaws in later testing. The 2008 performance tests were an improvement over previous tests: DNDO physically tested some of the limits of the systems, although shortcomings remain. DHS needs to address these shortcomings for a rigorous approach.
Scientific Rigor: To make the testing and evaluation more scientifically rigorous, the committee recommends an iterative approach with modeling and physical testing complementing each other. DNDO’s current approach is to physically test small portions of the threat space (possible threat and cargo configurations) and to use other experimental data to test algorithms in the systems. However, the set of combinations of threats and cargo configurations is so large and multidimensional that DNDO needs an analytical basis for understanding the capabilities of its detector systems. In a more rigorous approach, scientists and engineers would use models of
threat objects, radiation transport, and detector response to simulate performance and use physical experiments to validate the models’ fidelity and enable developers to refine the models iteratively. Much of the foundation for modeling sources, radiation transport, and detector response is already in place in the national laboratories. This kind of interaction between computer models and physical tests is standard for the development of high-technology equipment and is essential for building scientific confidence.
The idea of an iterative approach also extends to deployment: the committee recommends a process for incremental deployment and continuous improvement, with experience leading to refinements in both technologies and operations over time. As a first step in this process DHS should deploy its currently unused low-rate initial production ASPs for primary and secondary inspection at various sites to assess their capabilities in multiple environments without investing in a much larger acquisition at the outset.
Cost-Benefit: DHS’ definition of a “significant increase in operational effectiveness” is a modest set of goals. Preliminary estimates indicate that the cost increases from replacing the PVT/RIID combination with ASPs outweigh the cost reductions from operational efficiencies. Therefore, a careful cost-benefit analysis will need to reveal the advantages of ASPs among alternatives. The cost-benefit analysis was not complete when this report was written, but it should include three key elements: a clear statement of the objectives of the program; an assessment of meaningful alternatives; and a comprehensive, credible and transparent analysis of in-scope benefits and costs. The committee recommends that DHS not proceed with further procurement until it has addressed the findings and recommendations in this report and the ASP is shown to be a favored option in the cost-benefit analysis.
