This report describes merits, deficiencies, and options for improving testing, evaluation, and analysis of costs and benefits of advanced spectroscopic portals (ASPs). Specifically, the report addresses the Domestic Nuclear Detection Office’s (DNDO’s) 2008 performance tests, its characterization of results of the tests, and the scope and implementation of DNDO’s draft cost-benefit analysis, as well as deployment of ASPs.
Testing The design and evaluation of DNDO’s 2008 ASP performance tests have shortcomings that impair DHS’ ability to draw reliable conclusions about the ASP’s likely performance. The physical tests were not and have not been structured as part of an effort using modeling (computer simulations) and physical tests to build an understanding of the performance of the ASPs against different threats over a wide range of configurations and operating environments, as was suggested in the committee’s interim report.
Evaluation In characterizing and evaluating the results of the tests comparing the relative performance of the ASP and the handheld radioisotope identification device (RIID), DNDO’s analysis used a figure of merit that is not technically meaningful and could be misleading. The committee recommends that DNDO use the more particularized results from its report to create a different figure of merit and suggests some options.
Costs and Benefits The estimated net cost of ASPs exceeds that of the existing polyvinyl toluene radiation portal monitors (PVT RPMs) and RIIDs, so it would make sense to procure ASPs only if the security benefits justify the additional investment. In its draft cost-benefit analysis, DNDO carried out both a breakeven analysis and a capabilities-based plan to account for security benefits from ASPs, but the DNDO draft analyses the committee examined still need substantial improvement to support decision making. Three major problems remain: (1) The strategic justification for the chosen alternative or preferred option was not provided; (2) the set of alternatives analyzed is too narrow; and (3) DNDO used quantitative modeling techniques and therefore quantified factors that could not be justifiably quantified, when the analysis could have been carried out effectively with qualitative reasoning.
With respect to the narrow alternatives, DNDO followed a suggestion in the committee’s interim report, examining the effect of using improved software and algorithms in conjunction with the current handheld RIIDs used in secondary inspection. The results show dramatic improvements, such that the performance of the RIIDs with a state-of-the-art algorithm could outperform the tested ASP systems (2008 hardware and software configurations) in some cases, although they were still poorer in other cases. There are drawbacks to using handheld detectors for external screening of cargo containers, but this low-cost option, which substantially increases scanning effectiveness, should be an alternative in the cost-benefit analysis, and it might ultimately prove to be the preferred option.
Deployment The committee previously recommended an incremental approach to deployment, exploiting the modularity required in the ASP product specifications to match the best hardware with the best data-analysis algorithms and to upgrade as experience is gained with the system. It appears that DNDO has not gotten the modularity from the vendors that was mandated in the specification. This deficiency should be corrected and DNDO should encourage a broader effort to improve data-analysis algorithms, additionally engaging experts outside of the very small community of researchers engaged to date.