Click for next page ( 45


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 44
4 Cost-Benefit Analysis A well-constructed cost-benefit analysis (CBA) is designed to provide insight about the effects of alternative decisions, whether the benefits of a given program exceed its costs, and which choices are most cost-effective. This provides a structure for analyzing whether a proposed action or program is reasonable and justified. As part of the study task, the committee was asked to evaluate the Domestic Nuclear Detection Office’s (DNDO’s) CBA of the advanced spectroscopic portal (ASP) technology. As of February 2009, the committee had not seen DNDO’s completed CBA, but was provided with briefings on the status of the CBA, most recently in October of 2008. At that point, much of the information in DNDO’s CBA was described to the committee as still in draft form. There is no single definitive approach to doing analysis of such complex cost-risk tradeoffs. The committee has chosen to provide suggestions to guide DNDO in completing its CBA, with the understanding that DNDO’s analysis will not be complete until after testing and technical evaluation are completed, in the hopes that they will help DNDO make the best case it can for each option, but recognizing that the result will still be subject to criticism. In addition, the committee hopes that carrying out these analyses will lead DHS to reexamine assumptions, practices, and objectives to create a firm foundation for continuing to improve. For example, DHS ought to consider whether the version of the DOE guidance on threat quantities and configurations that DNDO is using is operationally realistic and relevant to the threat and the nature of commerce. ASSESSING COSTS AND BENEFITS OF THE ASP PROGRAM There could be two bases for saying that a decision to procure ASPs might meet cost- benefit criteria: (1) It might lead to a reduction in net dollar costs of procurement, deployment, and operation, 37 or (2) it might increase significantly the likelihood of detecting threat materials and increase the deterrent value of the systems at a reasonable cost. The preliminary analysis presented to the committee by DNDO suggests that the former (criterion 1) likely is not the case for ASP deployments. FINDING Because DNDO’s preliminary estimates indicate that the cost increases from replacing the PVT/RIID combination with ASPs outweigh the cost reductions from operational efficiencies, it is important to consider carefully the conditions under which the benefits of deploying ASPs justify the program costs. 37 In standard accounting practices, costs that are assigned dollar values are associated with a relative time of expenditure for comparison and discounted appropriately to give net present value, or some other, similarly consistent discounting method is applied. It is the committee’s understanding that DNDO is following standard practice for discounting. 44

OCR for page 44
CHAPTER 4: COST-BENEFIT ANALYSIS 45 When, where, and how terrorists might attempt to use a nuclear weapon is unknown. This uncertainty makes it difficult to decide whether or not to invest in a system for detecting nuclear weapons. The consequences of a successful nuclear detonation by a terrorist group could be catastrophic. The potential magnitude of the consequences is a major factor in justifying investments to reduce the risks from nuclear scenarios. The risk of such scenarios depends also on how likely it is that they might occur. However, because the likelihood of such an attack cannot be precisely specified it is difficult to estimate the risk from nuclear weapons and the extent to which this risk is reduced through defensive countermeasures. Despite this uncertainty, a structured CBA can help to guide decision-making in such a situation. It is important to consider the standards used to measure ASP performance and whether meeting these standards is sufficient to warrant the program expenditure. A well-constructed cost-benefit analysis can aid in evaluating whether the criteria in DHS’s definition of a “significant increase in operational effectiveness” (shown in Sidebar 3.1) add benefits sufficient to justify the cost of the ASP program. FINDING DHS’ definition of “significant increase in operational effectiveness” is a modest set of goals: The increases in operational efficiency do not by themselves appear to outweigh the cost increases from replacing the PVT/RIID combination with ASPs, based on DNDO’s preliminary estimates, and the criteria do not require significantly improved ability to detect special nuclear material in primary screening (see Sidebar 3.1). If the ASPs meet the defined criteria and are able to detect the minimum quantities of nuclear threat material that DOE recommends (DOE guidance), DHS still will not know whether the benefits of the ASPs outweigh the additional costs associated with them, or whether the funds are more effectively spent on other elements of the Global Architecture. In particular, to determine whether the benefits outweigh the costs, the following issues need to be addressed:  The relative effect of the ASP system, relative to the existing PVT/RIID system, on reducing the probability that an adversary would try to smuggle nuclear material into the United States (deterrence);  The relative effect of the ASP system, relative to the existing PVT/RIID systems, on the probability that an adversary would succeed in smuggling nuclear material into the United States.  Whether any benefits identified in the above effects assessments and the improvements required by the SIOE criteria merit the cost of the improved technology deployment; RECOMMENDATION The CBA should provide a convincing narrative involving all relevant costs and benefits in order to justify spending funds on the ASP program. Many of the costs and particularly the benefits involved in threat detection systems such as the ASPs are not easily quantifiable. However, the CBA should provide a convincing

OCR for page 44
46 EVALUATING TESTING, COSTS, & BENEFITS OF ASPs: INTERIM REPORT narrative involving the relevant costs and benefits in order to justify spending funds on the ASP program. This narrative should provide qualitative justifications and explanations where quantitative justifications are not available. It may be difficult to justify a threat detection program based on only easily quantifiable benefits such as the benefits of reduced man-hours spent scanning cargo. However, a major benefit of threat detection programs is an increase in security, and careful consideration needs to be given to addressing these benefits in a thorough way. The committee provides some guidance to DNDO on addressing security benefits later in this chapter. In a structured CBA, several key elements need to be addressed thoroughly, quantitatively where possible, and qualitatively where necessary, as discussed above. These key elements have been defined by the United States Office of Management and Budget (OMB, 2003) as:  A clear statement of the objectives of the program, which would address what the ASP technology is meant to accomplish relative to the polyvinyl toluene (PVT) technology and how it fits into the rest of the Global Architecture (defined in Chapter 1).  An assessment of meaningful alternatives, which would address a full range of reasonable options and the benefits of the ASP program relative to these options, including a good baseline (typically a no-action alternative).  A comprehensive, credible and transparent analysis of benefits and costs as appropriate, which would address a full range of qualitative and quantitative benefits, including security benefits, as thoroughly as possible. In the following sections, the committee offers guidance to DNDO in these three areas. STATEMENT OF THE PROGRAM OBJECTIVES RECOMMENDATION A cost-benefit analysis should clearly define the ASP program objectives, including:  Describing the new and unique capabilities of the ASPs in the context of their role in the Global Architecture; and  Defining a realistic baseline alternative against which to compare the ASP deployment. A structured CBA begins with a clear description of the objectives of the program and the specific needs it is designed to meet. For the ASP program, the committee judges that the major issues to be addressed when stating the program objectives are:  Describing the unique and new capabilities of the ASPs that will enable them to meet the program objectives and clarifying their role in the Global Architecture; and  Defining a realistic baseline scenario, to give the program full credit for benefits and costs. The following sections discuss some specific recommendations from the committee for DNDO’s future work that concern a clear statement of the program objectives.

OCR for page 44
CHAPTER 4: COST-BENEFIT ANALYSIS 47 Describing new and unique capabilities and the role in the Global Architecture RECOMMENDATION The larger context of global security should be considered in the ASP program’s cost-benefit analysis. In particular, DHS should consider tradeoffs and interactions among different elements of the Global Architecture. Alternative approaches can be used to prevent the smuggling of nuclear materials into the United States. Some are alternative approaches to the cargo screening problem, while others are outside the program scope (such as prevention of smuggling via rail, aircraft, or small boats). It is important to show clearly the new and unique capabilities that deploying the ASPs will provide, relevant to stated goals and operational outcomes. In particular, this will involve considering how the ASP’s capabilities contribute to the larger context of the threat detection system intended to prevent nuclear and radiological threat material from entering the United States, known as the “Global Architecture.” Radiation portal monitors (RPMs) detect threat materials entering the United States in cargo containers on trucks via land border crossings and seaports, and constitute one piece of this system. The Global Architecture also encompasses screening for nuclear threat material brought across U.S. borders by plane, by personal watercraft, by rail, or that is transferred promptly to on-dock rail cars at seaports. In October 2008, DNDO presented the scope of its CBA as limited to the then-current deployment plans, which included ASP-C 38 (land and sea cargo) and ASP-D (the wide-load variant) but excluded rail deployments. 39 The committee had not seen any more recent information regarding the scope of DNDO’s CBA as of the writing of this report (February 2009). There are tradeoffs that need to be considered among the many programs that make up the Global Architecture. Given the limited resources available, investment used to strengthen the Global Architecture can be applied towards: (1) using different (newer) technologies to fill the same gap; or (2) filling different gaps, for example, different threats, different geographies, and different modes of transport. Furthermore, the preferred modes and routes of shipping and transportation are not static. Nor are threats. A more comprehensive evaluation of security benefits would factor in such trends For example, the enhanced capabilities provided by the ASP-C system are relevant to cargo containers entering the United States by truck, but not by rail. In the future, it is probable that less cargo will be brought directly into major U.S. seaports. Some of the fastest-growing ports in North America are in Canada and Mexico, 40 and it is expected that these ports will handle increasing amounts of cargo destined for the United States. Much of this cargo will be unloaded onto on-dock rail and will cross U.S. borders on rail. The tradeoffs among different spending options need to be considered in the ASP CBA or in a higher level analysis about allocation of efforts and funds. Indeed, it may be more appropriate for such an analysis to be carried out at a higher level so that it can provide guidance and support for multiple programs in a coordinated fashion. The committee has, however, seen no evidence that the higher level tradeoff analysis has been done, and a recent report by the GAO 38 Anything that is legal to drive on a road can pass through the ASP-C variant. 39 The ASP-D is considered a minor variant; however, DNDO informed the committee in October, 2008 that because the modifications needed to accommodate on-dock rail were significant, they would not be including on- dock rail as part of the program scope. 40 Some of the fastest-growing ports in North America include Manzanillo, Lazaro Cardenas, and Vancouver. For further information, see http://aapa.files.cms-plus.com/PDFs/North_American_Container_Traffic.pdf.

OCR for page 44
48 EVALUATING TESTING, COSTS, & BENEFITS OF ASPs: INTERIM REPORT (2009) tends to confirm this view. Furthermore, an ASP CBA should at the very least consider alternatives arising from work in the same office on the same mission (e.g., improved RIIDs). Defining a good baseline The benefits and costs of a program are defined in comparison with a clearly stated baseline, typically a “no-action” alternative. This is the type of baseline DNDO indicated to the committee that they had chosen as of October 2008: In this baseline, no ASPs would be deployed, and PVT deployment would be expanded. PVTs would continue to be used in both primary and secondary screening, and the handheld RIIDs would be used in secondary screening according to Custom and Border Protection’s (CBP) current Concept of Operations (CONOPS). It is important that the baseline option reflect the key features of the actual systems as they are deployed today, to ensure that costs and benefits are being accurately assigned to the ASP program. In considering DNDO’s baseline scenario, one must take into account several assumptions about the current operations of the PVTs to give the ASPs appropriate benefit for providing increased security. In this case, a good baseline might include:  Using the actual sensitivity settings of PVTs deployed in various ports; and  Correctly accounting for the range of densities of typical materials in containers. As of DNDO’s October 2008 presentation to the committee, the baseline alternative in the CBA presumed that the PVTs at all ports were set to the same detection level and operated identically. As discussed earlier in this report, the PVTs signal a “detection” when the observed radiation exceeds a given level. When this occurs, the conveyance is pulled aside for a secondary screening using a second PVT (the truck passes through the detector at a slower rate) and a hand-held RIID operated by a CBP officer. However, cargo containing NORM (for example, granite countertops or porcelain toilets) can set off these detectors, which are unable to distinguish between NORM cargo and threat materials. If the PVT is set more sensitively (the system alarms when it detects lower levels of radiation) then more NORM cargo is diverted to secondary screening than if the PVT is set less sensitively. If the alarm threshold is set too low (the system is too sensitive), the flow of commerce can be affected by the amount of NORM cargo sent to secondary screening. If the alarm threshold is set too high (the system is not sensitive enough) then material of concern could pass through without an alarm. More background regarding thresholds and sensitivity can be found in Chapter 2. To the extent that the ASPs allow detectors to be operated with a greater level of sensitivity, they would be expected to detect threat materials more reliably. In addition, the baseline needs to recognize and account for (to the extent possible) the range of material densities in typical containers that is brought into a given port. Threat material could be shielded by cargo brought into the United States in ordinary commerce. An operationally realistic range of material densities can then be used to define the range for comparative evaluation between the ASPs and the PVT/RIID systems. At present, the DOE threat guidance uses a single value for cargo density, and that value does not represent the upper limit of the range of typical cargo densities. Lower density cargo provides less shielding and therefore a less challenging detection problem.

OCR for page 44
CHAPTER 4: COST-BENEFIT ANALYSIS 49 ASSESSMENT OF A RANGE OF MEANINGFUL ALTERNATIVES RECOMMENDATION A cost-benefit analysis for the ASP program should demonstrate that a full set of meaningful alternatives has been assessed, including alternative deployments, operations, and technologies. A well-performed CBA demonstrates that a full set of meaningful alternative approaches to the proposed program has been assessed relative to the baseline. According to OMB Circular A-4 (OMB, 2003), a well-performed analysis “describe[s] the alternatives available and the reasons for choosing one alternative over another … [i]t is not adequate simply to report a comparison of the agency’s preferred option to the chosen baseline.” DNDO informed the committee in October of 2008 that three potential deployment plans for the ASPs are currently being considered, in addition to a baseline plan, discussed above. These three plans include: 1. Deployment in secondary: expand the deployment of PVTs in primary screening, and deploy ASPs in secondary screening; 2. Deployment in primary and secondary: deploy ASPs in primary screening and secondary screening; and 3. Hybrid deployment: For primary screening, deploy some primarily in high traffic ports, and retain PVTs in other, lower-traffic ports. For secondary screening, install ASPs at all ports. The committee was not shown assessments of other alternatives (apart from the baseline alternative) as of February 2009, although DNDO may have analyzed others. Increased security might be achieved without the deployment of ASPs, and a good CBA would clearly demonstrate that all reasonable possibilities have been assessed, or give reasons why these possibilities were not assessed. 41 Several alternative approaches are possible. The comparative costs and benefits of changes to CBP’s CONOPS could be assessed. For example, detection equipment could be deployed at every exit and CBP could select random nuisance alarms to be examined, potentially resulting in heightened deterrence effects; or secondary scanning times could be increased, providing time for high purity germanium (HPGe) detectors to perform Identifications. Another alternative could be to maintain current CONOPS, but deploy alternative technologies, such as improved RIIDs. The use of newly available RIIDs and associated software could improve the performance of secondary screening: newer, more sensitive models of RIIDs are available. Alternatively, software on the existing RIIDs could be improved. 42 This list is meant to be illustrative and not comprehensive. DNDO and CBP may have insights into alternatives of which the committee is not aware. At the same time, for practical reasons, the set has to be finite and relatively small, so the cases should be chosen carefully to represent the most promising alternatives. 41 In addition, note that CBP officers may prefer the use of handheld detectors, and some deployment of RIIDS may be needed even in the case that the ASPs are deployed in secondary. 42 For example, RIIDs using high purity germanium detectors are available. Improvements to RIIDs and to the associated software are discussed in more detail in Chapter 3 of this report.

OCR for page 44
50 EVALUATING TESTING, COSTS, & BENEFITS OF ASPs: INTERIM REPORT A COMPREHENSIVE, CREDIBLE, AND TRANSPARENT ANALYSIS OF BENEFITS AND COSTS RECOMMENDATION A cost-benefit analysis for the ASP program should include a comprehensive, credible and transparent analysis of benefits and costs, although the committee acknowledges that DNDO will not be able to perform a full, quantitative cost-benefit analysis. DNDO’s CBA is intended to provide guidance about whether the additional project costs associated with deploying the ASP systems are outweighed by the improvements in detection and other benefits. In October of 2008, DNDO presented a preliminary set of total life cycle cost estimates (LCCE) for the three ASP deployment scenarios outlined previously as well for a baseline scenario. However, the committee did not see a breakdown of these costs into categories such as design and development; procurement; deployment; maintenance; operations; or decommissioning, so it is unable to assess the validity of the projected costs. The approach that DNDO described to the committee for the ASP LCCE appeared to use a reasonable methodology. The committee did not see the details of this assessment; however, they suggest that it is essential to a valid CBA to supply uncertainties associated with the projected costs. The cost assessment should cover all phases of the acquisition life cycle in a manner that is independent of contractor or program office biases and assess the risk of cost escalation associated with the estimate. It is also possible that adoption of new technologies will lead to cost reductions, although this is less common in such procurements. As of the writing of this report (February 2009), DNDO had not yet presented an assessment of the security benefits of the ASP program. DNDO had considered some benefits, such as the ability for CBP officers to be reassigned to other missions, time saved in secondary screening, and a reduced number of conveyances referred to secondary screening. However, according to DNDO (2008b) these benefits alone are unlikely to justify the costs of the ASP program, and other, more difficult to quantify benefits of will need to be taken into account, including security benefits. This is a point on which DNDO has asked for specific advice from the committee. The committee recognizes the likely inability of the DNDO (or anyone else) to perform a full, quantitative cost-benefit analysis for the ASP program. Despite these difficulties, at minimum, a logical connection of the program effort to its goals needs to be presented. A well-performed CBA that helps in procurement decisions for ASPs is not going to be a simple analysis following standard formulae commonly used in other kinds of procurements. DNDO expressed its difficulty in assessing two of the cost-risk elements in the cost-benefit analysis with respect to equipment performance: assessing the probabilities of failure to detect threat material, and factoring in potential consequences of such a failure. There are four probabilities involved in analyzing security benefits: Pdetect, or the probability of detecting the threat material; Pidentification, or the probability of correctly identifying the threat material; Pinterdict, or the probability of interdicting the threat material; and Pencounter, the probability of an adversary attempting to smuggle threat material into the United States in the first place. The last probability is likely to be highly complex to evaluate, and indeed impossible to determine definitively. Although it is difficult to estimate such probabilities with high confidence, analysts need to

OCR for page 44
CHAPTER 4: COST-BENEFIT ANALYSIS 51 understand what they can about these probabilities based on analytical tools and input from the intelligence community and other sources.43 The consequences are likewise uncertain, although in this case because of indeterminacy. The consequences of a successful nuclear or radiological attack could be assessed within reasonable uncertainty bounds if the nature of the weapon, its location, and all of the environmental conditions, including where the people are (e.g., is it rush hour?), were specified. But in a site-generic, time-independent assessment, the variability in the possible consequences is very broad. All of these factors make it quite difficult to consider avoided events in the cost- benefit analysis. However, that difficulty does not make it less important to consider these factors. Security benefits can result from changes in any of the above probabilities. The benefits can take the form of higher detection, identification, and interdiction probabilities. They can also result from deflection (e.g. to overseas targets) or deterrence (effectively reducing the probability of encounter), so the security benefits associated with these factors also need to be considered. (Note, however, that deflection can push adversaries to use different avenues to the same target, as explained below.) The existence of some radiation monitoring at seaports and land border crossings may provide sufficient discouragement to potential adversaries from smuggling via this route. However, it may also increase the probability that the terrorist will focus on other gaps in the nation’s security that are identified as easier targets. For these reasons, benefits from increased detection probabilities may be modest as long as there are significant gaps in the Global Architecture. Improved detection can be expected to become more beneficial as those gaps are filled. There are several analytical approaches that may be useful to DNDO in performing an analysis of the security benefits of the different alternatives proposed for the ASP program. Below are three examples. Each of these options suffers a common shortcoming – they do not answer the question of whether the benefits of implementing ASP exceed the program’s costs. However, each in a different way can provide insights that could help the Secretary weigh the merits of acquiring and deploying ASPs or alternative nuclear detection technologies. A capability-based planning approach would provide a structured assessment of how alternative detection technologies or deployment strategies reduce the risk of a nuclear detonation in the United States. This approach has been applied in defense applications to compare and contrast a set of options for approaching a given operational challenge across a wide range of circumstances (Davis 2002). In the case of ASP, the operational challenge is to prevent a nuclear or radiological attack by detecting, identifying and interdicting materials or a weapon smuggled into the United States. The set of options could include alternative deployments of ASPs and PVTs, deployment of alternative RIIDs, or (depending on the scope of analysis) shifting emphasis between port-of-entry (POE) and non-POE detection. The circumstances considered would include relevant dimensions of the adversary capability and tactics, the operating environment, and the technologies themselves. For example, circumstances could include different types of nuclear materials, use of different shielding or masking methods, technology performance now or in the future, different operating environments, different numbers of weapons available to the adversary, or failure of primary and secondary inspection 43 Other parts of DHS have conducted expert elicitations of threat probabilities with members of the intelligence community. A recent report by the National Research Council cautioned about limitations of this approach for the bioterrorism risk assessment. The committee, however, sees value in factoring in what the intelligence community knows and suspects, accounting at the same time for the confidence that can be placed in that knowledge.

OCR for page 44
52 EVALUATING TESTING, COSTS, & BENEFITS OF ASPs: INTERIM REPORT due to a common source of failure. Using a model that reflects the detection systems performance, the alternatives are then compared across multiple metrics appropriate for these circumstances (e.g., probability of detection of a representative threat object. The strength of a capabilities-based planning approach is that it can provide a rich comparison of the security benefits emphasizing the circumstances under which each option might be preferred. The weakness of this approach is that exploring the circumstances that affect the systems capability can quickly lead to large and complex analysis and judgment of the analysts is required to balance this complexity against and ability to draw salient insights about a systems capabilities. Game theory could provide insight into the benefits from deterrence associated with the PVTs and ASPs. Studies in other areas have found that the simple presence of security can significantly deter individuals from choosing the action that the security is meant to protect against. For example, Ayres and Leavitt (1998) used game theory to predict that the ability of a criminal to observe security measures affects the deterrent ability of that security measure and validated this prediction with observations from vehicle theft statistics: An increase in the percentage of lojack-equipped 44 vehicles in a given area is associated with a substantial decline in auto theft. 45 In contrast, observable security devices against car theft tended to merely shift the risk of theft to other vehicles, but not lower overall rates of theft. In another example of game theory being used for security policy, Kunreuther (2005) demonstrated the public policy opportunity that exists in commercial aviation because of tipping effects that would lead to mass adoption of baggage screening technologies under the right policy incentives. Researchers who have used game theory to assess protection of critical infrastructure from terrorism are just beginning to explore the utility of these approaches to decisions about radiation portal monitors (Bier and Azaiez 2009; Dighe et al. 2009), as noted earlier. In the context of ASP procurement, one would have to look at the incremental benefit of installing new detectors. The general weakness of game theory is that if analysts are unable to estimate parameters of their model, they are only able to draw broad and conditional conclusions about adversary behavior. Examples of parameters from security applications that are difficult to estimate include the value to the adversary of different outcomes (i.e., what might constitute successes and what are the costs of being caught), the probability of attack, and the costs of the defender falsely suspecting an attack is occurring. In the absence of being able to establish values for model parameters, broad or conditional conclusions might or might not provide actionable advice to policymakers. 46 Finally, cost-effectiveness analysis and break-even analysis are related approaches that have been used to assess costs and benefits when performing a complete cost-benefit analysis is difficult or impossible. Cost-effectiveness analysis is used when valuation of benefits is contentious and it is not possible to quantify benefits in monetary terms. Because the security goals of the ASP program may be difficult to value monetarily, comparing program alternatives using cost-effectiveness measures such as dollars per life saved or dollars per attack avoided could provide insights into their relative merits. In contrast, break-even analysis can be used 44 Lojack is a hidden radio-transmitter device used for retrieving stolen vehicles. 45 This occurs without a drop in other types of crimes. (Ayres and Leavitt, 1998) 46 Consider the case of deterrence. Partial screening (screening a fraction of the total number of containers entering the United States) may provide for effective deterrence (or deflection), if detection probabilities are sufficiently high and if smugglers cannot predict which containers will not be screened. This benefit quickly evaporates if adversaries are able to stage several smuggling attempts simultaneously because the chance of at least one attempt succeeding grows rapidly with the number of attempts (Bier and Haphuriwat, 2009). Like other game theoretic analyses, this enters a psychological realm, ascribing logical values to the adversary (e.g., that the threat material is a scarce and valuable asset and that the risk of discovery at a port of entry is not desired).

OCR for page 44
CHAPTER 4: COST-BENEFIT ANALYSIS 53 when it is difficult to even assess the magnitude of benefits in any units. Break-even analysis determines conditions that must be met for benefits to exceed costs. In security applications, these conditions could be a required reduction in overall risk (see Willis and LaTourrette 2008 analysis of the Western Hemisphere Travel Initiative) or a baseline estimate of a threat of attack that exists (see Martonosi et al. 2006 analysis of 100% container inspection). In some instances, framing the problem in this manner has proven useful because it also can provide a simple, yet fully parameterized model of the system being evaluated which a policymaker can explore to understand conditions that must be met for benefits to exceed costs (see von Winterfeldt and O’Sullivan analysis of acquisition of MANPADs defenses on commercial aviation). In cases where break-even analysis identifies meaningful bounds on decisions, that is threshold conditions that can easily be judged to exist or not exist, this approach can simplify decisionmaking. The downfall of break-even analysis is that these conditions do not always exist. The committee reiterates that methods for evaluating security benefits, examples of which are provided above, can provide different insights based on their approach, and none is likely to provide fully quantitative and definitive results. But most policy decisions are made without fully quantitative and definitive results, so DNDO should incorporate these benefits to provide the most informative CBA it can. The committee recommends that DHS not proceed with further procurement until it has addressed the findings and recommendations in this report and the ASP is shown to be a favored option in the cost-benefit analysis.