4

APPLICATION OF OTHER RISK METHODOLOGIES

This report has reviewed the current assessment methods that inform policy makers and decision makers on technical issues related to proliferation risk of a given nuclear fuel cycle. Case-by-case analyses conducted by multidisciplinary teams of subject matter experts are most frequently used to address a variety of technical issues related to host-state proliferation risk (see Chapter 2). Predefined framework methodologies have been used to evaluate the proliferation resistance rather than proliferation risk of nuclear fuel cycles (see Chapter 3). These types of methodologies focus on the technical details of fuel cycle processes and have not considered factors of specific countries or been used to evaluate the probability that a host state would choose to proliferate using a particular fuel cycle technology, nor have they been used to evaluate the consequences of a successful attempt. In addition to these limitations on scope, the committee identified a number of deficiencies in the execution of predefined frameworks (Finding 2.1). This chapter focuses on the third task:

TASK 3: Assess the potential for adapting risk assessment methodologies developed in other contexts (such as safety and security) to host-state proliferation risk assessments—including both qualitative and quantitative approaches—their benefits, limitations, and the challenges associated with adapting these methodologies to proliferation risk assessment.

We consider whether other risk assessment methodologies could be useful in evaluating the larger question of proliferation risk and whether their established practices might address the noted deficiencies of the predefined frameworks. The chapter is split into three sections. The first section considers the most challenging component of the proliferation risk problem: the analysis of host-state factors. The next section focuses on probabilistic risk assessment (PRA) and how it has contributed to other engineering-based problems. The chapter concludes with identification of benefits and limitations of PRA and other approaches to the problem of proliferation risk assessment.

In addressing this task, the following types of risk assessment methodologies were considered by the committee: adaptive adversary models (including PRA and game theory), quantitative methods proposed by social and political science approaches, and nuclear safety risk assessments (using PRA).



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 41
4 APPLICATION OF OTHER RISK METHODOLOGIES This report has reviewed the current assessment methods that inform policy makers and decision makers on technical issues related to proliferation risk of a given nuclear fuel cycle. Case-by-case analyses conducted by multidisciplinary teams of subject matter experts are most frequently used to address a variety of technical issues related to host-state proliferation risk (see Chapter 2). Predefined framework methodologies have been used to evaluate the proliferation resistance rather than proliferation risk of nuclear fuel cycles (see Chapter 3). These types of methodologies focus on the technical details of fuel cycle processes and have not considered factors of specific countries or been used to evaluate the probability that a host state would choose to proliferate using a particular fuel cycle technology, nor have they been used to evaluate the consequences of a successful attempt. In addition to these limitations on scope, the committee identified a number of deficiencies in the execution of predefined frameworks (Finding 2.1). This chapter focuses on the third task: TASK 3: Assess the potential for adapting risk assessment methodologies developed in other contexts (such as safety and security) to host-state proliferation risk assessments—including both qualitative and quantitative approaches—their benefits, limitations, and the challenges associated with adapting these methodologies to proliferation risk assessment. We consider whether other risk assessment methodologies could be useful in evaluating the larger question of proliferation risk and whether their established practices might address the noted deficiencies of the predefined frameworks. The chapter is split into three sections. The first section considers the most challenging component of the proliferation risk problem: the analysis of host-state factors. The next section focuses on probabilistic risk assessment (PRA) and how it has contributed to other engineering- based problems. The chapter concludes with identification of benefits and limitations of PRA and other approaches to the problem of proliferation risk assessment. In addressing this task, the following types of risk assessment methodologies were considered by the committee: adaptive adversary models (including PRA and game theory), quantitative methods proposed by social and political science approaches, and nuclear safety risk assessments (using PRA). 41

OCR for page 41
42 APPLICATION OF OTHER RISK METHODOLOGIES ASSESSING HOST-STATE PROLIFERATION RISK Evaluation of proliferation risk of a fuel cycle deployed in a particular host state is a more complex concept than proliferation resistance and includes analysis of country- specific issues such as the probability of host-state proliferation (including its intent, motivations, and technical capabilities) and the consequences of proliferation. Proliferation is an act carried out by a motivated host state, not a failure of an engineered system. Because the process of proliferation generally occurs in secret, detailed data on the factors relevant to the probability that a host state would choose to proliferate along a particular path or the probability of success along that pathway are not readily available. In addition, factors influencing proliferation risk will change over time, for example, a change in regime or in perceptions of regional security. Choices about proliferation pathways also could evolve over time, based on the ability to overcome safeguards and avoid detection, as well as advances in technical capabilities or the ability to acquire technology covertly. The resources a host state might allocate to achieving success would depend on the strength of its motivation, which in turn would depend on its perception of the benefits of nuclear weapons. Because of the limited number of cases of proliferation in general, and the secrecy of the process, data on all these factors are very limited, and trends that could guide a technical analysis of probability have been difficult to determine. Historical case studies provide some relevant data about countries that have given up nuclear weapons programs in the past, but limited data are available on successful efforts. Close cooperation with inside experts familiar with successful proliferation efforts (either in the past, or ongoing) is unlikely. Several National Research Council reports have addressed the issue of whether and how risk assessment could be applied in contexts where, unlike traditional risk assessments for natural hazards, action by intelligent adversaries gives rise to the risk. A 2010 report on the use of risk assessment for terrorism concluded: However, with the exception of risk analysis for natural disaster preparedness, the committee did not find any DHS risk analysis capabilities and methods that are yet adequate for supporting DHS decision making, because their validity and reliability are untested. Moreover, it is not yet clear that DHS is on a trajectory for development of methods and capability that is sufficient to ensure reliable risk analyses other than for natural disasters. (NRC 2010, p. 80) Similarly, a more recent report (NRC 2011) concluded that, because adversaries may be intelligent, creative, and adaptive, a basis for assigning probabilities of attack has not been established. A somewhat more optimistic view of the potential for using risk assessment in the context of intelligent adversaries was taken in Department of Homeland Security Bioterrorism Risk Assessment: A Call for Change (NRC 2008). This report includes a table titled “Natural Hazards Versus Terrorism Risks: Comparison of Key Characteristics” and makes the observation that:

OCR for page 41
APPLICATION OF OTHER RISK METHODOLOGIES 43 When dealing with an intelligent, goal-oriented, and resourceful adversary, not with a force such as nature that randomly determines whether unwanted events occur, this committee believes that the use of probabilities to represent bioterrorism decisions must be tempered by a thorough understanding of how these probabilities have been assessed (whether by means of formal game-theoretical models, elicitation of subject-matter experts, or other means). For decision problems as complex as those motivating BTRA [bioterrorism risk assessment], the assessment of the probabilities that adversaries will choose courses of action should be the outputs of analysis, not required input parameters. The BTRA has reversed this preferred approach by requiring that subject-matter experts predict, a priori, how adversaries will behave. For this approach to make sense, the subject-matter experts must grasp nuances of alternatives and outcomes and render opinions founded on an analysis of the entire decision process, which would be very difficult for a process this complex. The committee saw no evidence that this level of analysis was used. Moreover, the static probabilities used are not appropriate when intelligent adversaries can observe and react dynamically to any earlier decisions made by the United States. (NRC 2008, p. 27) The committee considered risk analysis approaches proposed for situations with intelligent adversaries, including those based on game theory, but concluded that although such approaches appear promising, their effectiveness has not been demonstrated via evidenced-based records of success in real-world situations. The committee is aware of risk-based approaches for assessing other security- related problems. The committee learned about Department of Homeland Security’s PRA approach (Streetman 2012) for modeling an adaptive adversary. Modeling an adaptive adversary is an area of longstanding interest to the intelligence community, but the field is not yet mature, and its potential for success is a matter of debate. There are also efforts to use PRA as an analytical tool for assessing the risk of terrorism, but these remain controversial (NRC 2010). We were unable to identify successful applications of PRA in the security context, possibly because of classification issues.19 As noted above, predefined framework assessments have not included country- specific factors such as compliance with nonproliferation norms and obligations, historical interest in nuclear weapons, or possible motivations to develop nuclear weapons. Some have suggested that value could be added by combining these predefined framework methodologies with results of political science methodologies that seek to establish correlations between country-specific factors and proliferation. To evaluate this suggestion, the committee reviewed selected literature and held a focused meeting on social science–based research to understand factors that may influence a country to pursue nuclear weapons, including some efforts to develop models of how factors such as security concerns, type of government, and technical capability influence decisions (Coles et al. 2009, Gartzke 2012, Way 2012). 19 The fact that we could not identify such cases does not mean they do not exist.

OCR for page 41
44 APPLICATION OF OTHER RISK METHODOLOGIES Political science research using both detailed historical case studies and statistical methods has produced four main findings. First, historically, there are examples of states (fewer than 20) that have either tentatively explored or actively pursued a nuclear weapons program but have abandoned their efforts before successful development (Singh and Way 2004; Sagan 2011). Second, there is a positive correlation between the level of security threat that senior state leaders perceive and their subsequent level of nuclear weapons interest. States that face enduring rivalries with nuclear-armed states are more likely to explore getting their own nuclear deterrent than are other similar states without such rivals, although this proliferation interest can be reduced if the state has a security guarantee with another nuclear weapons state (Jo and Gartzke 2007). Third, the majority of states that are non–nuclear weapons state (NNWS) members of the Nuclear Non- proliferation Treaty (NPT) have neither explored nor pursued nuclear weapons since they joined the treaty. Indeed, one study estimates that of the 184 NNWSs in the NPT, fewer than 10 have cheated on their treaty commitment not to seek or acquire nuclear weapons after they joined the treaty. Fourth, although both democracies and autocracies have developed nuclear weapons and started nuclear programs historically, only autocratic governments have started nuclear weapons programs while they were forbidden to do so under their NPT commitments (Sagan 2011, Way and Weeks 2012). These insights from the political science literature are consistent with considerations of nonproliferation decision makers and multidisciplinary teams performing systematic technical assessments. However, these methods do not ameliorate the challenges of lack of evidence about decisions to proliferate, the most likely pathways, or consequences of proliferation. PROBABILISTIC RISK ASSESSMENT The committee considered how PRA methodologies might be applied to the problem of assessing proliferation risk. We reviewed the underlying principles of PRA, as well as examples of their application to evaluate risks in a range of fields including nuclear power plants, space exploration, chemical munitions cleanup, and natural disasters. A PRA starts with a definition of the full system under consideration and its desired state. Next, it identifies and characterizes threats to the system and develops scenarios for “what can go wrong” (see Box 4.1). The scenarios are then quantified in terms of their likelihood and consequences. Structuring the scenarios is a complex undertaking and requires evidence to support both the listing of threats, the system response to the threats, and the likelihood of any particular event occurring. Evidence is also needed to quantify consequences. Evidence can be provided by either physical data or expert judgment. PRA methodologies are particularly good at dealing with incomplete information or uncertainties. Incomplete information is addressed by assigning probability distributions based only on the evidence that is available. However, a comprehensive understanding of the system is essential to performing a PRA. Without detailed knowledge of how a system works and responds to threats, no amount of probabilistic analysis will answer the risk question. The thought process of PRA is valuable as a tool to guide analysis and collect data (see Box 4.1). PRA experts have significant and relevant experience identifying data

OCR for page 41
APPLICATION OF OTHER RISK METHODOLOGIES 45 BOX 4.1 Practitioners’ Thoughts on Probabilistic Risk Assessment In evaluating the applicability of PRA to the problem of nuclear proliferation, it is important to recognize the tenets of the probabilistic thought process on which PRA is based. Its image is often one of logic diagrams (e.g., fault trees and event trees) supported by an extensive amount of experiential information and system details. However, PRA is fundamentally a thought process based on the rules of logic and plausible reasoning for answering three basic questions about events, systems, or activities. The questions are: What can go wrong? How likely is that to happen? What are the consequences if it does happen? It is not a formula, a computer program, a detailed formalism, or an event tree, even though all may be involved. This set of questions and thought processes leads to probabilistic, evidential, and inferential analysis of the response of events, systems, or activities to different challenges. PRA was developed to provide insights on matters for which there was very little information, thus leading to an often-heard phrase, “the less information one has about the risk of something, the more important it is to do a quantitative risk assessment.” This of course, is only if it is an important risk that is not statistically obvious or easily exposed by simplified methods. The basic thought process of PRA generally applies to any situation. Thus, the overarching question is not so much does it apply to a particular situation, but rather does it add value to the question being asked and is it correctly implemented. SOURCE: B. John Garrick that might otherwise be overlooked. In many ways it is compatible with the case-by-case assessment methods used by multidisciplinary teams, as described in Chapter 2. Of particular relevance to this study, PRA methods have strengths in  full representation of a complex system;  evidential and inferential analysis of information sources, such as subject matter experts;  rigor of the expert elicitation practices; and  accounting for uncertainties and sensitivities Benefits The committee judges that the strengths of PRA previously identified regarding the elicitation of expert knowledge and accounting for uncertainties and sensitivities could address the shortcomings identified in the execution of predefined framework methodologies and might be useful for case-by-case assessment approaches. The U.S. Nuclear Regulatory Commission and the Environmental Protection Agency use PRA approaches to manage risk and guide decisions. They have provided guidance for rigorous expert selection and elicitation (Kotra et al. 1996, Savy et al. 2002). For example, PRA methods require that experts provide evidence and underlying assumptions to support their judgments. This increases repeatability of the assessments

OCR for page 41
46 APPLICATION OF OTHER RISK METHODOLOGIES and reduces variances among the judgments. PRA methodologies quantify uncertainties and carry them through to the results, which provide a measure of additional information needed to improve the quality of the analysis. However, not all situations require such in-depth assessments as provided with a comprehensive PRA. Bounding and approximation methods may be adequate for many generic applications. Consideration should be given as to which approach would add the most value to predefined framework methodologies. Limitations Because proliferation resistance assessments generally have been made on fuel cycle concepts, but not on deployed systems, there will be little available expertise on the way a particular fuel cycle system works in a particular country and what might constitute a threat. The application of PRA to the safety of nuclear energy facilities has been widely adopted and presents an interesting analogy for proliferation risk. Nuclear power safety PRAs using detailed information on existing facilities can highlight unintended interactions and vulnerabilities. One important lesson learned from the experience with nuclear power plant PRAs (and all U.S. plants have done such an assessment) is that each plant is different and that the risk is very plant-specific. The results can differ significantly even for the same type of unit, located side-by-side. This highlights the importance of facility details in the assessment of risk (or proliferation resistance). In the case of limited shelf-life of an assessment based on unknown details, the PRA thought process or expert elicitation practices cannot overcome the lack of facility detail and does not add value to the question being asked. Finally, the committee notes that a potential weakness of PRA (as with the predefined frameworks or case-by-case analyses) is the failure to consider all pathways and scenarios. To summarize, PRA methodologies have proven highly valuable for assessing risk in engineered systems. They have well-established practices for selecting experts, eliciting expert judgment, and for dealing with uncertainties. Adopting these practices could improve the execution of predefined framework methodologies. However, a successful PRA requires detailed data or information about the system under consideration as well as the ability to quantify the likelihood of threats to that system and consequences of failure. A successful PRA of proliferation risk of a particular fuel cycle in a particular state would therefore require detailed data about the facility, the motivations and thought processes of host-state decision makers, possible clandestine technical capabilities and activities, as well as other information. The committee notes that the risk assessment methods considered by the committee cannot overcome the lack of these important data. The thought process of PRA is valuable as a tool to guide analysis and account for uncertainties, and in many ways is compatible with the methods used by multidisciplinary teams. It would be worthwhile to consider whether elements of the PRA process could improve such analysis.

OCR for page 41
APPLICATION OF OTHER RISK METHODOLOGIES 47 FINDING 3.1: Some of the identified deficiencies in the implementation of the existing predefined frameworks for assessing proliferation resistance could be improved by adopting expert elicitation and data-gathering practices developed by other fields of risk assessment. For example, probabilistic risk assessment (PRA) methods have well-defined and established processes for gathering, quantifying, analyzing, and presenting data that could benefit the nonproliferation community. However, the challenges of an adaptive adversary and of lack of data on proliferation events in particular limit applicability of all of the risk-based methodologies to the assessment of proliferation risk considered by the committee. RECOMMENDATION 3.1: DOE-NE and NNSA should consider whether elements of a formal PRA approach could improve multidisciplinary assessments of proliferation risk, especially the quantification of uncertainties. Although the committee concluded that work on understanding motivations to develop nuclear weapons and modeling an adaptive adversary do not have evidence-based records of success in real-world situations, it supports the inclusion of such approaches into proliferation risk analysis when and if they have an established quantitative basis.