3

UTILITY OF PROLIFERATION ASSESSMENTS

The preceding chapter focused on the first task by identifying the types of policy questions that technical analysis of host-state proliferation risk could inform. This chapter focuses on the second charge of the study task:

TASK 2: Assess the utility for decision makers of existing and historical methodologies and metrics used by DOE and others (such as the International Atomic Energy Agency [IAEA]) for assessing proliferation risk, both for considering the deployment of these facilities domestically as well as the implications of deployment outside the United States.

The committee describes the general characteristics of a selected set of predefined framework methodologies, discusses their strengths and weaknesses, and then comments on their utility to policy makers and decision makers. The first part of the chapter describes the characteristics of each of the methodologies used by the Department of Energy (DOE) and others to assess technical aspects of proliferation risk, recognizing that the methodologies were originally developed to assess proliferation resistance. The committee then evaluates the methodologies and their execution, and identifies shortcomings and applications for which the methodologies are well-suited. The chapter concludes with the committee’s evaluation of the utility of predefined frameworks for decision makers.

In Chapter 2 the committee noted that technical assessments related to proliferation risk are made using a variety of approaches and often are designed to address a question about a particular nuclear technology or capability in the context of a specific country or region. Such assessments are frequently performed on a case-by-case basis by multidisciplinary teams of subject matter experts, often in close collaboration with the intelligence community. As far as the committee was able to determine, these assessments were systematic but did not follow predefined, structured frameworks or use probabilistic risk assessment.

Within the area of nonproliferation policy, questions concerning technical analysis of host-state proliferation risk of a given nuclear fuel cycle represent a relatively small subset. Technical assessments of proliferation resistance address an even smaller subset because, as noted previously, proliferation resistance is contained within the



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 26
3 UTILITY OF PROLIFERATION ASSESSMENTS The preceding chapter focused on the first task by identifying the types of policy questions that technical analysis of host-state proliferation risk could inform. This chapter focuses on the second charge of the study task: TASK 2: Assess the utility for decision makers of existing and historical methodologies and metrics used by DOE and others (such as the International Atomic Energy Agency [IAEA]) for assessing proliferation risk, both for considering the deployment of these facilities domestically as well as the implications of deployment outside the United States. The committee describes the general characteristics of a selected set of predefined framework methodologies, discusses their strengths and weaknesses, and then comments on their utility to policy makers and decision makers. The first part of the chapter describes the characteristics of each of the methodologies used by the Department of Energy (DOE) and others to assess technical aspects of proliferation risk, recognizing that the methodologies were originally developed to assess proliferation resistance. The committee then evaluates the methodologies and their execution, and identifies shortcomings and applications for which the methodologies are well-suited. The chapter concludes with the committee’s evaluation of the utility of predefined frameworks for decision makers. In Chapter 2 the committee noted that technical assessments related to proliferation risk are made using a variety of approaches and often are designed to address a question about a particular nuclear technology or capability in the context of a specific country or region. Such assessments are frequently performed on a case-by-case basis by multidisciplinary teams of subject matter experts, often in close collaboration with the intelligence community. As far as the committee was able to determine, these assessments were systematic but did not follow predefined, structured frameworks or use probabilistic risk assessment. Within the area of nonproliferation policy, questions concerning technical analysis of host-state proliferation risk of a given nuclear fuel cycle represent a relatively small subset. Technical assessments of proliferation resistance address an even smaller subset because, as noted previously, proliferation resistance is contained within the 26

OCR for page 26
UTILITY OF PROLIFERATION ASSESSMENTS 27 analysis of proliferation risk.12 Regardless, “predefined framework methodologies” have been developed specifically for assessing proliferation resistance. Such predefined frameworks provide a generic structured approach for comparing the technical features of different generalized nuclear fuel cycles by breaking the full fuel cycle into individual processing steps and analyzing each step against a threat using a set of metrics (or attributes). These predefined framework methodologies will be the focus of this chapter because they are used by both the Department of Energy Office of Nuclear Energy (DOE-NE) and National Nuclear Security Administration (NNSA) and are being developed by others such as the International Atomic Energy Agency (IAEA) (ANS 2012, Herczeg 2012, Lockwood 2012 ). The committee has not included other methodologies for assessing proliferation resistance in the structured evaluation in this chapter. However, the committee did consider other analyses in Chapter 2, such as case- by-case analyses, and in Chapter 4 in which other methods for assessing risk are reviewed. BACKGROUND OF THE DEVELOPMENT OF PREDEFINED FRAMEWORKS Numerous methods for analyzing technical aspects of proliferation risk of the nuclear fuel cycle have been developed over the decades (GENIV Forum 2011a; Charlton 2012). The Technical Opportunities to Increase the Proliferation Resistance of Global Civilian Nuclear Power Systems (TOPS; NERAC TOPS Task Force, 2001) report formulated a set of qualitative attributes relevant to the intrinsic (materials and inherent properties of the fuel cycle) and extrinsic (institutional and application of safeguards) aspects of a nuclear fuel cycle. The TOPS report was also used as a basis for identifying R&D priorities for DOE and recommended, “Development of improved methodologies for assessing the proliferation resistance of different systems, including those that further the understanding of the trade-offs between intrinsic and extrinsic measures.” Most predefined frameworks under consideration today were developed in response to the TOPS report and the R&D priorities that it identified. In 2001, a program plan for development of a nonproliferation assessment methodology was prepared for NNSA’s Office of Nonproliferation and International Security (NA-24) (NNSA 2001). The goal of the program and eventual working group (DOE 2002, Mladineo et al. 2003; DOE 2002) was to:  Develop a standardized methodology or set of methodologies for assessing proliferation resistance of different reactor and fuel cycle systems, and other nuclear systems and processes.  The methodology must be capable of measuring trade-offs among concepts and systems.  The methodology must include quantitative tools to ensure technical rigor. 12 As discussed in the Introduction, “proliferation resistance” focuses on the engineering aspects of a particular nuclear fuel. “Proliferation risk” is a broader concept and includes analysis of country-specific issues such as the probability that an adversary will choose to proliferate along a particular pathway, and the probability of success along that path. For more details, see Chapter 1.

OCR for page 26
28 UTILITY OF PROLIFERATION ASSESSMENTS In 2002, NA-24 issued functional requirements for development of a nonproliferation assessment methodology (NPAM) (DOE 2002) which identified the need for the methodology to:  address a broad range of questions to be addressed by the methodology including country-specific analyses;  address uncertainties and sensitivity analysis;  incorporate weighting techniques to vary priorities; and facilitate use of expert knowledge On the basis of the NPAM documents discussed above, NNSA and DOE-NE and several countries within the Generation IV International Forum (GIF) embarked on a joint effort to develop a methodology for use in diverse applications including country-specific applications, which resulted in the Proliferation Resistance and Physical Protection (PR&PP) methodology. OVERVIEW OF PREDEFINED FRAMEWORKS Predefined frameworks are methodologies designed to consistently and transparently evaluate proliferation resistance through a standardized set of predetermined attributes (or metrics) that are evaluated throughout the individual processing steps of a given fuel cycle (Mendez et al. 2006, Ford 2010).They provide a framework to gather and organize data to score the attributes, which requires expert knowledge and experiential data. Such data are gathered and combined into higher-level scores to give the fuel cycle an overall measure of proliferation resistance (usually high, medium, or low or numerical equivalents). This “black box” approach can mask the underlying reliance on subject-matter experts and lead to assumptions that the frameworks are objective models or simulations. They are not. A typical set of metrics of proliferation resistance includes  technical difficulty of proliferation,  cost required to overcome barriers to proliferation,  time needed to proliferate,  type of material to be acquired for proliferation purposes, and  the probability of detection (or transparency). The committee judges these metrics to be reasonable, and many are cited in the proposed key policy questions found in Chapter 2. The committee identified six predefined framework methodologies for more thorough review based on their variety of approaches for combining assessments at each processing step (e.g., multi-attribute utility analysis or event-tree logic). Several were selected on the basis of their use internationally and their use most commonly in the United States. This list and the committee’s assessment of the methodologies were reviewed by a set of practitioners of various predefined frameworks (see Appendix B). The following frameworks were selected:

OCR for page 26
UTILITY OF PROLIFERATION ASSESSMENTS 29  TOPS methodology,  Japan Atomic Energy Agency (JAEA) methodology,  Simplified Approach for Proliferation Resistance Assessment (SAPRA) methodology,  Texas A&M University Multi-Attribute Utility Analysis (TAMU MAUA) methodology,  Risk-Informed Proliferation Analysis (RIPA) methodology, and  Generation IV International Forum Proliferation Resistance & Physical Protection (GIF PR&PP) methodology. Perhaps noteworthy by its absence is the IAEA’s International Project on Innovative Reactors and Nuclear Fuel Cycles (INPRO). Initial efforts to use INPRO precepts to assess proliferation resistance led by the Republic of Korea under IAEA auspices (IAEA 2011b; Lee et al. 2012) have been published, for example, the Proliferation Resistance: Analysis/Diversion Pathway Analysis (PRADA) project on Direct Use of Pressurized Water Reactor Spent Fuel in CANDU (DUPIC). However, in the most recent guidance on use of the INPRO assessment methodology (IAEA 2008 volume 5, Annex A, p. 39) it is stated: The goal of a PR [proliferation resistance] evaluation is to provide guidance for the nuclear energy system development groups that will develop the proliferation resistant technology, and to present results, showing how the non-proliferation goals will be met, to institutions responsible for deciding which nuclear concepts to pursue. and At present, the [INPRO] evaluation method in the field of proliferation resistance is not complete. The group of consultants working in this area did not come to a common conclusion on scales for some indicators as well as on acceptance limits of criteria. Thus, final presentation of the evaluation results is not yet defined. Thus, the Korea-led effort which involved developing detailed metrics within high-level INPRO basic principles of proliferation resistance and five general “user requirements” has not yet been adopted by the IAEA. Rather than further developing a framework for INPRO, the international community is pursuing an approach of harmonizing the relatively broad INPRO methodology with the more detailed PR&PP methodology (ANS 2012) in the Proliferation Resistance and Safeguardability Assessment (PROSA) project which began in February 2012. Therefore, the committee chose not to review a work in progress. Assessment of the PR&PP methodology can be seen below. Below, the committee provides high-level descriptions of these six methodologies. Additional details on their characteristics are also noted in a summary table in Appendix B.

OCR for page 26
30 UTILITY OF PROLIFERATION ASSESSMENTS Technical Opportunities to Increase the Proliferation Resistance of Global Civilian Nuclear Power Systems The TOPS task force was established by DOE-NE’s Nuclear Energy Research Advisory Committee (NERAC). The task force was charged “to identify near- and long- term technical opportunities to further increase the proliferation resistance of global civilian nuclear power systems.” This included an R&D focus on methods for assessing proliferation resistance by evaluating “the relative proliferation resistance of specific fuel cycles in terms of a generic set of ‘attributes.’ The attributes are derived by first defining the barriers to proliferation inherent in the design of the system, its materials and facilities, and its modes of operation” (NERAC TOPS Task Force 2001). The approach followed a few basic steps for each fuel cycle under consideration:  Define the different processing steps.  Identify potential proliferation pathways in each processing step.  Assign values to the attributes based on the threat (e.g. host state or subnational threat). Attributes describe the relationship between the elements of a fuel cycle, the threats to those elements, and the effectiveness of barriers to inhibit these threats. Intrinsic attributes are technical features such as material properties (isotopic or chemical barriers, amount of material, detectability) and extrinsic attributes are institutional and operational barriers such as safeguards and inspections. The evaluation of the attributes relies on subject matter expertise and existing data.13 Although the 2001 TOPS report made an initial attempt at comparative assessment of a set of fuel cycles following its proposed attributes methodology, it is not currently a methodology that is used. It is included in this list as a historic methodology. The proliferation resistance attributes for a proposed fuel cycle (Fuel Cycle A) against a generic threat (sophisticated host state, covert diversion) are shown in a notional example of a TOPS-like assessment in Table 3.1. The table includes the list of intrinsic and extrinsic barrier attributes, the weighting factors for aggregation, a list of processing steps, and values of each attribute for each processing step. The values within this particular table are purely randomly assigned because it is a notional example; in a real assessment, the values would be determined by subject matter experts. Other fuel cycle 13 It should be noted that among the quantitative measures, some require underlying judgments made by experts. For example, the determination of threshold values for seemingly technical attributes, such as the 100-rem/h dose rate at 1 m below which nuclear material is assumed to not be self-protecting (NRC 2011a) may be an important demarcation in a proliferation risk assessment, but it has been questioned by other experts who believe a value of 500-rem/h at 1 m is more appropriate (Bathke et al. 2009).

OCR for page 26
UT TILITY OF PR ROLIFERATI ION ASSESSM MENTS 3 31 TABLE 3.1 Noti ional Examp of a TOPS-Like Asse ple essment of PProliferation Resistance Attrib butes for a Proposed Fue Cycle A Against a Ge P el A eneric Threat (Sophistica Host t ated State, Covert Divversion). SOUURCE: Mod dified from I Inoue et al (2 2004). and/o threat opti or ions would have a simila spreadshe et developed to allow fo relative h ar d or comp parison at thi or fully ag is ggregated lev vels. The val lues within t table can be the n comb bined through weighting functions so that a final single valu of prolifer o l, ue ration resist tance is deterrmined and can be comp pared with o ther fuel cyc cles. Japan Atomic Ene A ergy Agency Methodolo y ogy The Japan Atomic En n nergy Agency (JAEA) de y eveloped an assessment methodolog gy based primarily on the TOPS attributes methodology to provide a qualitative relative d o S m y e assessment of the proliferatio resistance of systems, processes, a nuclear facilities as e on e , and part of its strategy to commer o y rcialize fast- -reactor techhnology. Two threats are considered: o e : a cov diversion by a state and theft by a subnationa group. Th methodolo defines vert n a al he ogy mater and tech rial hnical barrier attributes of mass and v r o volume of mmaterial, radiiation fields and isotopic and chemical composition an like TOP primarily relies on q nd, PS, y qualitative exper knowledge to evaluate the attribut Sensitivi analyses have been p rt e e tes. ity performed again specific attributes, bu the charact nst a ut terization of uncertaintie of each at f es ttribute and how the uncertain t nties are carr through the analysis are not disc ried h s cussed or inc cluded in thee result ts. Simp plified Appr roach for Pr roliferation Resistance Assessmen n e nt SAPRA was develope by the Fre w ed ench Workin Group on Proliferatio Resistanc ng n on ce and Physical Prot P tection, whic includes representativ from the Foreign Aff ch r ves e ffairs and Indus ministrie French Safety Institu (IRSN), A stry es, ute Atomic Ener and Alte rgy ernative Energ Commis gies ssion (CEA) Electricity of France (E ), y AREVA, Inc. SAPRA EDF) and A

OCR for page 26
32 UTILITY OF PROLIFERATION ASSESSMENTS further expands TOPS and JAEA (Greneche et al 2007) by delineating the steps to proliferation as four stages: diversion, transportation, transformation and nuclear weapons fabrication. SAPRA also introduces several different attributes (e.g. “dangerousness” instead of isotopic and chemical barriers to include other factors, such as reactivity of a chemical to water) but the approach to assigning values using experts and combining the values to a single result is consistent with the TOPS approach. The result of the analysis is a quantitative result that includes an assessment of each of the four defined stages of proliferation. At each stage, the values of the attributes are aggregated and normalized to 1. SAPRA considers only the case of state proliferation, not acquisition of material or a weapon by a subnational group. Uncertainty is not included in the assessment of the attributes nor is it reported in the final results. Sensitivity of the barriers to different threats is considered but an analysis of which attributes are most affected was not reported. Texas A&M University Multi-Attribute Utility Analysis A methodology for computing proliferation resistance was developed at Texas A&M University (TAMU) using a multi-attribute utility analysis (MAUA) method that assigns utility functions to each proliferation attribute (Charlton 2007, 2012). The result of this method is a numeric ”nuclear security measure” on a scale from 0 to 1, where “0” implies complete vulnerability to proliferation and “1” complete proliferation resistance or proliferation-proof. The method assigns weighting factors to a set of critical material and facility attributes of which there are 14 (further expanding the list of attributes beyond the SAPRA method for the purpose of decoupling interdependencies between previous attribute lists). The weighting factors are based on input from various experts in nuclear proliferation–related fields and contain both objective and subjective information. The methodology focuses the proliferation resistance assessment on the flow of material through the fuel cycle as a function of time—from its initial input into the fuel cycle to its eventual disposal. The TAMU MAUA methodology limits the threat to diversion of nuclear material by a host state but does not address other threats such as theft or terrorism. It accounts for the intent of the host state to proliferate by assigning it a maximum and constant value in the analysis. Uncertainty is not included in most of the utility functions that are established for each of the attributes, although “measurement uncertainty” is included as part of the higher-level metric of “Difficulty of evading detection by the accounting system.” There is no mention of sensitivity analysis. Risk-Informed Probabilistic Analysis The RIPA methodology assesses the most likely paths for proliferators to acquire nuclear weapons, including the cost and time required for each using various risk- informed assessment techniques (Blair et al. 2002, Rochau et al. 2012). The goal of RIPA was to create a set of separable components of the proliferation risk problem that could be analyzed by experts and reused as needed for future analysis. RIPA includes (1) influence diagrams and resulting proliferation pathways, (2) proliferation scenarios, and (3) the proliferation measures. The influence diagrams present the steps (nodes) that must

OCR for page 26
UTILITY OF PROLIFERATION ASSESSMENTS 33 be followed to succeed in building a nuclear weapon. There were no examples of RIPA being used for specific applications. The documentation reviewed by the committee did not specify how uncertainties would be analyzed or reported nor did it mention sensitivity analysis. Generation IV International Forum Proliferation Resistance & Physical Protection The PR&PP methodology was developed as an approach for assessing the proliferation resistance of advanced nuclear energy systems and is the outcome of the NPAM documents described earlier. It is one of the more developed methods for analyzing proliferation resistance in the current literature and is used by GIF and others.14 For a given system, the goal of the method is to define a set of challenges, analyze the system response to those challenges, and assess the outcomes (see Figure 3.1). The challenges to the nuclear energy systems are the threats posed by potential proliferant states and by subnational adversaries. The response to these challenges is determined by evaluating the technical and institutional attributes of the proposed Generation IV nuclear energy systems. The outcomes of the system response are expressed in terms of proliferation resistance and physical performance measures. The PR&PP approach considers multiple facility types and activities to enable the proliferation pathways from acquisition and processing of material to fabrication of a nuclear explosive device as concealed and overt misuse of nuclear facilities. Types of proliferation considered are overt and concealed diversion, concealed breakout from treaty agreements and misuse, and clandestine facilities. For proliferation resistance, the top-level metrics are technical difficulty, proliferation cost, proliferation time, fissile material type, detection probability, and detection resource efficiency. The final steps in the methodology are the integration of the findings of the analysis and the interpretation of the results. The form of the results includes best estimates from subject matter experts for numerical and linguistic descriptors that characterize the overall proliferation resistance of the fuel cycles (GENIV Forum 2007, 2011b). The PR&PP studies to date have focused on comparing the proliferation resistance of existing and future fuel cycles. The challenges (threats) are defined by descriptions of generic (not specific) states and adversaries, and limited details are included on the fuel cycle facilities.15 14 For example, safeguards design for new CANada Deuterium Uranium (CANDU) reactors, and accelerator-driven nuclear systems, MYRRHA, in Belgium. 15 General assumptions and categories have been made regarding the technical capabilities of the proposed host state. More information can be found at http://www.gen-4.org/Technology/horizontal/proliferation.htm.

OCR for page 26
34 UTILITY O PROLIFER OF RATION ASSE SESSMENTS FIGUURE 3.1. PR R&PP’s meth hodology for assessment of prolifera r t ation resistan and nce physi protectio SOURCE GENIV (2 ical on. E: 2011a). EVAL LUATION OF METHO O ODOLOGIE ES The methodologies were evaluate against a s of charac w ed set cteristics dev veloped by the coommittee. Appendix B describes the evaluation and provide a summary table of the A d e es y e resultts. The commmittee found that none of the method d dologies curr rently captur specific re host-state factors and therefore none curr rently assess proliferation risk, for exxample, the probaability that a specific hos state will choose to pr st c roliferate alo a particu pathway, ong ular the prrobability of success alo that path and the con f ong h, nsequences o proliferat of tion summed d over all possible pathways (T Takakai et al. 2005, Pom meroy et al. 22008). The mmethodologie es do asssess prolifer ration resista ance to allow for relative comparison between a set of given w e n n fuel cycles. There are some attempts to capture state- c e a -specific asppects via genneric descrriptions (e.g., a non–nucl weapons state with nuclear ener systems, technically lear rgy , comp petent, and a signatory to the Treaty on the Non- o -Proliferation of Nuclear Weapons n r (NPT GENIV Fo T, orum2011b or by assuming the inten to prolifer is maxim o nt rate mized and const but there are no exam tant e mples of a sp pecific host state or an e existing nucl energy lear 16 facili ity. The gener approach to assessing proliferatio resistance is the same for all of ral h g on e e the methodologie to outline the detailed steps of the given nuclear fuel cycl and to m es: e d e le evalu a set of metrics (or attributes) th characteri barriers t proliferati against a uate m a hat ize to ion given threat. The number and details of th attributes and how th are comb n d he s hey bined differ betwe methodo een ologies but th results ar consistent with each o he re other and are in e agreeement with general accep judgme on prolif g pted ents feration barrriers within nnuclear fuel 16 Breaakout scenarios could be asse essed by consid dering only the intrinsic resis e stance (and rem moving any barrier introduced by extrinsic me rs b easures such as safeguards). s

OCR for page 26
UTILITY OF PROLIFERATION ASSESSMENTS 35 cycles (e.g., enrichment and reprocessing steps have the lowest proliferation barriers).. The committee judges this predefined framework approach to be sound in defining a set of steps in a complex system for analysis and organization of data to allow for comparisons between fuel cycles. However, the committee determined that there were shortcomings in the execution of the assessments and inherent limitations on their application. Shortcomings in Execution Expert Elicitation At their core, all predefined frameworks rely on expert knowledge. Information is obtained from subject matter experts leading to a score for each attribute and weighting factors for aggregation. Because expert knowledge is combined into higher-level outputs, the details of these practices and their impact on the results can be obscured. The committee found that two important processes related to gathering data from experts were found to be poorly implemented: selection of experts and expert elicitation. The U.S. Nuclear Regulatory Commission (USNRC) has published a technical position (NUREG-1563) on expert elicitation that “describe[s] acceptable procedures for conducting expert elicitation when formally elicited judgments are used to support a demonstration of compliance” (Kotra et al. 1996). The U.S. Environmental Protection Agency has also developed clear procedures for gathering expert judgment and knowledge (EPA 2009). These practices when implemented have produced results of higher quality than other methods and are therefore considered “best practices” for expert elicitation. The guidance from the USNRC describes how formal expert elicitations should be performed: Expert elicitation is a formal, highly structured, and well-documented process whereby expert judgments, usually of multiple experts, are obtained. Although informal expert judgment involves only subject-matter experts, formal expert elicitations usually involve normative experts, generalists, and subject-matter experts. (Kotra et al. 1996, p. 3) A normative expert is one trained in decision analysis, statistics, and probability; a generalist has broad training across the entire elicitation subject area; and subject matter experts are specialists concerning subparts of the subject area. In none of the six methodologies was the selection process of experts identified nor was there mention of the use of a normative expert (GENIV Forum 2011a) or a generalist. The most recent PR&PP documentation (GENIV Forum 2011a) does provide some general guidance on criteria for selection of experts, although the need for a normative expert or generalist is not part of the guidance. The USNRC and EPA guidance also describes methods for eliciting information from the experts and the importance of documenting the results. In most of the methodologies, the process for collecting expert information was not stated. Some of the methodologies did indicate that expert knowledge was collected through the use of questionnaires and surveys (TAMU MAUA and PR&PP). Additionally, the committee notes that surveys and questionnaires have been found to be problematic in the methods

OCR for page 26
36 UTILITY OF PROLIFERATION ASSESSMENTS of expert elicitation. The impact of poor execution of expert elicitation was recently recognized by the PR&PP GIF working group: informal expert elicitations often provide demonstrably biased or otherwise flawed answers to problems. Without a formal process and strong controls, experts may be asked to provide judgments on issues that go beyond their expertise, or their estimates might be combined in misleading ways which distort the results (GENIV Forum 2011a, p.67; Budlong-Sylvester et al. 2006). However, the committee has found the proposed changes to the process of expert elicitation in PR&PP Revision 6 to be insufficient to effectively address these specific shortcomings (GENIV Forum 2011a) because experts are still solicited through a survey form, for example.17 Uncertainty Analysis In technical assessments utilizing expert knowledge, the aggregate uncertainty of the results to changes in assumptions and information is an important factor in determining the confidence level of the results. It is important to define the uncertainties at the expert elicitation phase and to properly account for the uncertainties through their aggregation to the final result. In the predefined framework methodologies that were reviewed, the TAMU MAUA methodology accounts for measurement uncertainties as part of one high-level attribute but does not otherwise include them in its analysis. The PR&PP methodology notes that the results contain uncertainties because the measurement ranges are gross (high, medium, low). The PR&PP, TAMU MAUA, and RIPA methodologies are, in theory, capable of being extended to doing quantitative uncertainty analyses. To date, these methodologies, in practice, do not provide a mechanism for addressing or quantifying uncertainty, although guidance for using the PR&PP framework indicates that this should be done (see PR&PP overview above). Sensitivity Analysis Sensitivity analysis of the final results (which may not be a single number or qualitative rating) to changes in the information gathered and the underlying assumptions by the experts often provides significant insight and may be more important than the final result per se. Only the SAPRA and JAEA methodologies reported sensitivity analyses against a subset of attributes and highlighted the effect on the results. As noted above for uncertainty analyses, the PR&PP, TAMU MAUA, and RIPA methodologies, in theory, can perform quantitative sensitivity analyses but did not include these analyses in the examples seen by the committee. 17 In best practices for expert elicitation established by the EPA and USNRC, a discussion leader requires the experts to explain the basis for their evaluation in front of the other experts. This leads to (a) making sure that each expert has the same understanding of the question, and (b) questions and challenges from other experts and modification of views in response to new facts and assumptions. The reason surveys are undesirable is that this interaction of the experts is lost or is not as efficient.

OCR for page 26
UTILITY OF PROLIFERATION ASSESSMENTS 37 Limited Applicability Facility details (e.g. technical design features, operational modalities, institutional arrangements and safeguards measures) are not known for deployments of future fuel cycles and may not be known during and throughout construction, especially for some countries in which the United States may have proliferation concerns. Country-specific and facility-specific factors will strongly affect the final proliferation resistance of a particular fuel cycle facility at the time of fuel cycle deployment and for several decades thereafter. Therefore, proliferation resistance assessments of future fuel cycles are limited to the known engineering details related to the intrinsic attributes of the proposed fuel cycle. Expert-guided assumptions and estimates are made for extrinsic attributes (which should have well-documented uncertainties associated with them). These assessments have a “shelf life” because the proliferation resistance will change as the barriers to proliferation are further defined throughout the development of the fuel cycle and the design of the facility. Assessments should not be considered final and should be periodically updated as more details of the specific fuel cycle or facility are determined. Unfortunately, decision makers and policy makers need to make informed decisions related to future fuel cycles, nuclear exports, and peaceful nuclear agreements, as discussed in Chapter 2. This leads to an inherent limitation for the predefined frameworks because in these applications, the facility and host-state details are not defined. Of course, the case-by-case and other analyses are similarly limited by lack of data. To summarize, the committee considered a set of predefined frameworks and found the following:  shortcomings in their execution because of  poor and/or undocumented expert elicitation processes and  lack of sensitivity and uncertainty analyses;  inherent limitation of applicability because of  unknown facility and host-state details for future fuel cycles and  limited shelf life of assessments. UTILITY OF PREDEFINED FRAMEWORKS FOR DECISION MAKERS In addressing this task, the committee notes that the “utility” of a methodology is subjective and dependent on the individual and/or organization. The committee considered the frequency of use as an indirect measure of utility and discussed the apparent effect of the methodologies’ results with policy makers and decision makers to determine their subjective opinions. The committee considered the extent to which predefined frameworks are used and have been applied to various types of decisions. Only two examples were provided despite requests to both sponsors and to many of the policy and decisions makers who provided briefings or information to the committee (see Appendix D). These two examples involve the use of the PR&PP methodology: (1) during the process of

OCR for page 26
38 UTILITY OF PROLIFERATION ASSESSMENTS BOX 3.1 Examples of the Use of Predefined Framework Methodologies When asked for examples in which predefined framework analysis played a part in policy or decision making within the U.S. government, the following two examples were cited by NNSA. GNEP: During the process of formulating plans for the Global Nuclear Energy Partnership (GNEP) initiative, the Unites States used a predefined framework methodology (PR&PP) to comparatively assess the resistance to proliferation of different future fuel cycle alternatives in three categories: once-through, full actinide recycle, and partial actinide recycle (DOE 2008b). Results highlighted strengths and weaknesses of the different fuel cycles and summarized the results as follows: “Because the alternatives present complementary risks and benefits, this assessment does not identify a preferred alternative or alternatives” (DOE 2008b, p. xv). This assessment was performed in parallel with preparation of the draft GNEP Programmatic Environmental Impact Statement (PEIS) (DOE 2008b) which stated that change to a closed fuel cycle represented DOE’s preferred option. Pyroprocessing: In the context of discussions with South Korea about renewing its Nuclear Cooperation Agreement with the United States, South Korea has asked the United States for its consent to use pyroprocessing for its U.S.-origin spent fuel. When pyroprocessing was first considered by South Korea, the United States made a decision that pyroprocessing was not considered to be reprocessing and was acceptable. A more recent technical analysis using the PR&PP methodology (DOE 2008b, p. 68) determined that there was negligible difference in proliferation resistance between pyroprocessing (now further developed) and PUREX as compared to the once-through fuel cycle. This study was used in part to justify a change in policy with respect to pyroprocessing, which the South Koreans continue to contest. The two countries have subsequently undertaken a joint 10-year study on spent fuel management, one part of which will consider the proliferation resistance of pyroprocessing. This example highlights that proliferation resistance assessments made on technologies under development should be performed throughout the development cycle as further details are determined and new information that may have significant impact on proliferation resistance becomes known. formulating plans for GNEP and (2) to compare pyroprocessing with traditional approaches to recycling. Details are provided in Box 3.1. In its meetings with DOE and its contractors this committee heard that NNSA did not usually use any predefined framework methodology to guide nonproliferation decisions, especially in cases involving country-specific considerations. Reasons given for this include:

OCR for page 26
UTILITY OF PROLIFERATION ASSESSMENTS 39  not wanting to rely on a single rolled-up measure,18  concern that the framework will make (instead of inform) a decision or might “box the decision maker in,”  concern that “proliferation resistance” may be misinterpreted as “proliferation-proof.” As noted in Chapter 2, the committee found that nuclear nonproliferation policy makers and decision makers use their own knowledge, coupled with technical analysis by multidisciplinary teams of subject matter experts established on a case-by-case basis to provide insight into nuclear proliferation risk, and have shown little interest in multidisciplinary teams using predefined frameworks or formal risk-based approaches for such decision making. Based on the results to date, there is little expectation that predefined framework methodologies will provide additional insight, a view supported by previous findings including a brief, initial proliferation resistance assessment referenced in the TOPS report (NERAC TOPS Task Force 2001, p.15): “In some respects, most of the findings from the analyses were not new and simply reinforced the judgments that had arisen over the years.” The committee found several examples in other domains within the U.S. government in which decision makers utilize predefined framework-like tools to inform decisions. Examples include the Office of Cooperative Threat Reduction within the Department of State in which an assessment tool is used to inform the prioritization of countries for engagement on nuclear, chemical, and biological security (Dolliff 2012); and the Domestic Nuclear Detection Office within the Department of Homeland Security, which uses a risk-based tool to guide the optimization of architectures for global nuclear detection (Streetman 2012). Such methodologies are apparently useful to policy makers and decision makers dealing with complex problems and with a willingness to engage in the analysis process and not simply the results. In these cases, the users recognize that the tools are not predictive and that they are not beholden to the results. Rather, the tools provide a structure for organizing complex problems with a large number of variables and assessing which factors are most important to the results that inform the final decisions. While recognizing the limitations of predefined frameworks, the committee judges that those frameworks can have value if well executed. They provide a structured approach that causes the analyst to explore a range of possibilities in assessing proliferation resistance. They also have value as a way to consistently compare the attributes of potential future nuclear fuel cycles and for identifying where safeguards can be most effective in raising barriers to proliferation. They provide a common lexicon and structure for communicating with international partners about nuclear energy decisions. In addition, they provide a valuable structure for education of the next generation of experts on nuclear energy and nonproliferation. 18 In the committee’s review of the frameworks, it was noted that many predefined framework approaches do not yield only a single output and none of them have to do so.

OCR for page 26
40 UTILITY OF PROLIFERATION ASSESSMENTS FINDING 2.1: Predefined frameworks have been developed and used to assess the proliferation resistance of partial or full nuclear fuel cycles. These methods provide a useful framework for comparing the intrinsic metrics or “attributes” of existing and potential future nuclear fuel cycles and for identifying where safeguards can be most effective in raising barriers to proliferation. However, these comparisons address a small subset of the wider range of issues faced by policy makers and the committee was able to determine that the frameworks have rarely been used to inform policy decisions. Additionally, there have been shortcomings in their execution.