FINDING 4.1: The understanding of security risks at nuclear power plants and spent fuel storage facilities can be improved through risk assessment. Event trees and other representational formalisms can be used to systematically explore terrorist attack scenarios, responses, and potential consequences. Expert elicitation can be used to rank scenarios; develop likelihood estimates; and characterize adaptive adversary responses to various preventive, protective, or deterrence actions. The identification of scenarios may be incomplete, and the estimates developed through expert elicitation are subjective and can have large uncertainties. Nevertheless, risk assessment methods that focus on the risk triplet—scenarios, likelihoods, and consequences—can contribute useful security insights.
RECOMMENDATION 4.1A: The U.S. nuclear industry and the U.S. Nuclear Regulatory Commission should strengthen their capabilities for identifying, evaluating, and managing the risks from terrorist attacks. Particular attention is needed to broaden scenario identification, including asymmetric attacks; account for the adaptive nature of adversaries; account for the performance of plant security personnel in responding to the identified scenarios; estimate the potential onsite and offsite consequences of attack scenarios, including radioactive releases
and psychological impacts; and develop strategies for countering the identified threats.
RECOMMENDATION 4.1B: The U.S. Nuclear Regulatory Commission should sponsor a spent fuel storage (wet and dry storage) security risk assessment for U.S. nuclear plants. The primary objectives of this assessment should be to (1) develop and exercise the appropriate methodologies for characterizing risk and estimating uncertainties, and (2) explore the benefits of risk assessment for enhancing security at U.S. nuclear plants. This assessment should be subjected to independent review by technical peers (i.e., peer review) as part of the development process.
Sidebar 5.1 provides definitions for some terms that are used in this chapter.
Risk assessment is a formalized thought process for answering the following triplet of questions (Kaplan and Garrick, 1981):
- What can go wrong?
- How likely is that to happen?
- What are the consequences if it does happen?
Probabilistic risk assessment (PRA) is a highly developed methodology for performing a risk assessment that is widely applied by the nuclear industry and its regulator. The adjective “probabilistic” is included in this terminology to emphasize that the likelihood (second item) of an event is expressed in the assessment. As noted in Appendix I in the committee’s phase 1 report (NRC, 2014), PRA describes the application of risk assessment to accidents at nuclear plants. In the following, security risk assessment will refer to assessments in which the likelihood of terrorist events is included in the evaluation process.
The specific metrics chosen to express the risk depend on the system or activity to be informed by the risk assessment. Crucial to all modern risk assessments (e.g., USNRC, 1990) is the recognition that their results are uncertain and that this uncertainty needs to be reflected in the results. Results are typically presented in terms of uncertainty distributions rather than point values (e.g., EPRI, 2012b; USNRC, 2013). The U.S. Nuclear Regulatory Commission (USNRC) Regulatory Guide 1.200 (USNRC, 2009b) and the PRA consensus standard published jointly by the American Nuclear Society and the American Society for Mechanical Engineers (ASME/ANS, 2009) emphasize the importance of identifying and understanding uncertainties for achieving technical acceptability in a PRA.
The identification of terrorist threats against reactors and spent fuel pools is a necessary part of security planning at all nuclear plants (Sidebar 5.2). Analyses or exercises can be undertaken for each identified threat to explore whether the terrorist is likely to succeed in causing significant damage, and defenses can be adjusted accordingly. But whether the identified set of threats is complete is generally unknown. As discussed in Chapter 3, there is also a pressing need to more systematically identify potential cyber, insider, and asymmetric1 security threats. More formalized processes for identifying and analyzing threats—for example PRA—could help to improve security at nuclear plants.
The National Research Council public report (NRC, 2006) questioned the feasibility of applying PRA to nuclear plant security because of the difficulty of developing a complete set of bounding attack scenarios and estimating their likelihoods of occurrence. The report noted correctly that attack probabilities depend on factors such as terrorist motives, expertise, and access to technical means, which may be difficult or impossible to
1Asymmetric attacks refer to attacks where there are dissimilarities in the capabilities, strategies, and/or tactics between an adversary and a defending force. Additional discussion of asymmetric attacks is provided in Chapter 3.
know. Although NRC (2006) expressed reservations about the possibility of quantifying risk, it also indicated that qualitative judgments could be made about the relative vulnerabilities of spent fuel storage facilities to various terrorist attack scenarios described in the classified report (NRC, 2004).
The present committee agrees with NRC (2006) that there are technical challenges associated with identifying terrorist attack scenarios and quantifying their likelihoods. Nevertheless, the committee judges that the risks of terrorist attacks on nuclear plants and spent fuel storage facilities can be characterized by adapting well-established risk assessment methods. For example,
- Event trees and other representational formalisms can be used to systematically explore terrorist attack scenarios2 and their potential consequences. One can, for example, generate event trees with associated likelihoods that represent “baseline” conditions at a nuclear plant at the beginning of a terrorist attack scenario. These event trees and their associated likelihoods can be modified as the scenario unfolds to reflect defensive actions by plant personnel and terrorist adjustments to those actions.
- These scenarios and their associated likelihoods (usually expressed as frequency distributions) can be developed and estimated, respectively, using established methods such as expert elicitation and CARVER analysis (see Appendixes 5A and 5B). It may be possible to use likelihood estimates to rank scenarios and identify their comparative importance if their uncertainties are small and/or uncorrelated.
The committee recognizes, of course, that the set of scenarios identified using such methods will likely be incomplete, and likelihood estimates and scenario rankings will be subjective and may have large uncertainties. Nevertheless, the use of these methods provides greater technical rigor and transparency to an analysis than traditional deterministic methods such as the design-basis threat (DBT) (see Chapter 3).
The use of risk assessment can help to
- Broaden scenario identification for both physical and cyber terrorist attacks, including insider and asymmetric attacks;
- Account for the performance of plant security personnel in responding to the identified scenarios;
- Identify potential onsite and offsite consequences of such scenarios, ranging from radioactive releases to psychological impacts; and
- Characterize uncertainties in the scenarios, likelihoods, and consequences.
In fact, risk assessment can provide useful security insights that are analogous to the insights derived from safety risk assessments.
Risk assessment allows for the orderly development of conclusions that reflect the totality of information available about a system’s performance in particular circumstances. Such assessments can provide evaluations of risks in terms of the frequencies of occurrence of random events or consequences conditional on occurrence of a particular event such as a terrorist attack. Both types of assessments can be valuable for identifying the relative impor-
2 Such a scenario would describe the characteristics of the attack including the size and weaponry of the attacking force, its tactics, and the plant systems that are targeted.
tance of the various contributors to performance outcomes and the more sensitive elements of the system in affecting such outcomes. For either type of risk assessment, explicit treatments of uncertainty are essential, because the components of the assessment (e.g., scenarios, likelihoods, and consequences) can have substantial uncertainties.3
The adaptation of PRA to security would establish a common framework for assessing risks at nuclear plants. This would provide a consistent basis for operational and regulatory decision making about risks, including at the safety-security interface; it could provide further opportunities to risk-inform security regulations; and it could help improve cost-benefit analyses associated with the backfit rule (see Chapter 5 in NRC ).
The current state of development of risk analysis for nuclear plant security is similar to that for safety risk analysis in the early 1970s. At that time it was argued that characterizing the likelihoods of physical accidents was infeasible or at least impractical because they had such low probabilities of occurrence and large uncertainties. There has been considerable technical progress in the use of risk assessment for nuclear plant safety4 over the past four decades. The committee judges that it is not unreasonable to expect that similar progress can be achieved with security risk assessment. In fact, efforts are already under way to further develop this methodology.
The USNRC sponsored a Risk Informed Security Regulation Workshop5 in 2014 to discuss the current state of efforts to use risk assessment in nuclear plant security and to identify opportunities to risk-inform security regulations. The workshop identified sabotage-initiating-event definition and uncertainty estimation as key areas for further development, but it did not recommend explicit actions to achieve such development. It did, however, suggest that a security risk assessment effort be undertaken.
In a keynote address at the workshop, then-Commissioner George Apostolakis argued that the DBT paradigm for nuclear plant security was too restrictive, and he discussed the usefulness of the expert judgement elicitation-based approach for characterizing threats and for integrating safety and security assessments. He noted that a common framework for safety and security would enable consistent decision making and the explicit treatment of the safety and security interface.
3 Presentation and propagation of uncertainties is particularly important in the case of security risk assessment because the uncertainties associated with initiating threat events and consequences are likely to be much larger than for natural events and accidents. Consequently, reliance on measures of central tendency may be particularly misleading.
4 See Appendix I of the present committee’s phase 1 report (NRC, 2014) for a discussion of the history and current state of risk assessment practice as well as needs for further technical advancement.
Progress is currently being made in adapting and extending risk analysis to security applications. Procedures for conducting security risk assessments do not yet have consensus-level agreement from professional standards organizations as do safety risk assessments, but there do exist “how-to, step-by-step” methods for performing security risk assessments (see, for example, EPRI, 2004; Garrick et al., 2004; Hirschberg et al., 2016). These methods were developed after the September 11, 2001, terrorist attacks.
Garrick et al. (2004) use the risk triplet as the organizing principle for conducting a security risk assessment for terrorist attacks that could have catastrophic consequences. They note that “there is an urgent need for (1) understanding the threats involved, (2) appreciating vulnerabilities, and (3) an analytical process for assessing the risk and mitigating the threat” (p. 131). A major contribution of this paper is a procedure for quantitative risk assessment of threats.
Their methodology involves three major steps: (1) analyzing and quantitatively assessing the threats, (2) characterizing the success state of the system under attack, and (3) performing a quantitative vulnerability assessment in which the threat analysis generates the initiating events for the vulnerability assessment. Garrick et al. (2004) remark that “initiating events are application dependent and require extensive involvement of experts—those who develop and analyze intelligence and those who are expert in the nature of the threat …” (p. 136). They illustrate and discuss their methodology with examples from various large critical infrastructures that need to be protected against terrorist threats.
The Electric Power Research Institute (EPRI) methodology uses the following definition for risk:
Risk = frequency of threat occurrence × probability of threat success (given the threat occurrence) × probability of consequence.
The frequency of threat occurrence is the most challenging factor to estimate quantitatively. EPRI suggests that the frequency of large-scale terrorist threat scenarios can be estimated by leveraging the national terrorism experience base. In the EPRI study, considerations in estimated frequency of terrorist threat included the following:
- Large-scale threats require significant planning and preparation to execute, so they have an annual frequency of occurrence in the United States of less than 1.0 per year.
- There are many potential high-profile targets in the United States other than nuclear plants.
- If nuclear plants are chosen as a target, only 1 out of the 103 (then) operating nuclear reactors would be expected to be attacked.
As an outgrowth of the EPRI study, the U.S. Department of Homeland Security (DHS) supported a more simplified, semiquantitative conditional risk assessment process for particular threats of interest to them; this process is referred to as the Risk Assessment and Management for Critical Asset Protection (RAMCAP) process.6 RAMCAP was used to support an industry-led assessment of risks at each U.S. nuclear plant including reactors, spent fuel pools, and dry cask storage. Tabletop assessments were conducted for a spectrum of postulated security threats. Insights gained from the analysis included the following:
- Important risk scenarios are site specific.
- Compliance with the design-basis threat does not necessarily ensure negligible risk.
- Modest changes in design and/or procedures can make the facility more resistant to security threats and reduce health and/or economic consequences due to security threats.
Hirschberg et al. (2016) proposed an analytic approach that leverages intelligence community knowledge to derive quantitative risk estimates for terrorist threats with potential for catastrophic consequences. Their estimates are based on three elements:
- Probability that an attack is conducted. This estimate is derived based on historical evidence of attractiveness of the target and evidence of terrorist activity in the country of interest.
- Probability that a given terrorist scenario can be successfully implemented. This estimate is based on assessments of the required resources, time, know-how, and countermeasures in place.
- The consequences of an attack in terms of fatalities, injuries, and land contamination.
The approach enables diverse sources of knowledge to be integrated within a common framework to generate a more complete picture of the likelihood of a successfully executed attack and the resulting consequences. Sources of knowledge include expertise from the political sciences and intelligence communities on the motivations of terrorists; knowledge from the military and security communities on scenario planning; and physical assessments of the performance of the engineered systems to derive consequences.
The authors acknowledged that the risks associated with natural and engineered systems can be assessed with higher confidence than the motivations and actions of terrorists. Nevertheless, the analyses can provide useful insights in spite of the large variation in uncertainty. For example, the graphs (e.g., Hirschberg et al., 2016, Figure 12) presented by the authors indicate that there is considerable variability in predicted risk and suggest that the terrorist threat should not be neglected compared to accident risks for both hydro and nuclear facilities.
There have been a number of other approaches to security risk analysis including work by Clauset et al. (2007), Willis et al. (2007), and Willis and LaTourrette (2008).
The usefulness of PRA for assessing security risks has been discussed and debated at length in the technical literature, including in reports from the NRC.7 One common line of argument against the application of PRA to security has to do with the lack of knowledge of adversaries and their capabilities, motivations, and strategies. This creates challenges for developing a complete set of attack scenarios as well as estimating attack probabilities.
For example, the NRC’s Committee on Risk-Based Approaches for Securing the DOE Weapons Complex (NRC, 2011) expressed reservations with respect to quantification of risk largely because of difficulties in defining attack strategies that adversaries might employ and their success probabilities. The committee did, however, note that some of the tools and techniques associated with risk assessment, particularly the structured thinking process, could be useful for developing a comprehensive “systems” approach to security.
Attack probabilities are widely acknowledged to be the most challenging to estimate because they require knowledge, data, or modeling of the motivations, capabilities, and intentions of terrorists. All such estimates will benefit from guidance from knowledgeable experts, for example, members of the intelligence community who have the appropriate personnel security clearances to access sensitive national security information on terrorist threats. In light of the paucity of historical data8 for terrorist attacks on
7 Articles by Ezell et al. (2010) and Brown and Cox (2011a,b) provide good discussions of the advantages and disadvantages of applying PRA to security.
8 Two publicly accessible data sources are the Global Terrorism Database (GTD) (http://www.start.umd.edu/gtd/) and the RAND Database on Worldwide Terrorism (http://www.rand.org/nsrd/projects/terrorism-incidents.html). About 0.02 percent of the entries (26 out of 140,000 in the GTD) make reference to commercial nuclear facilities. The committee did not examine classified or private databases on terrorism, sources that analysts with appropriate security clearances would be able to utilize.
commercial nuclear facilities, from which one might hope to derive attack frequencies, such reliance seems essential. Estimates of attack probabilities by experts can be provided as distributions. While the resulting estimates may have large uncertainties, they would be informed by the best available knowledge at the time of the analysis. Appendix 5A provides a discussion of expert elicitation methods.
The Committee to Review the Department of Homeland Security’s Approach to Risk Analysis (NRC, 2010) expressed reservations regarding the possibility of conducting an all-hazards risk assessment that combines risks associated with natural hazards with security risks. However, the committee was more optimistic about using an integrated approach if the goal was to compare the benefits of multiple alternative options for reducing risks. They pointed out that an integrated analysis might illuminate options for simultaneously reducing the risks arising from natural hazards and terrorism.
NRC (2010) also recommended that “DHS should strengthen its scientific practices, such as documentation, validation, and peer review by technical experts external to DHS. This strengthening of its practices will also contribute greatly to the transparency of DHS’s risk modeling and analysis. DHS should also bolster its internal capabilities in risk analysis as part of its upgrading of scientific practices” (p. 3).
Another line of argument against the application of PRA to security is that the probabilities associated with the likelihood of particular threats may shift in response to defensive actions. Terrorists, unlike natural hazards or engineered systems, are intelligent adaptive adversaries. The probability of an earthquake will remain fixed whether or not steps are taken to mitigate its consequences. However, the probability of a terrorist attack against a facility might change in response to protective or mitigative actions that make it a less attractive target.
This line of argument was made by the Committee on Methodological Improvements to the DHS’s Biological Agent Risk Analysis (NRC, 2008). This committee reviewed DHS’s tool9 for assessing the risks associated with the intentional release of biological threat agents. The committee argued that terrorist threats, unlike natural hazards and engineered systems, are intelligent, goal-oriented, resourceful, and adaptive adversaries. Consequently, PRA methods that rely on static event trees and associated probabilities are not appropriate for modeling adversary strategy sets. The committee argued that DHS should use decision-oriented models “that explicitly recognize terrorists as intelligent adversaries who observe U.S. defensive preparations and seek to maximize achievement of their own objectives” (NRC, 2008, p. 3).
9 The tool is referred to as the Biological Threat Risk Assessment.
Brown and Cox (2011b) further elaborated this line of argument. They noted the following:
“The capacity of terrorists to seek and use information and to actively research different attack options before deciding what to do raises unique features of terrorism risk assessment that are not adequately addressed by conventional PRA for natural and engineered systems—in part because decisions based on such PRA estimates do not adequately hedge against the different probabilities that attackers may eventually act upon.” (p. 196)
They argued that the very existence of a PRA that suggested differences in attack likelihoods might cause attackers to change their behavior, negating its value.10
Ezell et al. (2010) and Ezell and Collins (2011) addressed the challenges of modeling attack strategies of intelligent adversaries that were raised by NRC (2008) and Brown and Cox (2011b). They acknowledged the added complexity of modeling adaptive adversaries, but they also argued that one could develop a baseline of current terrorist motivations, intent, and capabilities and facility defenses and assess probabilities conditional on this baseline. In other words, event trees in a risk assessment can be thought of as a snapshot in time of threats, vulnerabilities, and consequences that are subject to change as an attack scenario progresses. Of course, this snapshot may be incomplete and can have large uncertainties. Once new defensive (i.e., preventative, protective, or deterrence) measures are introduced, event trees and their associated probabilities are reassessed and updated as needed.
The present committee is acutely aware that its Finding 4.1 is a substantial departure from previous conclusions of other NRC committees on the use of risk assessment in security applications. In developing this finding, the committee examined advances in risk assessment science and practice since the previous NRC spent fuel study (NRC, 2004, 2006) and deliberated on the potential for future advancements. Some of these advances are cited and discussed in this chapter. This finding is intended to encourage the nascent efforts by the USNRC and nuclear industry to develop security risk assessments—and also to encourage their further development and application by the broader risk assessment community.
The present committee recognizes that additional work will be required to further develop security risk assessment methodologies. Work is particu-
10 Of course, security risk assessments contain security-related information and are therefore not publicly releasable.
larly needed to develop and exercise processes for estimating the elements of the risk triplet (i.e., scenarios, likelihoods, and consequences); estimating their uncertainties; and appropriately communicating results, including uncertainties, to decision makers. The usefulness of risk assessment for informing resource allocation, design and operational enhancement, and regulatory decisions will improve as these elements and associated tools are further developed.
Even at their present stage of development, however, security risk assessments can be useful for making relative comparisons of design or operational alternatives within a particular system/facility or between facilities—particularly when analyses are conducted by the same group of experts applying comparable assumptions. Such assessments could help to identify potential gaps in the current security frameworks and reveal vulnerabilities that are missed by conventional security assessments. See, for example, the present committee’s Recommendation 4.11 (Chapter 7) for assessing the risks of storing spent fuel in pools versus dry casks.
The present committee concurs with Ezell et al.’s (2010) arguments about the usefulness of PRA for security assessments. One can construct event trees and assign probabilities based on expert judgement with the full understanding that base probabilities can change when different types of preventive, protective, or deterrence actions are considered. The identification of scenarios may be incomplete, and the probability estimates may have large uncertainties. Nevertheless, there is no fundamental technical limit to performing a quantitative analysis, even though the probability calculations will be more complicated and will need to account for adversary behavior.
The present committee also concurs with NRC (2010) about the usefulness of PRA for considering multiple alternative options for reducing risks. Moreover, that report’s advice to DHS about strengthening its scientific practices for security risk assessment (see Section 5.4) is also applicable to the USNRC and the nuclear industry. Such strengthening can improve analytical rigor and transparency and help to advance the state of the art in risk modeling and analysis.
There is a variety of risk analysis frameworks that could be used to support terrorism risk analysis, either as inputs to a PRA or as additional complementary perspectives to support decision making. Most particularly, as recommended by the Committee reviewing the DHS Bioterrorism Risk Assessment (NRC, 2008), it would be advantageous to explore frameworks that model terrorists as intelligent adversaries. Potential approaches for representing the beliefs and behaviors of intelligent adversaries include use of decision trees, attack trees, Bayesian belief networks, game theory, and agent-based models (see Ezell et al.  and Brown and Cox [2011a] for further discussion of alternative representational frameworks and models).
Terrorists, unlike natural hazards and engineered systems, are intelligent agents that will condition their behaviors on the behaviors of defenders. Thus, a major challenge for adapting PRA to security has been the quantification of adversary actions. Although there may be great uncertainty in how these actions evolve and, arguably, in the ability to quantitatively produce meaningful risk estimates based on these actions, the present committee nevertheless judges that it is worthwhile to explore security risks using relevant methods, concepts, and tools arising from the risk triplet framework as an important adjunct to the conventional approaches that are now used by the U.S. nuclear industry and its regulator.
Sidebar 7.4 in Chapter 7 illustrates the added complexities that arise when considering intelligent adversaries versus natural hazards or engineered systems. Earthquakes are just as likely to occur during any operating cycle of a nuclear plant, but terrorist attacks may be most likely to occur during certain operating cycles. Security risk assessments would need to recognize, represent, and numerically propagate this added level of behavioral complexity.
Much remains to be learned about the effectiveness of deterrent or delaying actions as well as the potential consequences should an attack succeed. Quantitative evaluations, however crude, could help the nuclear industry and its regulator develop strategies for preventing and/or mitigating terrorist attacks.
The continued expression of terrorist threats in society, including cyber and insider threats, underscores the need to develop improved approaches for understanding, preventing, and mitigating them, particularly threats directed against civilian nuclear facilities. Indeed, it would be imprudent not to consider the potential benefits of risk assessment, which has served to advance understanding and management of safety risks, to nuclear plant security. Moreover, only by developing and testing risk assessments through specific applications will its limitations become more fully understood and improvements to overcome them will be made.
Expert elicitation is a process for obtaining and synthesizing the judgments of subject-matter experts when the available knowledge base (e.g., empirical data and formal models) is incomplete, unreliable, uncertain, or open to alternative interpretation. Expert elicitation can be used to
- Predict future events;
- Provide estimates on new, rare, complex, or poorly understood phenomena;
- Integrate or interpret existing information; or
- Determine what is currently known, how well it is known, or what further exploration is needed about a subject or field.
Expert elicitation has been applied in a wide range of fields including probabilistic seismic hazard analysis (Budnitz et al., 1998), analysis of the health impacts of air pollutants (Cooke et al., 2007), assessment of the impact of new train technologies on human reliability (Wreathall et al., 2004), and analysis of risks of terrorism and the effectiveness of security policies to reduce those risks (DHS Bioterrorism Risk Assessment as reported by Ezell et al., 2010).
Expert elicitation can be used in risk assessment to obtain expert judgments about the three elements of the risk triplet:
- What can go wrong? (scenarios)
- How likely is that to happen? (likelihoods)
- What are the consequences if it does happen? (consequences)
Expert elicitation can also be informal or formal. Informal methods for eliciting expert judgements, although sometimes producing good results, are usually problematic because they have no built-in controls for bias, relevance, and consistency of knowledge across experts or variability in interpreting the questions posed. The Proliferation Resistance and Physical Protection Evaluation Methodology Working Group of the Generation IV International Forum (GIF, 2011) notes, “Without a formal process and strong controls, experts may be asked to provide judgments on issues that go beyond their expertise, or their estimates might be combined in misleading ways which distort the results” (p. 67).
More formal structured expert elicitation methods have been developed to overcome the limitations of informal methods (Budnitz et al., 1998; Keeney and von Winterfeldt, 1991; Morgan, 2014). Formal expert elicita-
tion is a structured process that makes use of people who are sufficiently knowledgeable about particular subject areas to make meaningful assessments. Key elements of formal expert elicitation processes include
- Careful selection of experts to ensure broad representation of relevant areas of expertise and perspectives;
- Training for elicitation, including sensitization to known cognitive biases;
- Providing models and tools to support problem formulation and exploration;
- Providing opportunity for extensive, highly structured expert interaction to maximize a shared understanding of the available relevant empirical database, models, and reasoning processes; and
- Uncovering and documenting areas of clear agreement as well as legitimate diversity of assessment.
The use of formal processes for expert elicitation has improved the credibility and acceptance of expert judgment because of the rigor and transparency of the results (Budnitz et al., 1998).
A good example of an expert elicitation application is provided by Budnitz et al. (1998). They developed and exercised methodological guidance on how to perform a probabilistic seismic hazard analysis that relied heavily on expert elicitation. Expert elicitation was needed in this case because there were major limitations in the research community’s understanding of the mechanisms that cause earthquakes and the processes that govern how an earthquake’s energy propagates, despite advances in seismic knowledge. The authors leveraged the knowledge of experts in seismic analysis to develop estimates of the likelihood of various levels of earthquake-caused ground motions at particular locations for a given future time period that reflected the current state of knowledge.
Among the major methodological contributions the authors made to the expert elicitation literature was to explicate the various types of consensus that can exist in a group of experts. For example, experts may all agree on the same deterministic model or the same value for a particular variable. Alternatively, experts may differ on particular models or parameter values but agree that a particular composite probability distribution represents the composite beliefs of the overall scientific community.
Budnitz et al. (1998) argued that it is far easier to get a group of experts to agree on how to represent the informed community’s diversity of beliefs about a scientific issue than it is to get them to agree on the resolution of a particular technical issue. As a consequence, the probability distributions produced using formal expert elicitation can produce an accurate representation of the level of uncertainty associated with particular likelihood assessments.
The successful application of expert elicitation methods to security risk assessments depends on the rigor and transparency of the methodologies employed. Of particular importance is the selection of experts who collectively possess the necessary range of expertise needed to develop scenarios, event/fault trees, likelihood estimates, and uncertainty ranges. This would include individuals from the intelligence community with access to knowledge about terrorist motivations, intent, and capabilities. It would also include individuals from the security community who understand capabilities to prevent or respond to attacks, as well as experts in physical systems to assess the physical consequences of (low-likelihood) successful attacks.
Another key element of expert elicitation is to provide models and tools to support problem formulation and exploration. In the case of terrorist threat this includes providing models and tools that encourage consideration of terrorists as intelligent, goal-driven adversaries that gather information about our own defensive preparations and seek to maximize the achievement of their own objectives. The CARVER analysis method described in Appendix 5B is one example of a tool that encourages domain experts to consider the attack space from the perspective of an intelligent, motivated adversary. Decision trees, attack trees, and game-theoretic formulations are other examples of models and tools that can provide structure for eliciting, representing, and exploring the consequences of interaction among multiple intelligent agents that include adversaries and defenders. Game-playing exercises (e.g., red teams and cyber hacking teams) may also be useful.
The USNRC has considerable experience in the use of expert elicitation and has issued guidance documents on its use (Kotra et al., 1996; see also Budnitz et al., 1998; Forester et al., 2007). The USNRC can draw on this experience to develop security risk assessments for which limited observational data are available (Frye, 2013). Additionally, there continues to be active research on methods for eliciting and combining expert assessments to improve the sharpness1 and reliability of estimates that can also be drawn upon (Mellers et al., 2015; Satopää et al., 2014; Wang and Bier, 2012).
1 The sharpness of an analysis describes the ability to differentiate among its results or outcomes. The greater the sharpness of the analysis, the greater the confidence that can be placed on the uncertainty ranges of its outcomes. See Satopää et al. (2014) and Gneiting et al. (2007).
The WWII U.S. Office of Strategic Services developed a targeting doctrine for optimizing scarce resources in attacks on the German military in then-occupied Europe. This targeting methodology has been used by U.S. Special Operations for decades to plan for small unit raids and has been employed by the DHS and numerous international private security enterprises. It can also be used as a qualitative vulnerability analysis tool. This methodology is referred to as the CARVER Vulnerability Assessment Methodology. This methodology has been used by the U.S. government for more than 40 years (USNRC, 2007a, Enclosure 5; see also Bennett, 2007). CARVER is defined as follows:
CARVER = Criticality + Accessibility + Recuperability + Vulnerability + Effect + Recognizability1
The CARVER factors are described in the Department of the Army Field Manual (AFM, 1991). The following descriptions of a CARVER analysis are taken from that manual:
- Criticality: A target (or target-specific critical node) is considered critical when its partial or complete destruction has significant military, political, psychological, or economic operational impacts. Evaluation of critical nodes or single points of failure associated with a given target is done within the context of the target’s primary mission.
- Accessibility: A target is considered accessible when sufficient personnel and equipment can physically emplace explosives or other devices or employ stand-off weapons to degrade or destroy it. The evaluation of accessibility requires the identification of critical operational paths to achieve the mission objectives and factors that can aid or impede target access.
- Recuperability: A target’s recuperability is the length of time and level of effort required to repair, replace, or bypass damage or destruction and restore mission capability.
- Vulnerability: A target is vulnerable if the attacking force has the means and the expertise to achieve the desired effect. When determining the vulnerability of a target, the ability to disrupt or
1 George C. Marshall Center for Security Studies, The Executive Course, Distance Learning Seminar #732, Terrorism and Its Implications for Democratic States (http://pubs.marshallcenter.org/732/lesson1/lesson1.html).
destroy a critical component is compared with the attacking force’s operational capabilities and weaponry.
- Effect: An attack on a target may have desirable as well as undesirable military, political, economic, psychological, and sociological impacts. Effect is a measure of these impacts. Planners can use this factor in conjunction with the criticality factor to select particular targets for attack. From a terrorist’s perspective, special consideration is given to the effect on local populations, potential for media interest, potential psychological and sociological impacts, and possible effects on the target nation’s political and economic systems.
- Recognizability: The target is recognizable if it can be identified by the attacker(s), intelligence collection team, or reconnaissance element under varying conditions and circumstances.
The CARVER selection factors assist in determining the best targets or components of targets to attack. Each of the factors above is given a numerical value for the targets being considered. This value represents the desirability of attacking the target. The values are then placed into a decision matrix and summed for each target. The targets with the highest values are considered to be the best targets to attack, giving consideration to existing and future operational constraints.