Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
4 A Risk-Informed Approach: Procedures and Criteria for Risk Assessment to Support an Exemption Process In this chapter, the committee explains the key attributes of the risk assessment process that it recommends and outlines steps by which risk assessment can be performed and integrated into a process to inform de- cisions on the alternative dispositions of certain high-level and tran- suranic legacy wastes. The intent of this chapter is to describe the characteristics and provide an example of a process for risk-informed decision making in support of exemption determinations, not to prescribe the specific process. 4.1 USING RISK ASSESSMENT The general methods of risk assessment for environmental concerns are well established in numerous earlier publications. These include a series of reports on risk assessment methods and processes by the Na- tional Research Council (NRC, 1983, 1994a, 1996). Other groups and individuals have written similar general texts on principles of risk analy- sis (e.g., PCCRARM, 1997a, 1997b). Agencies of the U.S. government 87
88 RISK AND DECISIONS ABOUT TRU AND HLW have codified specific methods for performing risk calculations in par- ticular applications. For example, the U.S. Environmental Protection Agency (U.S. EPA) has specific guidance for performing risk assess- ments acceptable for identifying clean up plans under CERCLA (U.S. EPA, 1989, 1990a, 1990b, 1991a, 1991b, 1991c, 1991d, 2001a, 2001b). The U.S. Nuclear Regulatory Commission (U.S. NRC) also has recom- mended methodologies for carrying out performance assessments for low-level radioactive waste disposal sites and for nuclear reactors (see, e.g., U.S. NRC, 2000). Appendix A summarizes this conceptual back- ground on some of the existing literature on risk assessment and risk management procedures. The approach described here conforms to the general principles and methods that have been established for risk analy- sis, but it has been crafted specifically to support the exemption process for high-level waste (HLW) and transuranic (TRU) waste described in Chapter 3. The risk terminology used in this report, including the distinc- tion between risk assessment and risk analysis, is described in Sidebar 4.1. Sidebar 4.1: Risk Terminology Risk analysis is an umbrella term that includes risk assessment, risk communication, and risk management. Risk assessment is the scientific evaluation of known or potential ad- verse effects for an individual, group, society, or the environment result- ing from exposure to hazards. The risk-assessment process consists of the following steps: (1) hazard identification, (2) hazard characterization, (3) exposure assessment, and (4) risk characterization. The definition includes quantitative risk assessment, which emphasizes reliance on numerical expressions of risk, qualitative expressions of risk, and charac- terization of the attendant uncertainties (see Appendix A for a more de- tailed definition from the NRC, 1983). It is distinguished, also per NRC (1983), from risk management, although good practice requires iteration between risk assessment and risk management. Note that in this report the terms analysis and assessment are not used as shorthand for risk analysis and risk assessment. Risk management is âthe process of evaluating alternative regulatory actions and selecting among them. Risk management, which is carried out by regulatory agencies under various legislative mandates, is an agency decision-making process that entails consideration of political, social, economic, and engineering information with risk-related informa-
A RISK-INFORMED APPROACH 89 tion to develop, analyze, and compare regulatory options and to select the appropriate regulatory responseâ (NRC, 1983). Refined risk assessment: An iterative approach to risk involves doing an initial risk assessment with available data and models. Sensitivity analysis is applied to identify aspects of the assessment that require greater complexity or precision before a decision can be made from the assessment results. After further data collection and/or model develop- ment indicated by the initial assessment, a refined risk assessment is prepared using the more complete or detailed information and models. A conceptual model is a description of the specific processes (e.g., chemical, physical, biological) that are believed to govern the behavior of the system to be assessed. In the case of contaminants in the ambient environment, site-specific geological, biological, and climatological condi- tions may make different processes relevant in different locations. For quantitative assessments, the conceptual model determines the types of equations and parameters that should be used to make estimates of con- taminant release, transport, transformation, and human exposure in a particular location, whereas the conceptual model determines how sys- tem behavior is described in qualitative assessments. Once these are defined, then it is possible to start to identify appropriate mathematical models to build, adapt, or adopt for use in the risk analysis, and the rele- vant types of data to start to assemble or collect. If there is uncertainty or disagreement about the conceptual model, then it may be important to employ alternative models in the risk analysis. When alternative conceptual models are considered, data collection and analysis may also be targeted to allow evaluation of which model or models are most supported by the data. If more than one model provides a similar quality of fit to the observations, risk estimates should be made from the multiple models. A risk model is a mathematical representation of the conceptual model. Once formulated mathematically, and estimates of all the neces- sary parameters and input data are developed, the conceptual under- standing can then be used to simulate or predict the behavior of contaminants in the system, to estimate the risks that may result from them. Mathematical models of the same processes and behavior can be developed with many levels of complexity. Since many of the relevant parameters are likely to be highly uncertain, one should begin with a simple model and use it to explore which uncertainties affect the risk es- timates most. A refined risk model is created by progressively adding mathematical complexity or more precisely estimated parameters and input data to an initial risk model. Such refinements should be guided by sensitivity analysis results using the initial model.
90 RISK AND DECISIONS ABOUT TRU AND HLW System behavior: When speaking of an environmental issue, the âsystemâ is the set of all environmental components that interact with each other, including the chemical compounds in the soil, the physical features of the land, the organisms that live in the subsurface and at the surface, and the waters and animals (including humans) that move through the area. The system may also include the air and atmospheric interactions. The âbehaviorâ of the system refers to the way that these interactions occur when a change is made, such as the addition of a con- taminant to the environment. Taken on its own, this chapter may appear to oversimplify the true complexity of the analyses that will have to be performed in the course of the process. The committee cautions that merely performing each step identified in this chapter in a âcookbookâ fashion is not sufficient for a successful application of the risk-informed approach. In particular, if the analyses associated with each step ignore or attempt to diminish the im- portance to decision making of the underlying complexities and un- knowns of the processes being modeled, the risk assessment supporting the risk-informed approach will likely be rejected (see Sidebar 4.2). Chapter 5 provides more discussion of these complexities, and describes how the committee thinks they should be incorporated into the risk analysis process. Considering carefully the issues that are discussed more fully in Chapter 5 and preparing a plan that consciously addresses them are essential to this process. 4.2 CHARACTERISTICS OF A GOOD RISK-INFORMATION APPROACH Chapter 3 notes the importance of creating an exemptions process 1 that is transparent, inclusive of stakeholders, and credible. These charac- teristics must pertain to the exemption-seeking process as a whole, but they are also important attributes of a good risk analysis in its own right. In addition to procedural considerations, achieving each of these charac- teristics places specific technical demands on the way that a risk analysis is conducted. Specifically, the risk analysis conducted in support of an exemption application must be as follows: 1 The term "stakeholder" here includes the interested and affected public.
A RISK-INFORMED APPROACH 91 Sidebar 4.2: Challenges and Difficulties in a Risk-Informed Approach Risk assessment sometimes is prescribed as a universal solution for problems faced by decision makers. Such thinking is counterproductive. Risk assessment is a tool that can help decision makers reach a solu- tion, but it is difficult to use well and does not guarantee a satisfactory outcome. Chapter 5 details some of the technical and institutional limita- tions of risk assessment, but it is worthwhile to mention two of them here as a cautionary note to give a fuller picture of the risk-informed process. Some people may question the credibility of risk assessments. This may happen because the analyses in a risk assessment are not credible, because the institution presenting the risk assessment is not credible, or for both these reasons. At its information-gathering meetings, the com- mittee repeatedly questioned the experts presenting results of risk as- sessments about how they had established the validity of the models they were using. Few presenters were able address the questions, and fewer still gave satisfactory responses. People without the time or re- sources to critique the analyses may use the credibility of the institution as a proxy for evaluating the credibility of the analyses. It is rare (and indeed suspicious to some) for an environmental risk assessment to yield results with small uncertainties. Uncertainties often are so large that the results of a risk assessment must be deemed inde- terminate. Even with these problems, however, the committee believes a risk- informed approach using risk assessment as a structured method to de- velop an understanding of, and to characterize, risks is the most promis- ing approach. ⢠Logical. The sequence of steps of the risk assessment must map out a coherent chain of cause-effect relationships that link as- sumptions about the handling and final disposition of the waste stream in question to a set of outcomes that are specifically iden- tified as relevant to the exemption decision. ⢠Well founded. The methods used in each step must be consistent with current scientific knowledge and practices. If the state of science has competing theories, the risk analysis must consider appropriate alternative approaches, rather than choosing one, unless it can be demonstrated that such a simplification would
92 RISK AND DECISIONS ABOUT TRU AND HLW not alter the estimates of outcomes sufficiently to affect the deci- sion between alternative waste disposition options. ⢠Transparent and traceable. Risk analyses can become too complex, obscure, or inaccessible for even knowledgeable out- side parties to understand, review, or reproduce. It is useful to re- sist this tendency by giving care from the beginning to make clear the steps, assumptions, and data that are being used. Identi- fying and making available sources of information, as well as identifying and characterizing uncertainties, enables others to check them. Documentation after the analysis is completed does not meet this requirement. ⢠Participatory. Most of the parties interested in the outcome of the risk analysis will not be engaged in performing the analysis itself, but they will not trust the results if they are not allowed to participate actively in discussions about how to structure and re- fine the analysis. Transparency also cannot be achieved unless stakeholders participate in the process from the beginning (NRC, 1996). The key feature of the risk analysis process described in this section is that the data, modeling, and any other calculations in estimating risk must be structured to inform a specific and well-defined decision. A coherent and efficient risk assessment requires that the decision criteria be well defined in advance of any computational or modeling steps. It also requires that a sufficient number of options from which to choose be considered in the decision to avoid excluding potentially superior op- tions. Therefore, the first step is that a decision be defined; and second that a list of decision alternatives from which to choose be considered. In this application, the options would be a list of two or more methods for disposing of a particular high-level or transuranic waste form. For example, if the Department of Energy (DOE) is seeking an ex- emption related to closure of a set of HLW tanks, the options must in- clude, at a minimum, (1) complete removal of all HLW for disposal in a deep geologic repository, and (2) a specific plan for how much of tank residuals may be left in a tank and any associated form of stabilization (e.g., grouting) suggested as an alternative disposition that might be al- lowed as an exemption. DOE may wish to consider more than one alter- native disposition, but there must be at least the default and one alternative listed.
A RISK-INFORMED APPROACH 93 The critical issue is that all the options be defined specifically before analysis starts. The second step is to identify all of the information needed for making a decision. That is, what are the types of risks, costs, and other criteria must be considered when determining whether one or some subset of the disposition options would be preferable to the others? As stated earlier, human health risks are an important consideration at any site, and other considerations could have equal or greater importance at some sites. Meaningful quantitative analysis cannot be initiated until these two structuring steps have been completed. The risk assessment can then be designed to address the specific set of relevant considerations for the specific disposition options that have been set forth. This discussion emphasizes good process rather than good decisions. The committee, unfortunately, cannot prescribe good decisionsâgood decisions are not definable. A good process is necessary but not always sufficient for a good decision, and can still lead to bad decisions and generate a great deal of outrage. An agency may follow a processâ posting a preliminary decision for a required amount of time, meeting a required number of times, listing public comments in a revised docu- ment, and issuing a final decisionâbut essentially undermine the value of the process by making it simply procedural, checking the box off and proceeding on its predetermined course. For the risk-informed approach to be successful, the agencies must embrace it as something not only use- ful and meaningful, but also essential to its planning. Risk assessments can be extremely complex, yet shed little light on whether one cleanup option is to be preferred over another. At the same time, the complexity can be a barrier to gaining confidence and trust in any recommendations that may purportedly be justified by the risk analy- sis. The approach described here embraces the principle that analytical detail and complexity should be limited to the minimum necessary to 2 distinguish the best option or options. If the relative risks and trade-offs can be established with a relatively simple analysis that is nonetheless grounded in empirical data, then this simpler analysis will be easier to explain and easier for non-analysts to understand. A simpler analysis is also easier to validate. This will allow the risk analysis process to move more quickly to the point at which there is communication among stake- holders on how to make the relevant trade-offs. By fostering such discus- 2 More discussion of the philosophy, structuring, and key elements of the ap- proach described here can be found in NRC (1996), Howard (1966), and Chap- ter 3 of Morgan and Henrion (1990).
94 RISK AND DECISIONS ABOUT TRU AND HLW sions early in the process, those managing the process can ensure that stakeholder input is considered and incorporated into the final product. It bears noting that the risk assessment developed in this process will probably have considerable complexity before it is completed. Also, con- siderable technical sophistication is needed to produce even a âsimpleâ analysis that can effectively guide the addition of detail. The critical point is that the quantitative analysis be initiated with minimal complex- ity and that the complexity be increased gradually, as the need for it is determined and as all participants gain understanding of the key elements 3 of the analysis that are driving its risk estimates. It is also possible that the forms of complexity resulting from this it- erative process will be quite different from the forms of complexity that would have been added if analysts were to strive to incorporate a final degree of complexity from the outset. For example, risk analysts may set out with models that incorporate detailed dynamics, yet it may turn out that only steady-state outcomes are important to the decision. Wasted effort on details that are merely distracting can be avoided in a process that is guided by insights from an initial, relatively simple analysis. The process of analysis may be more important in achieving trans- parency, trust, and understanding than thorough documentation of an analysis that is offered as a fait accompli, particularly if the final product on which decisions are made is complex. The committee observed in site visits and related presentations that DOE has tended to generate its analyses without such an iterative, staged process. DOE appears usually to present only the final product of its risk analyses to stakeholders, at a point where the analysis is exceedingly complex and difficult to review 4 or gain intuitive understanding. Focusing on the numerical results can 3 Sensitivity analysis is the key method advocated here for determining whether additional complexity is desirable. The junctures where sensitivity analysis is best performed are identified in the step-by-step process that is pro- vided in this chapter. 4 The Risk-Based End States (RBES) process, mentioned in Chapter 1 and described in greater detail in Appendix B, is an example of a process in which stakeholders were not engaged before apparent decision recommendations had been formed. However, there appears to have been little or no risk assessment underlying that process. In other cases, the committee was presented with risk analyses that had much detail, but failed to provide decision-relevant informa- tion such as sensitivity to key assumptions. The presentation on risk from resid- ual radioactive waste in groundwater at Hanford was a case in point (Thompson, 2004).
A RISK-INFORMED APPROACH 95 also obscure one of the most important products of the analysis: Beyond risk estimates, risk analysis can produce insights, which may be highly qualitative. One must also distinguish a âsimpleâ analysis from one that ignores uncertainty. Uncertainty is a major concern in making estimates of risks, particularly when considering long time scales. Key areas of uncertainty must be explored even in the first phase of the analysis. Using alternative ways of representing physical processes or alternative sets of assump- tions helps to address model uncertainty and to understand the sources of sensitivity and boundaries of the risk estimates. That is, one can learn how much risk estimates associated with each disposition option are likely to vary as a result of unknown factors or areas of disagreement. This can be accomplished using quite simple analyses; indeed, more complex analyses can present significant barriers to exploration of uncer- tainties. Identifying the boundaries of uncertainty created by alternative assumptions is more important than adding computational complexity while still relying on a single set of deterministic assumptions. Many of the risk analyses presented to the committee by DOE have indicated that the opposite has been the practice (see, e.g., DOE, 2002c; Thompson, 2004). Thus, there are several critical elements to the process for imple- menting a risk-based approach for deciding on the disposition of high- level and transuranic wastes: ⢠specific set of options to be considered, ⢠list of information or data needed for a decision, ⢠set of criteria for determining the best option(s) developed in ad- vance of any analysis, ⢠risk computations performed at the minimal level of complexity necessary to separate options according to the decision criteria, ⢠risk computations performed simulating competing views of the physical processes and using ranges of parameter values that re- flect the state of science, and ⢠uncertainties explicitly explored and retained unless they are demonstrated not to affect the relative ranking of the disposition options.
96 RISK AND DECISIONS ABOUT TRU AND HLW These are the key elements of what one might call a decision- oriented risk analysis approach, but it bears repeating that the process by which these elements are put together is exceedingly important to achiev- ing acceptance of any decisions that may be informed by such a risk analysis. The process must be iterative and participatory (see Sidebar 5 4.3). The institutional impediments outlined in Section 5.4 must be con- sidered in determining who specifically performs the risk assessment and how it is managed and evaluated at an individual site and across the DOE complex. All of the elements of risk management (which have to be reflected in the final exemption application) are analyzed in parallel. The analysis of each element must meet consistently high standards of quality and credibility so that they will withstand the scrutiny of external review. 4.3 A SIX-STEP PROCESS FOR RISK-INFORMED DECISION MAKING To better elucidate how one can combine process and analysis, the rest of this section provides an outline of the steps that DOE could fol- low. Readers familiar with the CERCLA process may find that many of Sidebar 4.3: Decision Support Systems A âdecision support systemâ is designed to enable greater participa- tion by non-experts and so could support some of the goals articulated in this chapter. A decision support system, in this context, would be a user- friendly computerized version of the risk models and relevant decision criteria that could be run on a microcomputer with only modest resource requirements. Such tools are intended to provide participants in the proc- ess, particularly non-modelers, with an ability to alter assumptions and elements of the risk analysis independently, and to quickly observe the impacts of such âwhat ifâ questions on the outcomes viewed as relevant to the decision. This capability can enable modelers and non-modelers alike to gain hands-on familiarity with how the risk model responds to 5 A recent example where a decision support system was integrated into an it- erative and participative public decision process like the risk analysis process described in this chapter can be found in Passell et al. (2003).
A RISK-INFORMED APPROACH 97 alternative assumptions, and help in identifying the most important ele- ments of the risk analysis. If developed well, a decision support system can enhance the sense of empowerment and participation of stake- holders, it can enhance dialogue among stakeholders and modelers, and it can enhance the iterative aspects of the process. It is important also to note that decision support systems can be expensive and time- consuming to develop, because each one requires a well-designed graphical user interface with functions tailored specifically to the risk analysis in question. The software design and development challenges can divert attention and resources away from simply performing a good decision-oriented risk analysis process. Also, if the resulting software product does not meet expectations of performance (which can often become unrealistic), it can unnecessarily undermine the goal of engaging stakeholdersâ trust and understanding of the analysis. A decision support system is not essential for a good iterative and participatory process, but it can be a valuable supplement if it is created with a proper understand- ing of its role and limitations, and if appropriate software development resources are allocated. its elements are familiar and that the whole process is compatible with 6 CERCLA. However, the following steps were devised specifically to illus- trate a process for the transuranic and high-level waste disposition decisions that DOE may wish to have considered in an exemption application. Recommendation 3: DOE and its regulators for HLW and TRU waste should adopt a six step process for risk-informed decision making: (1) initiate the process, laying out viable options and poten- tial decisions; (2) scope the information and analysis; (3) collect data and refine models; (4) prepare refined risk assessment; (5) develop additional analyses and data collection, as needed, to support deci- sions; and (6) finalize the decision. 6 For more about the CERCLA process see Section 5.4 or the U.S. EPA web site, available at: http://www.epa.gov/superfund/.
98 RISK AND DECISIONS ABOUT TRU AND HLW Step 1. Initiate the Process, Laying Out Viable Options and Potential Decisions To keep the risk analysis efficient and useful, the first step is neither data development nor computational. Rather, it lays the stage for clearly defining the decision that the process will be addressing, and of initiating contact and discussion with stakeholders, states, and regulators so that they are involved throughout the entire process. The intent of this step is to develop the structure needed to carry out a useful risk assessment. The risk assessment will not be useful if it has the appearance of justifying a new disposition option that has already been decided on. The most im- portant aspect is that the analysis be set up as a comparative exercise, rather than an attempt to merely prove that the alternative does not pose unacceptable risks (see Sidebar 4.4). The real question is whether the alternative presents a better outcome with respect to the relevant criteria, which include tradeoffs among risk, cost, and other considerations. The alternative may have âacceptable risks,â but it is important to be able to understand how much it changes the relevant risks and costs in compari- son to the presumptive disposition. A comparative focus is important for building confidence in the risk-based approach as well. If DOE attempts to suggest that risks are âacceptableâ for an alternative, it puts itself in the impossible position of having to prove that its risk calculations are precise and that uncertainties are not large. Under a comparative ap- proach, DOE and stakeholders need only come to a consensus about which alternative, if any, presents a distinct improvement over the others in terms of the overall balance among decision criteria. Sidebar 4.4: Which Is Safer and What Is Safe? Decision makers and stakeholders planning environmental cleanups and waste management ask, or are asked, Which option is safer (yields a lower risk), and is it safe? The comparative question is helpful in select- ing among cleanup options and often has the advantage of believability: analysts and others may believe the relative risk or the risk ranking even if they mistrust that the absolute magnitudes of the risks have been cal- culated accurately. Relative risk also is adequate to answer the important question, Does the cleanup reduce risk? However, a relative risk calcula- tion generally is not, by itself, sufficient. Stakeholders, regulators, and responsible parties want to know if it is safe for present and future gen- erations.
A RISK-INFORMED APPROACH 99 Indeed, some see the question, Is it safe? as the crucial one. Deci- sion makers need only select among options that result in acceptable risks, and non-risk factors may override relative risk benefits if all of the options are acceptable from a risk perspective. If none of the options meets a threshold level of acceptability, then another option that does must be found. A risk-informed process may, then, incorporate both of these approaches, using a threshold test as a screening criterion and further risk reduction as a benefit to be weighed with other factors. Yet the question, Is it safe? presumes that common understanding of what safe means can be found. Then, even if a threshold or âbright lineâ can be found, one must trust not only the accuracy but the precision of the absolute magnitude of the risk assessment results for the threshold to be used as a screening criterion. Still, there is an advantage to using a threshold: decision makers confronting the relative risk question can get stuck in an endless process of adding and assessing options trying to identify which option is safest. The absolute magnitude approach re- quires only that there be one good option. If that option results in accept- able risks, other features and impacts of the option may be considered and, if they are acceptable, the option may be adopted. For this ap- proach to work, the option must be clearly acceptable from a risk per- spective. The following elements are part of process initiation: ⢠Define the problem and issues. What is the specific waste or waste area for which a decision has to be considered? What is its current presumptive final disposition? Why does DOE feel it makes sense to consider alternative disposition? What are the general types of alternative disposition options? These questions must be considered in advance, and those seeking the exemption must be prepared to discuss them with stakeholders before initi- ating any analyses or defining a single specific alternative that it wishes to have considered as an exemption. DOE will benefit from coordinating and making use of experts across the complex. ⢠Engage public and regulators to discuss and refine issues. The relevant stakeholders must be identified and contacted to ex- plain that DOE wishes to initiate an exemption application proc- ess. Stakeholders are not necessarily a well-defined group. While the process must be open to all, DOE and its regulators will have
100 RISK AND DECISIONS ABOUT TRU AND HLW to grapple with the difficult question of which elements of the public should be included in the process. The process must be inclusive, striving for a breadth of perspectives to be represented. A set of stakeholders that consists only of the most vocal advo- cates (who are the most easily identified stakeholders) is not likely to fill the need. Funding is needed for groups to participate (see Sidebar 4.5). Sidebar 4.5: Technical Assistance There is a mismatch between the budgetary, technical, and legal re- sources available to DOE and to the other process participants. With its annual budget of more than $7 billion (approximately $2 billion of which goes to Hanford alone), DOEâs Office of Environmental Management has funding almost equal to that of the entire U.S. Environmental Protection Agency and far more resources than any state regulatory body. Many American Indian nations, local governments, and citizens groups rely on volunteers to keep them involved in decision making processes (such as those under the National Environmental Policy Act and CERCLA) at DOE sites because few resources are available to them. There are mechanisms that provide funds for communities that seek to participate in these processes. Technical assistance grants of up to $50,000 are available to communities that form nonprofit organizations for the purpose of participation in CERCLAâs public processes. Although much of the funding might be expended in meeting the organizational and reporting requirements for such a grant, they could enable some groups to participate in ways that otherwise would be impossible. DOE has funded site-specific advisory boards or citizensâ advisory boards at most of its major cleanup sites, and these boards advise DOE on citizens concerns, values, and priorities in the cleanup program. His- torically, DOE has also provided funds for technical support to the states that were under consideration for hosting deep geologic repositories. In New Mexico, DOE funds supported the Environmental Evaluation Group, which provided technical oversight of the Waste Isolation Pilot Plant. To enable states and American Indian tribes that are affected by the federal high-level radioactive waste disposal program to participate in the activi- ties prescribed by the Nuclear Waste Policy Act of 1982, including review and oversight, the act requires that DOE grant funds to the affected gov- ernments. These funds are typically appropriated explicitly by Congress and administered by DOE.
A RISK-INFORMED APPROACH 101 The committee did not research these mechanisms in depth, but makes two limited observations based on its examination of risk-informed approaches and DOEâs waste management and cleanup activities. First, participation requires resources. Having access to experts, designated representatives who can dedicate time to following DOEâs environmental management of a site, and even the funds to hold public meetings makes for more meaningful participation by the affected parties. Mean- ingful participation makes a participatory, risk-informed decision process, such as is recommended here, more effective. Second, in the cases of assistance both to the state of New Mexico and the state of Nevada, funds for oversight activities were reduced or suspended. This has been perceived by some as an attempt to stifle critics (NNMCAB, 2004; Bin- gaman, 2004). Without arguing the facts or the reasoning for or against this position, the committee notes that it undermines rather than strengthens a participatory, public process if parties perceived as biased can limit the ability of the other parties to access independent technical expertise and engage in the process in a meaningful way. ⢠Define potential decisions and land use options. The list of po- tential decisions to be evaluated must, at a minimum, include the current presumptive disposition and an alternative disposition approach. It is possible that no specific alternative will have been identified, but that DOE feels strongly that the presumptive path needs to be modified. In this case, the initial phase of the process includes brainstorming with technical experts, but also giving stakeholders a chance to comment or to specify an alternative that could be considered. Even if a specific alternative has al- ready been identified, for the process to work an open explora- tion among experts and stakeholders is needed to see if other alternatives also merit consideration. If another option is identi- fied later in the process, the parties must go through the full process for the new option. ⢠Define list of criteria. DOE will need an initial list of the criteria that are relevant to decision making. As noted by an NRC com- mittee, âA risk characterization must address what the interested and affected parties believe to be at risk in the particular situa- tion, and it must incorporate their perspectives and specialized knowledge. It may need to consider alternative sets of assump- tions that may lead to divergent estimates of risk; to address so-
102 RISK AND DECISIONS ABOUT TRU AND HLW cial, economic, ecological, and ethical outcomes as well as con- sequences for human health and safetyâ¦â (NRC, 1996). The decision criteria will obviously include health risk, costs, and regulatory compliance. However, DOE will have to define the metrics of risk that it will allow to have a significant influence on the decision. Metrics of risk that could be used include excess lifetime cancer risk to an individual, population-wide cancer in- cidence risk, and so forth. It is important that the relevant metrics of risk be identified clearly at this early stage so that DOE can ensure that any analysis it performs will be designed to provide those specific outputs. Distributional outputs, or the question of who bears the burden of risks and costs, are usually relevant in such questions. Those seeking the exemption need also to work with stakeholders at this stage to determine how to define the groups of concern and what measures of distributional impact it will be appropriate to use. ⢠Develop process plan. DOE will need an outline of the process by which analysis and consultation will proceed. Such a plan would contain each of the remaining steps outlined below, but would include timing and parties to be involved. The plan would clearly identify points throughout the process at which new in- formation releases will occur. ⢠Review and feedback. In completing each of the elements above, DOE, of course, has to make initial internal deliberations to prepare for meetings, but all of the elements of the analysis must be discussed and refined in a consultative process with stakeholders. Consultations can be both informal and formal. Be- fore proceeding to the next step, DOE must be able to summarize all of the elements coherently and specifically. Step 2. Scope Information and Analysis Once DOE has defined two or more disposition alternatives to be compared, and the key criteria on which they will be compared, it can begin the process of estimating the specific risks that will be part of that comparison. However, the process is a gradual one, and the next step involves only a relatively simple set of risk calculations, which can be characterized as a scoping analysis, before consultation occurs again.
A RISK-INFORMED APPROACH 103 Scoping analysis results may presage the likely direction of the decision, but it is unlikely that any decision can be finalized at this stage. The ele- ments of the scoping step follow: ⢠Sketch out the structure of the risk analysis. At this point DOE will know what measures of risk it needs to calculate and can prepare a blueprint of the types of calculations that are re- quired to make these estimates. This will help identify data and models needed. It may also help identify models that are not relevant. Often a risk assessment proceeds using models that are readily available, and these may not always provide the actual type of information that is needed for the decision at hand. The result is a side-tracking of the analysis effort onto results that are not of much use for guiding the decision at hand. The point of this step is to identify what is needed and to keep all analysis ef- forts targeted to those needs. ⢠Identify parameters, data, and models required. When the risk analysis blueprint is completed, it will reveal the inputs that are required, and these become the basis of a list of data and models to develop. Judgments must be made for how to model the system(s) with respect to scale, complexity, whether to use deterministic or probabilistic methods, and which conceptual models might apply. At this stage, differences of opinion about appropriate modeling methods may come to light. For example, many models of subsurface contaminant transport are based on the use of partitioning coefficients (Kd) to represent retardation of transport due to sorption of contaminants on solid surfaces. In many subsurface environments, however, small solid particles called colloids move with the groundwater. Sorption on these colloids facilitates transport rather than inhibiting it. Contami- nant transport in solution using Kd for retardation and colloidal transport represent competing models of the dominant physical processes that affect risk estimates from groundwater pathways. If competing conceptual models exist, and none can be proven to be the more appropriate for the specific situation being assessed,
104 RISK AND DECISIONS ABOUT TRU AND HLW all must be identified for use in the risk assessment, at least dur- 7 ing the scoping step. ⢠Collect and review existing needed information. The goal of this step is to find out how much of the needed information is al- ready known. Data probably exist on most problems, but if they are not necessary for the risk calculations that have been identi- fied in the structuring step, no effort need be placed on assem- bling them, particularly given the cost and difficulty involved in collecting some data. Doing so would waste resources, slow the process, and detract from the goal of helping stakeholders gain insight and trust in the decision. The criterion for collecting more information is whether having the additional information could actually change the ranking of the alternative disposition options under consideration. This can be established with scoping risk calculations. ⢠Perform scoping risk assessment and sensitivity studies to identify critical parameters requiring the greatest attention. The risk calculations at this stage may be simpler, principle- based analyses with less detail than may ultimately be required, and their main purpose is to identify the most important needs for additional data development and additional model complex- ity. Basic risk calculations will be performed following the risk analysis blueprint. If relevant model packages are readily avail- able, they could be used, but complex model development should not occur yet if the relevant models do not exist. Similarly, data gaps are to be tested for their significance at this stage to deter- mine how much effort is warranted for filling these gaps. For each data gap, educated guesses of their possible values, focus- ing on identifying a range of such values, are more useful than a 8 âbest guessâ value. Sensitivity analysis is then applied to these 7 Risks must be calculated with each potentially viable model. Results of in- dependent and mutually exclusive models must not, however, be treated simply as increasing the variance of a result distribution-characterized median or mean value. Each model generates its own result distribution, and mixing or averaging them corrupts the information they would otherwise yield, producing a meaning- less outcome. 8 A practice in some risk assessments (e.g., under CERCLA) is to fill data gaps with default values. EPA provides lists of such default values. Use of de-
A RISK-INFORMED APPROACH 105 estimated ranges to determine which data gaps are important to the final decision and how much precision is required when col- lecting the necessary new data. ⢠Describe data gaps and data collection plan. The most impor- tant output of the analysis at this stage is identification of infor- mation that is critical to making a decision. It should also be possible to determine whether competing model formulations have to be retained as the analysis moves forward. Insights from the sensitivity analysis are the most important determinants of a sound data collection plan. âValue of informationâ (VOI) is a method sometimes prescribed for determining what additional information to collect (Howard, 1966; Morgan and Henrion, 1990). VOI is useful for determining how much effort should be expended to resolve uncertainty regarding risk calculations or in- put assumptions that do affect the choice among alternatives. In its formal sense, VOI cannot be calculated unless the risk analy- sis is prepared to formally weight and combine all the decision- relevant criteria into a single âoutcomeâ that is to be optimized. This is rarely acceptable in a process to inform a complex policy decision that involves many different stakeholder groups. How- ever, there is value in gathering more precise information on any uncertain variable that is âsensitiveâ (i.e., that could alter the choice of disposition alternatives by taking on different possible values over its range of uncertainty). Thus, simple sensitivity analysis is a sound first step towards a well-targeted data collec- tion plan. How much effort to expend on data collection or model refinement for each sensitive input must be a judgment that balances the degree of its observed sensitivity, the cost of the additional data collection on that variable, and the likelihood that the additional data collection will reduce its range of uncertainty enough to help clarify the choice between the alterative waste disposition paths. ⢠Conduct review and feedback of data collection and analysis plans by experts and stakeholders. Review here need not be in- fault values has the drawback of obscuring rather than highlighting the impor- tance of the data gap to obtaining a sound risk estimate. At this scoping stage, the goal is to identify information to collect for the risk assessment, not to pro- duce an actual risk estimate. Use of default values would undermine this goal.
106 RISK AND DECISIONS ABOUT TRU AND HLW terpreted in the formal sense, but presentation and discussion of results must be conducted despite a likely sense on the part of DOE that it has no âreal resultsâ at this stage. The process of consultation is a critical element even at this preliminary stage. If results are not publicly presented and discussed at this stage, op- portunities to gain the confidence of stakeholders in the final risk assessment will likely be lost. Because this is a very simple level of analysis, it should be easy to present and explain to stake- holders. It should not be presented as if it is a final analysis, but only as a preliminary analysis intended to define further model and data development needs. The emphasis in communicating findings should be on the sensitivities identified, not on the risk estimates that were generated. When presenting results of this scoping analysis, all of the data gaps must be listed, along with the range of possible values explored and a summary of how much the risk estimates under each disposition alternative were altered as a result of varying the input over that range of values. The impact on results of using competing representation of physical processes must also be presented. ⢠Finalize work plan and move forward. Following discussions with multiple types of parties, DOE will need a clear but brief statement of the next steps of the analysis. Such a statement use- fully includes only elements that have been demonstrated in this step to be relevant to improving the choice among disposition al- ternatives. To instill this, DOE can explain briefly how each modeling or data collection element of the plan would affect the quality of the decision. To be useful, the statement should also indicate the degree of precision required for each type of infor- 9 mation to be gathered, and data that might help resolve uncer- tainties about appropriate conceptual models if the choice of model is itself still in question. 9 The overall goal and approach of the scoping step is consistent with the concept of data quality objectives that have been promoted by EPA (2000) for data collection efforts under Superfund and other programs. The express pur- pose of the data quality objectives process is to identify the quantity and degree of precision needed in a effort.
A RISK-INFORMED APPROACH 107 Step 3. Collect Data and Refine Models This represents a straightforward implementation of the data collec- tion plan developed in Step 2 after review of the scoping analysis results. ⢠Collect quality data that describe the waste and the site. The degree of precision (both sampling density and instrumentation) must meet the standards identified in the data collection plan. The uncertainties in resulting data must be characterized. ⢠Describe and collect data regarding engineering remedies under consideration (identified in Step 1). Often the emphasis in a risk assessment is on site-related data. However, for HLW and TRU waste disposition, there will also be substantial uncer- tainties regarding the technologies that would be used. During this step, the uncertainties in the performance and costs of the technologies that would be used also have to be described. ⢠Refine model logic. While data collection proceeds, modelers can be engaged in adding the types of refinements to the model- ing frameworks that were also identified as important during Step 2. Refinements must be tested carefully. A summary of the impacts of the model changes on estimated results would facili- tate review. Validation runs must be conducted and reported as well. ⢠Disclose the information. The new data and information assem- bled in this step must be clearly documented and made accessi- ble to non-DOE groups. Access to data must be made possible as they become available and in advance of a formal review. On- line posting of data files at a dedicated web site might be helpful in this regard. ⢠Review by experts and stakeholders. As with all the steps, an external review of the output of this step would build confidence before moving to the next step, which is to use these new data in revised risk calculations. In such a review, experts would be en- gaged to comment on the quality of the new data, and whether they sufficiently meet the objectives that were established in the scoping step. Other experts would review the way the revised
108 RISK AND DECISIONS ABOUT TRU AND HLW models function and comment on whether it appears that the models are producing realistic results for specific test cases. Step 4. Prepare Refined Risk Assessment In this step, the refined data and models are combined to produce risk estimates that would be used to inform the exemption decision. The elements of this step follow: ⢠Define the range of uncertainty in parameters for modeling, making use of collected data. The raw data will have to be in- terpreted into the form required for input to the model. Although data will have been refined by this stage, it is still important to define the remaining range of uncertainty and to preserve an un- derstanding of this uncertainty on risk estimates. ⢠Conduct analyses, including uncertainty analysis. Risk esti- mates are now produced. Uncertainty analysis need not entail probabilistic combinations of alternative assumptions. In fact, it may be most illuminating to continue to produce separate results using alternative conceptual models, if this model uncertainty could not be resolved as a result of the additional data collection. With regard to remaining uncertainties in the parameters, best es- timates must now be developed, and sensitivity analysis per- formed again. ⢠Perform a validity check (âlaugh testâ): Are results reason- able in light of real-world experiences? DOE would be well served if it were to require a conscious effort at this stage to sit back and ask itself if the risk estimates being produced by the re- fined model are reasonable. Are there any observable situations against which the modelâs outputs can be tested? The risk esti- mates themselves would be difficult to validate, but intermediate outputs of the model, such as contaminant concentrations and rates of transport, should be fundamentally possible to validate. Written summaries of the various ways in which DOE has been able to confirm that the model can reasonably well reproduce ob- served system behavior would make the models easier to believe and harder to discredit.
A RISK-INFORMED APPROACH 109 ⢠Perform a thorough quality assurance and quality control of model logic and data inputs. If the risk model passes the valid- ity checks, it is time to prepare a thorough review of all the code and data inputs that were used to produce the risk estimates. If any problems are detected, the risk estimates must be revised. ⢠Summarize the results of risk assessment, with uncertainty. In preparation for public and peer review, the results must be summarized in writing. As always, the summary should be open and thorough in explaining ranges of uncertainty that remain and critical sources of sensitivity. Often a given uncertain parameter will affect risk estimates for each of the alternatives under con- sideration in a similar manner. It is important that the summary of uncertainties be done in a comparative manner, indicating how they affect the differences in risks estimated for the disposition alternatives. If this is not done, one may get the useless result that all alternatives have large and overlapping ranges of uncer- tainty. However, the real case may be that one alternative is con- sistently lower in risk than the other(s) and that the only uncertainty is the absolute amount by which it is lower. Thus, differences in risk among alternatives are more important to re- 10 port than the absolute risk of each alternative. ⢠Peer review of model results. DOE would build greater credi- bility if an independent panel of individuals who have expertise in all the relevant aspects of data, modeling methods, and uncer- tainty analysis reviewed and commented on the findings. Revi- sions should be made to address any identified problems. ⢠Release results to the public per agreed plan. The results would be released after completion of the peer review. A meeting of stakeholders would be needed to present the results and dis- cuss their implications for decision making. In the course of the meeting, additional areas of desired refinement might be identi- fied. 10 Of course, the absolute risks of each alternative must be computed and should be reported as well, but the use of differences becomes an important way to help clarify the significance of the uncertainties.
110 RISK AND DECISIONS ABOUT TRU AND HLW Step 5. Conduct Additional Analyses and Data Collection as Needed to Support Decisions If additional analyses or refinements are needed, they would be done following a plan agreed on with stakeholders. It is likely that such plan- ning will result in analyses and data collection being iterative; additional data collection might be necessary as learning progresses and the need for additional analyses is identified. Step 6. Finalize Decision Risk estimates will never be completely free of uncertainty or even controversy. However, at some point the âsufficientâ information will have been developed, and DOE must decide whether the balance of evi- dence on risks and other decision criteria appears to support an alterna- tive disposition option. If so, DOE would summarize the findings, the process by which they were obtained, and the issues raised by stake- holders and use this summary to support a formal request for an exemp- tion from the appropriate authorities, as discussed in Chapter 3. In deciding to make such an exemption request, DOE must also be mindful of the need to negotiate changes in existing agreements with the states, U.S. EPA , and U.S. NRC and must consider the prospects for success in such a negotiation. If the prospects for success in negotiation are poor, this should have become apparent in the course of performing the risk assessment, especially if the iterative and participatory process above has been followed. 4.4 PROCESS FINDINGS AND RECOMMENDATIONS Finding 8: An effective and credible risk-informed decision-making process has several characteristics. It is (1) participatory; (2) logical; (3) consistent with current scientific knowledge and practice; (4) transparent and traceable; (5) reasonably independent of decision makers; (6) subjected to thorough, independent peer review; (7) technically credible, with believable results; and (8) framed to ad- dress the needs of the decision process.
A RISK-INFORMED APPROACH 111 Drawing from points throughout this chapter, the committee summa- rizes the characteristics of an effective and credible risk-informed decision- making process in the list above. A risk-informed process that fails to meet any of these eight essential characteristics would likely be ineffective. In order to be effective, a risk-informed approach must be trusted. The charac- teristics listed above are intended not only to ensure a result that can be trusted but, equally important, to create a process that can be trusted. For example, a technically credible risk-based approach that lacks participation or transparency would likely not be trusted and, therefore, would likely be ineffective in supporting a waste exemption process. With the process fully described, the committee reiterates that simply following these steps as a checklist is insufficient for a successful appli- cation of the risk-informed approach. These steps are difficult and there are impediments to effective use of a risk-informed approach. These are discussed in more detail in Chapter 5. Finding 9: The biggest challenges to developing a meaningful risk- informed decision process, such as recommended herein, are: mini- mizing disruption to existing laws, regulations, and agreements; cre- ating buy-in to the approach; and enabling meaningful participation by participants who have few resources. Disrupting existing laws, regulations, and agreements (e.g., changing the rules to allow potentially unsafe practices to proceed without due process) will tend to cause resistance and unintended consequences of an exemption process. Any meaningful decision process that involves stakeholders such as the risk-informed process recommended here, will require finding ways to implement an exemption process in the least dis- ruptive manner possible with regard to existing laws, regulations, and agreements. This process can help to maintain predictability, to create fewer unintended consequences, and to avoid destabilizing the policy equilibrium that has been reached as people have acted in reliance on the existing framework. The committee does not know how many exemp- tions DOE might seek. Assuming that the number will be relatively few, the committee has recommended exemptions because they can minimize disruption while preserving the desirable features of a risk-informed ap- proach. Recommendation 4: Congress, DOE, U.S. EPA, and U.S. NRC should take actions as necessary to enable DOE to implement effec-
112 RISK AND DECISIONS ABOUT TRU AND HLW tively the risk-informed approach recommended here. Specifically, they should provide for a formal, well-structured exemption process, institute technical review of the risk analysis independent of the agency producing the analysis, give decision-making authority to an agency outside DOE, and ensure that sufficient resources are relia- bly available for regulators, tribal nations, and stakeholders to par- ticipate meaningfully in the process. The committee did not develop detailed actions for each entity/agency for the steps necessary to implement this recommendation. There are many possible distributions of responsibilities; what one agency might contribute toward implementation of the recommendations depends heavily on what others would contribute. The implementation of this recommendation should be achieved jointly by the entities involved, without attempting to define in advance of inter-agency discussions what each should contribute.
