4
A Risk-Informed Approach: Procedures and Criteria for Risk Assessment to Support an Exemption Process

In this chapter, the committee explains the key attributes of the risk assessment process that it recommends and outlines steps by which risk assessment can be performed and integrated into a process to inform decisions on the alternative dispositions of certain high-level and transuranic legacy wastes. The intent of this chapter is to describe the characteristics and provide an example of a process for risk-informed decision making in support of exemption determinations, not to prescribe the specific process.

4.1 USING RISK ASSESSMENT

The general methods of risk assessment for environmental concerns are well established in numerous earlier publications. These include a series of reports on risk assessment methods and processes by the National Research Council (NRC, 1983, 1994a, 1996). Other groups and individuals have written similar general texts on principles of risk analysis (e.g., PCCRARM, 1997a, 1997b). Agencies of the U.S. government



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste 4 A Risk-Informed Approach: Procedures and Criteria for Risk Assessment to Support an Exemption Process In this chapter, the committee explains the key attributes of the risk assessment process that it recommends and outlines steps by which risk assessment can be performed and integrated into a process to inform decisions on the alternative dispositions of certain high-level and transuranic legacy wastes. The intent of this chapter is to describe the characteristics and provide an example of a process for risk-informed decision making in support of exemption determinations, not to prescribe the specific process. 4.1 USING RISK ASSESSMENT The general methods of risk assessment for environmental concerns are well established in numerous earlier publications. These include a series of reports on risk assessment methods and processes by the National Research Council (NRC, 1983, 1994a, 1996). Other groups and individuals have written similar general texts on principles of risk analysis (e.g., PCCRARM, 1997a, 1997b). Agencies of the U.S. government

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste have codified specific methods for performing risk calculations in particular applications. For example, the U.S. Environmental Protection Agency (U.S. EPA) has specific guidance for performing risk assessments acceptable for identifying clean up plans under CERCLA (U.S. EPA, 1989, 1990a, 1990b, 1991a, 1991b, 1991c, 1991d, 2001a, 2001b). The U.S. Nuclear Regulatory Commission (U.S. NRC) also has recommended methodologies for carrying out performance assessments for low-level radioactive waste disposal sites and for nuclear reactors (see, e.g., U.S. NRC, 2000). Appendix A summarizes this conceptual background on some of the existing literature on risk assessment and risk management procedures. The approach described here conforms to the general principles and methods that have been established for risk analysis, but it has been crafted specifically to support the exemption process for high-level waste (HLW) and transuranic (TRU) waste described in Chapter 3. The risk terminology used in this report, including the distinction between risk assessment and risk analysis, is described in Sidebar 4.1. Sidebar 4.1: Risk Terminology Risk analysis is an umbrella term that includes risk assessment, risk communication, and risk management. Risk assessment is the scientific evaluation of known or potential adverse effects for an individual, group, society, or the environment resulting from exposure to hazards. The risk-assessment process consists of the following steps: (1) hazard identification, (2) hazard characterization, (3) exposure assessment, and (4) risk characterization. The definition includes quantitative risk assessment, which emphasizes reliance on numerical expressions of risk, qualitative expressions of risk, and characterization of the attendant uncertainties (see Appendix A for a more detailed definition from the NRC, 1983). It is distinguished, also per NRC (1983), from risk management, although good practice requires iteration between risk assessment and risk management. Note that in this report the terms analysis and assessment are not used as shorthand for risk analysis and risk assessment. Risk management is “the process of evaluating alternative regulatory actions and selecting among them. Risk management, which is carried out by regulatory agencies under various legislative mandates, is an agency decision-making process that entails consideration of political, social, economic, and engineering information with risk-related informa-

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste tion to develop, analyze, and compare regulatory options and to select the appropriate regulatory response” (NRC, 1983). Refined risk assessment: An iterative approach to risk involves doing an initial risk assessment with available data and models. Sensitivity analysis is applied to identify aspects of the assessment that require greater complexity or precision before a decision can be made from the assessment results. After further data collection and/or model development indicated by the initial assessment, a refined risk assessment is prepared using the more complete or detailed information and models. A conceptual model is a description of the specific processes (e.g., chemical, physical, biological) that are believed to govern the behavior of the system to be assessed. In the case of contaminants in the ambient environment, site-specific geological, biological, and climatological conditions may make different processes relevant in different locations. For quantitative assessments, the conceptual model determines the types of equations and parameters that should be used to make estimates of contaminant release, transport, transformation, and human exposure in a particular location, whereas the conceptual model determines how system behavior is described in qualitative assessments. Once these are defined, then it is possible to start to identify appropriate mathematical models to build, adapt, or adopt for use in the risk analysis, and the relevant types of data to start to assemble or collect. If there is uncertainty or disagreement about the conceptual model, then it may be important to employ alternative models in the risk analysis. When alternative conceptual models are considered, data collection and analysis may also be targeted to allow evaluation of which model or models are most supported by the data. If more than one model provides a similar quality of fit to the observations, risk estimates should be made from the multiple models. A risk model is a mathematical representation of the conceptual model. Once formulated mathematically, and estimates of all the necessary parameters and input data are developed, the conceptual understanding can then be used to simulate or predict the behavior of contaminants in the system, to estimate the risks that may result from them. Mathematical models of the same processes and behavior can be developed with many levels of complexity. Since many of the relevant parameters are likely to be highly uncertain, one should begin with a simple model and use it to explore which uncertainties affect the risk estimates most. A refined risk model is created by progressively adding mathematical complexity or more precisely estimated parameters and input data to an initial risk model. Such refinements should be guided by sensitivity analysis results using the initial model.

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste System behavior: When speaking of an environmental issue, the “system” is the set of all environmental components that interact with each other, including the chemical compounds in the soil, the physical features of the land, the organisms that live in the subsurface and at the surface, and the waters and animals (including humans) that move through the area. The system may also include the air and atmospheric interactions. The “behavior” of the system refers to the way that these interactions occur when a change is made, such as the addition of a contaminant to the environment. Taken on its own, this chapter may appear to oversimplify the true complexity of the analyses that will have to be performed in the course of the process. The committee cautions that merely performing each step identified in this chapter in a “cookbook” fashion is not sufficient for a successful application of the risk-informed approach. In particular, if the analyses associated with each step ignore or attempt to diminish the importance to decision making of the underlying complexities and unknowns of the processes being modeled, the risk assessment supporting the risk-informed approach will likely be rejected (see Sidebar 4.2). Chapter 5 provides more discussion of these complexities, and describes how the committee thinks they should be incorporated into the risk analysis process. Considering carefully the issues that are discussed more fully in Chapter 5 and preparing a plan that consciously addresses them are essential to this process. 4.2 CHARACTERISTICS OF A GOOD RISK-INFORMATION APPROACH Chapter 3 notes the importance of creating an exemptions process that is transparent, inclusive of stakeholders,1 and credible. These characteristics must pertain to the exemption-seeking process as a whole, but they are also important attributes of a good risk analysis in its own right. In addition to procedural considerations, achieving each of these characteristics places specific technical demands on the way that a risk analysis is conducted. Specifically, the risk analysis conducted in support of an exemption application must be as follows: 1   The term "stakeholder" here includes the interested and affected public.

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste Sidebar 4.2: Challenges and Difficulties in a Risk-Informed Approach Risk assessment sometimes is prescribed as a universal solution for problems faced by decision makers. Such thinking is counterproductive. Risk assessment is a tool that can help decision makers reach a solution, but it is difficult to use well and does not guarantee a satisfactory outcome. Chapter 5 details some of the technical and institutional limitations of risk assessment, but it is worthwhile to mention two of them here as a cautionary note to give a fuller picture of the risk-informed process. Some people may question the credibility of risk assessments. This may happen because the analyses in a risk assessment are not credible, because the institution presenting the risk assessment is not credible, or for both these reasons. At its information-gathering meetings, the committee repeatedly questioned the experts presenting results of risk assessments about how they had established the validity of the models they were using. Few presenters were able address the questions, and fewer still gave satisfactory responses. People without the time or resources to critique the analyses may use the credibility of the institution as a proxy for evaluating the credibility of the analyses. It is rare (and indeed suspicious to some) for an environmental risk assessment to yield results with small uncertainties. Uncertainties often are so large that the results of a risk assessment must be deemed indeterminate. Even with these problems, however, the committee believes a risk-informed approach using risk assessment as a structured method to develop an understanding of, and to characterize, risks is the most promising approach. Logical. The sequence of steps of the risk assessment must map out a coherent chain of cause-effect relationships that link assumptions about the handling and final disposition of the waste stream in question to a set of outcomes that are specifically identified as relevant to the exemption decision. Well founded. The methods used in each step must be consistent with current scientific knowledge and practices. If the state of science has competing theories, the risk analysis must consider appropriate alternative approaches, rather than choosing one, unless it can be demonstrated that such a simplification would

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste not alter the estimates of outcomes sufficiently to affect the decision between alternative waste disposition options. Transparent and traceable. Risk analyses can become too complex, obscure, or inaccessible for even knowledgeable outside parties to understand, review, or reproduce. It is useful to resist this tendency by giving care from the beginning to make clear the steps, assumptions, and data that are being used. Identifying and making available sources of information, as well as identifying and characterizing uncertainties, enables others to check them. Documentation after the analysis is completed does not meet this requirement. Participatory. Most of the parties interested in the outcome of the risk analysis will not be engaged in performing the analysis itself, but they will not trust the results if they are not allowed to participate actively in discussions about how to structure and refine the analysis. Transparency also cannot be achieved unless stakeholders participate in the process from the beginning (NRC, 1996). The key feature of the risk analysis process described in this section is that the data, modeling, and any other calculations in estimating risk must be structured to inform a specific and well-defined decision. A coherent and efficient risk assessment requires that the decision criteria be well defined in advance of any computational or modeling steps. It also requires that a sufficient number of options from which to choose be considered in the decision to avoid excluding potentially superior options. Therefore, the first step is that a decision be defined; and second that a list of decision alternatives from which to choose be considered. In this application, the options would be a list of two or more methods for disposing of a particular high-level or transuranic waste form. For example, if the Department of Energy (DOE) is seeking an exemption related to closure of a set of HLW tanks, the options must include, at a minimum, (1) complete removal of all HLW for disposal in a deep geologic repository, and (2) a specific plan for how much of tank residuals may be left in a tank and any associated form of stabilization (e.g., grouting) suggested as an alternative disposition that might be allowed as an exemption. DOE may wish to consider more than one alternative disposition, but there must be at least the default and one alternative listed.

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste The critical issue is that all the options be defined specifically before analysis starts. The second step is to identify all of the information needed for making a decision. That is, what are the types of risks, costs, and other criteria must be considered when determining whether one or some subset of the disposition options would be preferable to the others? As stated earlier, human health risks are an important consideration at any site, and other considerations could have equal or greater importance at some sites. Meaningful quantitative analysis cannot be initiated until these two structuring steps have been completed. The risk assessment can then be designed to address the specific set of relevant considerations for the specific disposition options that have been set forth. This discussion emphasizes good process rather than good decisions. The committee, unfortunately, cannot prescribe good decisions—good decisions are not definable. A good process is necessary but not always sufficient for a good decision, and can still lead to bad decisions and generate a great deal of outrage. An agency may follow a process—posting a preliminary decision for a required amount of time, meeting a required number of times, listing public comments in a revised document, and issuing a final decision—but essentially undermine the value of the process by making it simply procedural, checking the box off and proceeding on its predetermined course. For the risk-informed approach to be successful, the agencies must embrace it as something not only useful and meaningful, but also essential to its planning. Risk assessments can be extremely complex, yet shed little light on whether one cleanup option is to be preferred over another. At the same time, the complexity can be a barrier to gaining confidence and trust in any recommendations that may purportedly be justified by the risk analysis. The approach described here embraces the principle that analytical detail and complexity should be limited to the minimum necessary to distinguish the best option or options.2 If the relative risks and trade-offs can be established with a relatively simple analysis that is nonetheless grounded in empirical data, then this simpler analysis will be easier to explain and easier for non-analysts to understand. A simpler analysis is also easier to validate. This will allow the risk analysis process to move more quickly to the point at which there is communication among stakeholders on how to make the relevant trade-offs. By fostering such discus- 2   More discussion of the philosophy, structuring, and key elements of the approach described here can be found in NRC (1996), Howard (1966), and Chapter 3 of Morgan and Henrion (1990).

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste sions early in the process, those managing the process can ensure that stakeholder input is considered and incorporated into the final product. It bears noting that the risk assessment developed in this process will probably have considerable complexity before it is completed. Also, considerable technical sophistication is needed to produce even a “simple” analysis that can effectively guide the addition of detail. The critical point is that the quantitative analysis be initiated with minimal complexity and that the complexity be increased gradually, as the need for it is determined and as all participants gain understanding of the key elements of the analysis that are driving its risk estimates.3 It is also possible that the forms of complexity resulting from this iterative process will be quite different from the forms of complexity that would have been added if analysts were to strive to incorporate a final degree of complexity from the outset. For example, risk analysts may set out with models that incorporate detailed dynamics, yet it may turn out that only steady-state outcomes are important to the decision. Wasted effort on details that are merely distracting can be avoided in a process that is guided by insights from an initial, relatively simple analysis. The process of analysis may be more important in achieving transparency, trust, and understanding than thorough documentation of an analysis that is offered as a fait accompli, particularly if the final product on which decisions are made is complex. The committee observed in site visits and related presentations that DOE has tended to generate its analyses without such an iterative, staged process. DOE appears usually to present only the final product of its risk analyses to stakeholders, at a point where the analysis is exceedingly complex and difficult to review or gain intuitive understanding.4 Focusing on the numerical results can 3   Sensitivity analysis is the key method advocated here for determining whether additional complexity is desirable. The junctures where sensitivity analysis is best performed are identified in the step-by-step process that is provided in this chapter. 4   The Risk-Based End States (RBES) process, mentioned in Chapter 1 and described in greater detail in Appendix B, is an example of a process in which stakeholders were not engaged before apparent decision recommendations had been formed. However, there appears to have been little or no risk assessment underlying that process. In other cases, the committee was presented with risk analyses that had much detail, but failed to provide decision-relevant information such as sensitivity to key assumptions. The presentation on risk from residual radioactive waste in groundwater at Hanford was a case in point (Thompson, 2004).

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste also obscure one of the most important products of the analysis: Beyond risk estimates, risk analysis can produce insights, which may be highly qualitative. One must also distinguish a “simple” analysis from one that ignores uncertainty. Uncertainty is a major concern in making estimates of risks, particularly when considering long time scales. Key areas of uncertainty must be explored even in the first phase of the analysis. Using alternative ways of representing physical processes or alternative sets of assumptions helps to address model uncertainty and to understand the sources of sensitivity and boundaries of the risk estimates. That is, one can learn how much risk estimates associated with each disposition option are likely to vary as a result of unknown factors or areas of disagreement. This can be accomplished using quite simple analyses; indeed, more complex analyses can present significant barriers to exploration of uncertainties. Identifying the boundaries of uncertainty created by alternative assumptions is more important than adding computational complexity while still relying on a single set of deterministic assumptions. Many of the risk analyses presented to the committee by DOE have indicated that the opposite has been the practice (see, e.g., DOE, 2002c; Thompson, 2004). Thus, there are several critical elements to the process for implementing a risk-based approach for deciding on the disposition of high-level and transuranic wastes: specific set of options to be considered, list of information or data needed for a decision, set of criteria for determining the best option(s) developed in advance of any analysis, risk computations performed at the minimal level of complexity necessary to separate options according to the decision criteria, risk computations performed simulating competing views of the physical processes and using ranges of parameter values that reflect the state of science, and uncertainties explicitly explored and retained unless they are demonstrated not to affect the relative ranking of the disposition options.

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste These are the key elements of what one might call a decision-oriented risk analysis approach, but it bears repeating that the process by which these elements are put together is exceedingly important to achieving acceptance of any decisions that may be informed by such a risk analysis. The process must be iterative and participatory (see Sidebar 4.3).5 The institutional impediments outlined in Section 5.4 must be considered in determining who specifically performs the risk assessment and how it is managed and evaluated at an individual site and across the DOE complex. All of the elements of risk management (which have to be reflected in the final exemption application) are analyzed in parallel. The analysis of each element must meet consistently high standards of quality and credibility so that they will withstand the scrutiny of external review. 4.3 A SIX-STEP PROCESS FOR RISK-INFORMED DECISION MAKING To better elucidate how one can combine process and analysis, the rest of this section provides an outline of the steps that DOE could follow. Readers familiar with the CERCLA process may find that many of Sidebar 4.3: Decision Support Systems A “decision support system” is designed to enable greater participation by non-experts and so could support some of the goals articulated in this chapter. A decision support system, in this context, would be a user-friendly computerized version of the risk models and relevant decision criteria that could be run on a microcomputer with only modest resource requirements. Such tools are intended to provide participants in the process, particularly non-modelers, with an ability to alter assumptions and elements of the risk analysis independently, and to quickly observe the impacts of such “what if” questions on the outcomes viewed as relevant to the decision. This capability can enable modelers and non-modelers alike to gain hands-on familiarity with how the risk model responds to 5   A recent example where a decision support system was integrated into an iterative and participative public decision process like the risk analysis process described in this chapter can be found in Passell et al. (2003).

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste alternative assumptions, and help in identifying the most important elements of the risk analysis. If developed well, a decision support system can enhance the sense of empowerment and participation of stakeholders, it can enhance dialogue among stakeholders and modelers, and it can enhance the iterative aspects of the process. It is important also to note that decision support systems can be expensive and time-consuming to develop, because each one requires a well-designed graphical user interface with functions tailored specifically to the risk analysis in question. The software design and development challenges can divert attention and resources away from simply performing a good decision-oriented risk analysis process. Also, if the resulting software product does not meet expectations of performance (which can often become unrealistic), it can unnecessarily undermine the goal of engaging stakeholders’ trust and understanding of the analysis. A decision support system is not essential for a good iterative and participatory process, but it can be a valuable supplement if it is created with a proper understanding of its role and limitations, and if appropriate software development resources are allocated. its elements are familiar and that the whole process is compatible with CERCLA.6 However, the following steps were devised specifically to illustrate a process for the transuranic and high-level waste disposition decisions that DOE may wish to have considered in an exemption application. Recommendation 3: DOE and its regulators for HLW and TRU waste should adopt a six step process for risk-informed decision making: (1) initiate the process, laying out viable options and potential decisions; (2) scope the information and analysis; (3) collect data and refine models; (4) prepare refined risk assessment; (5) develop additional analyses and data collection, as needed, to support decisions; and (6) finalize the decision. 6   For more about the CERCLA process see Section 5.4 or the U.S. EPA web site, available at: http://www.epa.gov/superfund/.

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste cial, economic, ecological, and ethical outcomes as well as consequences for human health and safety…” (NRC, 1996). The decision criteria will obviously include health risk, costs, and regulatory compliance. However, DOE will have to define the metrics of risk that it will allow to have a significant influence on the decision. Metrics of risk that could be used include excess lifetime cancer risk to an individual, population-wide cancer incidence risk, and so forth. It is important that the relevant metrics of risk be identified clearly at this early stage so that DOE can ensure that any analysis it performs will be designed to provide those specific outputs. Distributional outputs, or the question of who bears the burden of risks and costs, are usually relevant in such questions. Those seeking the exemption need also to work with stakeholders at this stage to determine how to define the groups of concern and what measures of distributional impact it will be appropriate to use. Develop process plan. DOE will need an outline of the process by which analysis and consultation will proceed. Such a plan would contain each of the remaining steps outlined below, but would include timing and parties to be involved. The plan would clearly identify points throughout the process at which new information releases will occur. Review and feedback. In completing each of the elements above, DOE, of course, has to make initial internal deliberations to prepare for meetings, but all of the elements of the analysis must be discussed and refined in a consultative process with stakeholders. Consultations can be both informal and formal. Before proceeding to the next step, DOE must be able to summarize all of the elements coherently and specifically. Step 2. Scope Information and Analysis Once DOE has defined two or more disposition alternatives to be compared, and the key criteria on which they will be compared, it can begin the process of estimating the specific risks that will be part of that comparison. However, the process is a gradual one, and the next step involves only a relatively simple set of risk calculations, which can be characterized as a scoping analysis, before consultation occurs again.

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste Scoping analysis results may presage the likely direction of the decision, but it is unlikely that any decision can be finalized at this stage. The elements of the scoping step follow: Sketch out the structure of the risk analysis. At this point DOE will know what measures of risk it needs to calculate and can prepare a blueprint of the types of calculations that are required to make these estimates. This will help identify data and models needed. It may also help identify models that are not relevant. Often a risk assessment proceeds using models that are readily available, and these may not always provide the actual type of information that is needed for the decision at hand. The result is a side-tracking of the analysis effort onto results that are not of much use for guiding the decision at hand. The point of this step is to identify what is needed and to keep all analysis efforts targeted to those needs. Identify parameters, data, and models required. When the risk analysis blueprint is completed, it will reveal the inputs that are required, and these become the basis of a list of data and models to develop. Judgments must be made for how to model the system(s) with respect to scale, complexity, whether to use deterministic or probabilistic methods, and which conceptual models might apply. At this stage, differences of opinion about appropriate modeling methods may come to light. For example, many models of subsurface contaminant transport are based on the use of partitioning coefficients (Kd) to represent retardation of transport due to sorption of contaminants on solid surfaces. In many subsurface environments, however, small solid particles called colloids move with the groundwater. Sorption on these colloids facilitates transport rather than inhibiting it. Contaminant transport in solution using Kd for retardation and colloidal transport represent competing models of the dominant physical processes that affect risk estimates from groundwater pathways. If competing conceptual models exist, and none can be proven to be the more appropriate for the specific situation being assessed,

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste all must be identified for use in the risk assessment, at least during the scoping step.7 Collect and review existing needed information. The goal of this step is to find out how much of the needed information is already known. Data probably exist on most problems, but if they are not necessary for the risk calculations that have been identified in the structuring step, no effort need be placed on assembling them, particularly given the cost and difficulty involved in collecting some data. Doing so would waste resources, slow the process, and detract from the goal of helping stakeholders gain insight and trust in the decision. The criterion for collecting more information is whether having the additional information could actually change the ranking of the alternative disposition options under consideration. This can be established with scoping risk calculations. Perform scoping risk assessment and sensitivity studies to identify critical parameters requiring the greatest attention. The risk calculations at this stage may be simpler, principle-based analyses with less detail than may ultimately be required, and their main purpose is to identify the most important needs for additional data development and additional model complexity. Basic risk calculations will be performed following the risk analysis blueprint. If relevant model packages are readily available, they could be used, but complex model development should not occur yet if the relevant models do not exist. Similarly, data gaps are to be tested for their significance at this stage to determine how much effort is warranted for filling these gaps. For each data gap, educated guesses of their possible values, focusing on identifying a range of such values, are more useful than a “best guess” value.8 Sensitivity analysis is then applied to these 7   Risks must be calculated with each potentially viable model. Results of independent and mutually exclusive models must not, however, be treated simply as increasing the variance of a result distribution-characterized median or mean value. Each model generates its own result distribution, and mixing or averaging them corrupts the information they would otherwise yield, producing a meaning-less outcome. 8   A practice in some risk assessments (e.g., under CERCLA) is to fill data gaps with default values. EPA provides lists of such default values. Use of de-

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste estimated ranges to determine which data gaps are important to the final decision and how much precision is required when collecting the necessary new data. Describe data gaps and data collection plan. The most important output of the analysis at this stage is identification of information that is critical to making a decision. It should also be possible to determine whether competing model formulations have to be retained as the analysis moves forward. Insights from the sensitivity analysis are the most important determinants of a sound data collection plan. “Value of information” (VOI) is a method sometimes prescribed for determining what additional information to collect (Howard, 1966; Morgan and Henrion, 1990). VOI is useful for determining how much effort should be expended to resolve uncertainty regarding risk calculations or input assumptions that do affect the choice among alternatives. In its formal sense, VOI cannot be calculated unless the risk analysis is prepared to formally weight and combine all the decision-relevant criteria into a single “outcome” that is to be optimized. This is rarely acceptable in a process to inform a complex policy decision that involves many different stakeholder groups. However, there is value in gathering more precise information on any uncertain variable that is “sensitive” (i.e., that could alter the choice of disposition alternatives by taking on different possible values over its range of uncertainty). Thus, simple sensitivity analysis is a sound first step towards a well-targeted data collection plan. How much effort to expend on data collection or model refinement for each sensitive input must be a judgment that balances the degree of its observed sensitivity, the cost of the additional data collection on that variable, and the likelihood that the additional data collection will reduce its range of uncertainty enough to help clarify the choice between the alterative waste disposition paths. Conduct review and feedback of data collection and analysis plans by experts and stakeholders. Review here need not be in-     fault values has the drawback of obscuring rather than highlighting the importance of the data gap to obtaining a sound risk estimate. At this scoping stage, the goal is to identify information to collect for the risk assessment, not to produce an actual risk estimate. Use of default values would undermine this goal.

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste terpreted in the formal sense, but presentation and discussion of results must be conducted despite a likely sense on the part of DOE that it has no “real results” at this stage. The process of consultation is a critical element even at this preliminary stage. If results are not publicly presented and discussed at this stage, opportunities to gain the confidence of stakeholders in the final risk assessment will likely be lost. Because this is a very simple level of analysis, it should be easy to present and explain to stakeholders. It should not be presented as if it is a final analysis, but only as a preliminary analysis intended to define further model and data development needs. The emphasis in communicating findings should be on the sensitivities identified, not on the risk estimates that were generated. When presenting results of this scoping analysis, all of the data gaps must be listed, along with the range of possible values explored and a summary of how much the risk estimates under each disposition alternative were altered as a result of varying the input over that range of values. The impact on results of using competing representation of physical processes must also be presented. Finalize work plan and move forward. Following discussions with multiple types of parties, DOE will need a clear but brief statement of the next steps of the analysis. Such a statement usefully includes only elements that have been demonstrated in this step to be relevant to improving the choice among disposition alternatives. To instill this, DOE can explain briefly how each modeling or data collection element of the plan would affect the quality of the decision. To be useful, the statement should also indicate the degree of precision required for each type of information to be gathered,9 and data that might help resolve uncertainties about appropriate conceptual models if the choice of model is itself still in question. 9   The overall goal and approach of the scoping step is consistent with the concept of data quality objectives that have been promoted by EPA (2000) for data collection efforts under Superfund and other programs. The express purpose of the data quality objectives process is to identify the quantity and degree of precision needed in a effort.

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste Step 3. Collect Data and Refine Models This represents a straightforward implementation of the data collection plan developed in Step 2 after review of the scoping analysis results. Collect quality data that describe the waste and the site. The degree of precision (both sampling density and instrumentation) must meet the standards identified in the data collection plan. The uncertainties in resulting data must be characterized. Describe and collect data regarding engineering remedies under consideration (identified in Step 1). Often the emphasis in a risk assessment is on site-related data. However, for HLW and TRU waste disposition, there will also be substantial uncertainties regarding the technologies that would be used. During this step, the uncertainties in the performance and costs of the technologies that would be used also have to be described. Refine model logic. While data collection proceeds, modelers can be engaged in adding the types of refinements to the modeling frameworks that were also identified as important during Step 2. Refinements must be tested carefully. A summary of the impacts of the model changes on estimated results would facilitate review. Validation runs must be conducted and reported as well. Disclose the information. The new data and information assembled in this step must be clearly documented and made accessible to non-DOE groups. Access to data must be made possible as they become available and in advance of a formal review. Online posting of data files at a dedicated web site might be helpful in this regard. Review by experts and stakeholders. As with all the steps, an external review of the output of this step would build confidence before moving to the next step, which is to use these new data in revised risk calculations. In such a review, experts would be engaged to comment on the quality of the new data, and whether they sufficiently meet the objectives that were established in the scoping step. Other experts would review the way the revised

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste models function and comment on whether it appears that the models are producing realistic results for specific test cases. Step 4. Prepare Refined Risk Assessment In this step, the refined data and models are combined to produce risk estimates that would be used to inform the exemption decision. The elements of this step follow: Define the range of uncertainty in parameters for modeling, making use of collected data. The raw data will have to be interpreted into the form required for input to the model. Although data will have been refined by this stage, it is still important to define the remaining range of uncertainty and to preserve an understanding of this uncertainty on risk estimates. Conduct analyses, including uncertainty analysis. Risk estimates are now produced. Uncertainty analysis need not entail probabilistic combinations of alternative assumptions. In fact, it may be most illuminating to continue to produce separate results using alternative conceptual models, if this model uncertainty could not be resolved as a result of the additional data collection. With regard to remaining uncertainties in the parameters, best estimates must now be developed, and sensitivity analysis performed again. Perform a validity check (“laugh test”): Are results reasonable in light of real-world experiences? DOE would be well served if it were to require a conscious effort at this stage to sit back and ask itself if the risk estimates being produced by the refined model are reasonable. Are there any observable situations against which the model’s outputs can be tested? The risk estimates themselves would be difficult to validate, but intermediate outputs of the model, such as contaminant concentrations and rates of transport, should be fundamentally possible to validate. Written summaries of the various ways in which DOE has been able to confirm that the model can reasonably well reproduce observed system behavior would make the models easier to believe and harder to discredit.

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste Perform a thorough quality assurance and quality control of model logic and data inputs. If the risk model passes the validity checks, it is time to prepare a thorough review of all the code and data inputs that were used to produce the risk estimates. If any problems are detected, the risk estimates must be revised. Summarize the results of risk assessment, with uncertainty. In preparation for public and peer review, the results must be summarized in writing. As always, the summary should be open and thorough in explaining ranges of uncertainty that remain and critical sources of sensitivity. Often a given uncertain parameter will affect risk estimates for each of the alternatives under consideration in a similar manner. It is important that the summary of uncertainties be done in a comparative manner, indicating how they affect the differences in risks estimated for the disposition alternatives. If this is not done, one may get the useless result that all alternatives have large and overlapping ranges of uncertainty. However, the real case may be that one alternative is consistently lower in risk than the other(s) and that the only uncertainty is the absolute amount by which it is lower. Thus, differences in risk among alternatives are more important to report than the absolute risk of each alternative.10 Peer review of model results. DOE would build greater credibility if an independent panel of individuals who have expertise in all the relevant aspects of data, modeling methods, and uncertainty analysis reviewed and commented on the findings. Revisions should be made to address any identified problems. Release results to the public per agreed plan. The results would be released after completion of the peer review. A meeting of stakeholders would be needed to present the results and discuss their implications for decision making. In the course of the meeting, additional areas of desired refinement might be identified. 10   Of course, the absolute risks of each alternative must be computed and should be reported as well, but the use of differences becomes an important way to help clarify the significance of the uncertainties.

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste Step 5. Conduct Additional Analyses and Data Collection as Needed to Support Decisions If additional analyses or refinements are needed, they would be done following a plan agreed on with stakeholders. It is likely that such planning will result in analyses and data collection being iterative; additional data collection might be necessary as learning progresses and the need for additional analyses is identified. Step 6. Finalize Decision Risk estimates will never be completely free of uncertainty or even controversy. However, at some point the “sufficient” information will have been developed, and DOE must decide whether the balance of evidence on risks and other decision criteria appears to support an alternative disposition option. If so, DOE would summarize the findings, the process by which they were obtained, and the issues raised by stakeholders and use this summary to support a formal request for an exemption from the appropriate authorities, as discussed in Chapter 3. In deciding to make such an exemption request, DOE must also be mindful of the need to negotiate changes in existing agreements with the states, U.S. EPA, and U.S. NRC and must consider the prospects for success in such a negotiation. If the prospects for success in negotiation are poor, this should have become apparent in the course of performing the risk assessment, especially if the iterative and participatory process above has been followed. 4.4 PROCESS FINDINGS AND RECOMMENDATIONS Finding 8: An effective and credible risk-informed decision-making process has several characteristics. It is (1) participatory; (2) logical; (3) consistent with current scientific knowledge and practice; (4) transparent and traceable; (5) reasonably independent of decision makers; (6) subjected to thorough, independent peer review; (7) technically credible, with believable results; and (8) framed to address the needs of the decision process.

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste Drawing from points throughout this chapter, the committee summarizes the characteristics of an effective and credible risk-informed decision-making process in the list above. A risk-informed process that fails to meet any of these eight essential characteristics would likely be ineffective. In order to be effective, a risk-informed approach must be trusted. The characteristics listed above are intended not only to ensure a result that can be trusted but, equally important, to create a process that can be trusted. For example, a technically credible risk-based approach that lacks participation or transparency would likely not be trusted and, therefore, would likely be ineffective in supporting a waste exemption process. With the process fully described, the committee reiterates that simply following these steps as a checklist is insufficient for a successful application of the risk-informed approach. These steps are difficult and there are impediments to effective use of a risk-informed approach. These are discussed in more detail in Chapter 5. Finding 9: The biggest challenges to developing a meaningful risk-informed decision process, such as recommended herein, are: minimizing disruption to existing laws, regulations, and agreements; creating buy-in to the approach; and enabling meaningful participation by participants who have few resources. Disrupting existing laws, regulations, and agreements (e.g., changing the rules to allow potentially unsafe practices to proceed without due process) will tend to cause resistance and unintended consequences of an exemption process. Any meaningful decision process that involves stakeholders such as the risk-informed process recommended here, will require finding ways to implement an exemption process in the least disruptive manner possible with regard to existing laws, regulations, and agreements. This process can help to maintain predictability, to create fewer unintended consequences, and to avoid destabilizing the policy equilibrium that has been reached as people have acted in reliance on the existing framework. The committee does not know how many exemptions DOE might seek. Assuming that the number will be relatively few, the committee has recommended exemptions because they can minimize disruption while preserving the desirable features of a risk-informed approach. Recommendation 4: Congress, DOE, U.S. EPA, and U.S. NRC should take actions as necessary to enable DOE to implement effec-

OCR for page 87
Risk and Decisions: About Disposition of Transuranic and High-Level Radioactive Waste tively the risk-informed approach recommended here. Specifically, they should provide for a formal, well-structured exemption process, institute technical review of the risk analysis independent of the agency producing the analysis, give decision-making authority to an agency outside DOE, and ensure that sufficient resources are reliably available for regulators, tribal nations, and stakeholders to participate meaningfully in the process. The committee did not develop detailed actions for each entity/agency for the steps necessary to implement this recommendation. There are many possible distributions of responsibilities; what one agency might contribute toward implementation of the recommendations depends heavily on what others would contribute. The implementation of this recommendation should be achieved jointly by the entities involved, without attempting to define in advance of inter-agency discussions what each should contribute.