Appendix I
Performance Assessment Process
The following is taken from the NRC report, Risk and Decisions about Disposition of Transuranic and High-Level Radioactive Waste (NRC, 2005b): Performance assessment is a process that “estimates the potential behavior of a system or system component under a given set of conditions. It includes estimates of the effects of uncertainties in data and modeling. In the context of radioactive waste, performance assessment is a systematic method for a repository risk assessment.” The key steps of the performance assessment (PA) process are presented in Figure I.1.
It is only meaningful to conduct a performance assessment in the context of a decision, such as Does the existing contamination require active remediation? or Does the proposed plan for waste disposal pose acceptable risk? The first step is identification of the performance objectives or decision criteria that will protect human health and the environment for the site and waste stream under consideration. An array of environmental regulations establish performance objectives to protect the following:
-
The general public
-
The inadvertent intruder
-
Groundwater resources
-
Air resources
-
Surface water resources
-
All ecosystem components associated with these media
Specifically, performance assessments evaluating disposal of low-level radioactive materials are developed to help assess compliance with the performance objectives of (Department of Energy) DOE Order 435.1 and Title 10 Part 61 of the Code of Federal Regulations. Some of these performance objectives are definite (e.g., 25 mrem per year exposure for a member of the public) and others are more fluid (e.g., releases to the environment must be as low as reasonably achievable [ALARA]), but determining compliance with even the definite performance objectives involves scientific judgment and regulatory policy decisions that are critical to the results. As part of the performance assessment, it is necessary to identify the points of compliance (or some surrogate, such as a point of calculation), which are the locations at which long-term risks to public health and the environment are evaluated.
After the performance objectives are identified and the point of compliance has been determined, the next step is the development of a conceptual model for the site and the waste that captures fundamental physical and chemical processes, issues of elemental dynamics, and features of the natural and engineered systems that will affect contaminant movement and concentration over time. With models of contaminant dynamics in the site environment, it is then possible to assess overall system performance in relation to identified objectives. The overall conceptual model for evaluating the impacts of a disposal facility must include models for waste release and contaminant transport in the relevant media.
The conceptual systems model is developed using monitoring information that can include historical data on system behavior as well as site characterization data. In addition to the data, a well-developed conceptual model utilizes the knowledge that people have accumulated concerning the site and environmental phenomena that affect the site. Neuman and Wierenga (2003) note that because each site is unique, general principles always must be supplemented by regional and site-specific data to be useful for conceptualization and modeling of subsurface flow and transport at a site, regardless of purpose. The same authors also point out that deficiencies in the conceptualization are far more detrimental to the predictive ability of a model than a suboptimal set of model input parameters. Systematic examinations of the results of variations in input parameter (i.e., sensitivity studies) can enable decision makers to make use of a performance assessment, even in data-sparse situations if the conceptual model represents the real environment reasonably well. However, no amount of parametric examination compensates for an inaccurate conceptual model.
An National Research Council (2000f) study Research
FIGURE I-1 Flow chart diagram illustrating the performance assessment process.
Needs in Subsurface Science for DOE’s Environmental Management Science Program recommended research on the development of tools and methodologies for conceptual modeling with an emphasis on heterogeneity, scale, and uncertainty bounds of field experimental data. An NRC (2001e) panel that described the processes through which conceptual models of flow and transport in the fractured vadose zone are developed, concluded that the development of the conceptual model is the most important part of the modeling process.1
The next steps are to convert the conceptual model into a set of mathematical equations, referred to as the mathematical model and to solve those equations. The site model is developed by inputting data and boundary conditions specific to the site when the computer code is run. For performance assessment, these mathematical models must be able to reconstruct or predict variations of quantities such as concentrations of a contaminant in air, surface water, or groundwater in both space and time. Computer codes have been developed to calculate results from the governing equations using analytic (exact) solutions or numerical methods to find approximate solutions, depending on the equations and the complexity of the environment to be modeled.
Numerical schemes subdivide time and space, which are continuous, into discrete blocks (the time step and spatial
grid) for computation. For the calculation, parameter values do not vary within the time step or within the spatial grid block. This calculation scheme (discretizing continuous variables) inherently produces errors, which are referred to as numerical errors. The magnitudes of these errors are proportional to the spatial grid size and the length of the computational time steps used in the model. In selecting a code, good analysts consider the suitability of the code for the specific application and verify and calibrate the code’s ability to simulate conditions in the site environmental system with acceptable prediction accuracy.
Code suitability and acceptable accuracy depend not only on the physical disposal system, but also on the performance objectives. For example, in nuclear waste disposal problems, the simulation time horizons typically are thousands of years. If compliance with the performance objective requires projection of environmental impacts extending thousands of years into the future, a good analyst will establish whether the cumulative errors produced by a model yield meaningful results. It should be noted that the development of a single numerical model to capture all of the processes identified in the conceptual model is unlikely. Actual performance assessment “models” in site-specific analyses typically use a set or ensemble of codes that are linked in a sequential manner. This ensemble may contain both analytically based codes and numerical solutions.
After the selection of the codes(s), the next step in constructing the site model is the development of numerical models that are intended to simulate site-specific conditions. Assumptions about site-specific conditions used in modeling are defined by assumptions about boundary conditions and model input parameters, which are based on site characteristics to the extent practical.
This is a critical phase of modeling because the best available site data must be used in this step of model development to ensure overall model quality and accuracy. Because monitoring data are limited for any site, the same data that are used in conceptual model development are often used in numerical model testing. Additional data may have to be assembled or gathered as required by the specific needs of the code. A characteristic of performance assessment modeling at these sites is the high level of sophistication needed in the code to simulate complex hydrogeochemical processes in the subsurface. This subsurface modeling places a greater demand on the type of data needed by state-of-the-art codes. This demand for specific data resources means that it is reasonable to expect that all of the data required for a model will not be available and the available data will have uncertainties.
The next step in development of the site model is calibration, which involves adjustment of the model parameters to match existing data or observations. The goal of the calibration step is to build confidence in the ability of the code to simulate contaminant movement accurately under site conditions. One expects a calibrated model to be able to simulate the behavior of the system during a period for which both input and output data on system response are available. In groundwater modeling, a rule of thumb is that predications can be made with some confidence for a period equal to the period of the calibrated match (Bredehoeft, 2003). The confidence of the predictions made beyond the length of the calibration period will diminish rapidly. When applied to the disposal of long-lived radioactive waste, the persistence of the hazard creates a tension between the need to meet the performance objectives over a long time horizon and the diminishing confidence in predictions for a time horizon that is orders of magnitude larger than the calibration period. Bredehoeft (2003) states that “the closer we can approach the idea of a long history with which to match the models, even models of nuclear waste facilities, the more confidence we will have in the analysis (and the models, including performance assessment).” He argues that prolonged periods of site monitoring, perhaps as long as 300 to 1,000 years, are required to update codes to calibrate models to gain confidence in long-term predictions before a nuclear waste facility is finally closed.
If the model is “calibrated,” a base prediction can be made by simulating the expected behavior of the system for future events. This simulation is a single outcome corresponding to the deterministic dataset used in building the model. This base prediction is not sufficient for performance assessment because uncertainty is inherent in all steps of the process, starting from conceptual model development to scenarios that are assumed in the simulations. According to the NRC (2001e) panel, “It is important to recognize that model predictions require assumptions about future events or scenarios, and are subject to uncertainty.” For example, the performance assessments include assumptions about future scenarios that may include the failure of engineered barriers or a change in climate that will alter the frequency and magnitude of floods, droughts, and precipitation. Data limitations and uncertainties associated with the characterization of subsurface heterogeneity also contribute to prediction errors. The reliability of deterministic approaches to conducting flow and transport analysis in complex subsurface systems has been questioned. Methods that use geostatistical techniques to describe the spatial variability and scaling and stochastic analysis of fluid flow and solute transport have become the trend (Neuman and Wierenga, 2003). Stochastic methods enable analysts to explore data uncertainties and alternative scenarios systematically. In the absence of a stochastic approach, a comprehensive, deterministic sensitivity study, which systematically examines the impact of different scenarios and parameter variations on the risk posed by the site, provides a similar examination of uncertainties and the impacts of assumptions, which is essential to decision making. Another informative approach is to examine results of a so-called practical worst case that would consider multiplicative effects in parameter variations in determining potential environmental exposures. The predictions based on
sensitivity analysis are then used for the assessment of long-term impacts and risk analysis as part of the performance objective evaluation. Such methods do not necessarily improve the accuracy (realism) of model results, but they can more accurately represent the consequences of what is known about the systems.
As presented, a conceptual model is developed as a first step in the PA process using the available data. As more data become available through continuous monitoring, this initial conceptual model may require updating. This need would be apparent through regular recalibration and testing of predictions made earlier. Neuman and Wierenga (2003) encourage an iterative approach to modeling, whereby a preliminary conceptual-mathematical model is gradually refined. In addition to updating the conceptual model, well-designed and well-thought-out monitoring systems could provide data to improve the numerical model. A critical component of this iterative improvement of performance assessments is the external review of a performance assessment at its conception, and continuing external review as the performance assessment is updated. If done well, the performance assessment will be an evolving procedure that builds confidence in predictions and in its ability to carefully define the risks posed to public health and the environment by waste discharges on a site.
Presentation of the performance assessment is as important as any other step in the method because if the results are not communicated clearly to decision makers, the performance assessment has not done its job. The performance assessment is best documented in a comprehensive report that describes the basis for all of the assumptions that underpin the conceptual model, the development of simulation models, model calibration, and predictive analysis. In a good performance assessment, the basis for selection of compliance points is also clearly described. Review of a performance assessment is easier to complete if the most recent assessment is readily available as a single, updated document, rather than as a series of reports that may have been released over a number of years.
