5

Incorporating Uncertainty into Decision Making

As outlined in Chapter 1, the committee focused on the uncertainty in three types of factors that can play a role in the decisions of the U.S. Environmental Protection Agency (EPA): health, technological, and economic. Historically, uncertainties in health estimates have received the most attention (see Chapter 2). Uncertainties in technological and economic factors have received less attention (see Chapter 3). In this chapter, the committee presents a framework to help EPA incorporate uncertainty in the three factors into its decisions. Where possible, the committee incorporates the lessons from other public health agencies discussed in Chapter 4.

Science and Decisions: Advancing Risk Assessment (hereafter Science and Decisions) (NRC, 2009) recommended a three-phase decision-making framework consisting of problem-formulation, assessment, and management phases. Both Science and Decisions and the framework it suggests emphasize the need to do a better job of linking the assessment of health risks to the particular problem that EPA is facing and also emphasize the importance of stakeholder involvement in each stage.

In this chapter this committee begins by building on that three-phase framework, incorporating into that framework uncertainty in the three factors (health risk estimates, technology availability, and economics) that play a role in EPA’s decisions. As with the framework from Science and Decisions (NRC, 2009) and other decision-making frameworks (see, for example, Gregory, 2011; Gregory et al., 1996; Spetzler, 2007), this committee’s framework emphasizes the importance of interactions between decision makers, stakeholders, and analysts. The modified framework is presented in Figures 5-1a and 5-1b and is discussed below. After introducing the



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 147
5 Incorporating Uncertainty into Decision Making A s outlined in Chapter 1, the committee focused on the uncertainty in three types of factors that can play a role in the decisions of the U.S. Environmental Protection Agency (EPA): health, technological, and economic. Historically, uncertainties in health estimates have received the most attention (see Chapter 2). Uncertainties in technological and eco- nomic factors have received less attention (see Chapter 3). In this chapter, the committee presents a framework to help EPA incorporate uncertainty in the three factors into its decisions. Where possible, the committee incorpo- rates the lessons from other public health agencies discussed in Chapter 4. Science and Decisions: Advancing Risk Assessment (hereafter Science and Decisions) (NRC, 2009) recommended a three-phase decision-making framework consisting of problem-formulation, assessment, and manage- ment phases. Both Science and Decisions and the framework it suggests emphasize the need to do a better job of linking the assessment of health risks to the particular problem that EPA is facing and also emphasize the importance of stakeholder involvement in each stage. In this chapter this committee begins by building on that three-phase framework, incorporating into that framework uncertainty in the three fac- tors (health risk estimates, technology availability, and economics) that play a role in EPA’s decisions. As with the framework from Science and Deci- sions (NRC, 2009) and other decision-making frameworks (see, for exam- ple, Gregory, 2011; Gregory et al., 1996; Spetzler, 2007), this committee’s framework emphasizes the importance of interactions between decision makers, stakeholders, and analysts. The modified framework is presented in Figures 5-1a and 5-1b and is discussed below. After introducing the 147

OCR for page 147
PHASE I: PHASE II: PHASE III: PROBLEM FORMULATION PLANNING AND CONDUCT OF 148 RISK MANAGEMENT AND SCOPING ASSESSMENTS • What problems are associated with Stage 1: Planning of the Assessments • What are the relative benefits of the existing environmental conditions? • For the given decision context, what are the attributes of studies proposed regulatory options? • If existing conditions appear to necessary to characterize the human health risks, technology, and • Given the decision context, how should pose a threat to human health, what economics that should be assessed for the proposed regulatory options? the three factors (human health risk, options exist for altering those technological, and economic factors) be conditions? considered in the decision? • What is the legal context of the • What are the relative magnitudes of decisions and, therefore, what Stage 2a: Human Health Risk Assessment uncertainty associated with each of those factors should EPA consider in its factors, and how should the relative decisions? Stage 2b: Technology Availability Assessment magnitude of each affect the decision? • Under the given decision context, • How readily reversible is the decision in what scientific or technical Stage 2c: Economic Analysis light of new information? assessments are necessary to • Can any uncertainty be decreased within an evaluate possible management acceptable timeframe through further decisions research? • What are the potential uncertainties Stage 3: Confirmation of Utility • Is there additional information that should be associated with the relevant factors •Do the assessments have the attributes—including the appropriate collected to inform future decisions? for decision making? design, conduct and analyses —called for in the planning phase? • How should other influences, such as • What populations should be •Do the assessments provide sufficient information to discriminate environmental justice considerations, public considered in assessments? among possible regulatory actions? value considerations, and the political • Should a quantitative or qualitative •Does the research contributing to the assessment have proper context, influence the decision and how assessment of those uncertainties be oversight, peer review, all available data included, and not appear to should those influences be communicated? conducted NO YES be influenced by potential conflicts of interest and biases? • How should the overall decision be • Is there research that could be •Are uncertainties in each of the assessments adequately described and communicated? conducted within an acceptable quantified? • Are qualitative, quantitative, or a mixture of timeframe that would decrease any both methods best for communicating the of the uncertainties? uncertainty? • Are there any deep uncertainties • When and how should the decision be related to the issue? Evaluation of decision, and refinement of evaluated? problem formulation as appropriate FORMAL PROVISIONS FOR INTERNAL AND EXTERNAL STAKEHOLDER INVOLVEMENT DURING ALL PHASES • The involvement of decision makers, technical specialists, and other stakeholders in all phases of the process leading to decisions should in no way compromise the technical assessment of risk, which is conducted under its own standards and guidelines. FIGURE 5-1a Framework for decision making. SOURCE: Modified from NRC, 2009.

OCR for page 147
INCORPORATING UNCERTAINTY INTO DECISION MAKING 149 Stage 2a: Human Health Risk Assessment Hazard A H d Assessment t •What are the adverse health effects associated with the agents of concern? •What are the uncertainties (qualitative or quantitative) associated with those estimates? Exposure Assessment •What exposures/doses are incurred by each population of interest under existing conditions? •How does each regulatory option affect existing conditions and resulting exposure/doses? •What are the uncertainties (qualitative or q (q quantitative) associated with those estimates? ) Risk Characterization •What is the nature and magnitude of risk associated with existing conditions? •What risk decreases (benefits) are associated with each of the regulatory options? •Are any risks likely to be increased? •What significant uncertainties remain and what are their potential impact on management decisions? Stage 2b: Assessment of Technology Availability •What technologies are available? •What technologies are likely to soon become available? •Will a change in the regulation spur the development of new technologies? •What costs are associated with different technologies? •What are the expected decreases in emissions or exposures anticipated from the different technologies? •What uncertainties are associated with each of those estimates? Stage 2c: Economic Analysis •What are the costs associated with different regulatory options? •What are the benefits associated with different regulatory options? •What uncertainties are associated with each of those estimates? FIGURE 5-1b1 Considerations for each assessment during phase 2. framework, the report then discusses different approaches to handling un- Figure 5-1b. certainty inConsiderations for each assessment during Phase 2factors and describes ways health, technological, and economic that stakeholder engagement may be encouraged.1 INCORPORATING UNCERTAINTY INTO A DECISION-MAKING FRAMEWORK Problem Formulation The need for EPA to make a regulatory decision might arise from concerns about a potential environmental hazard, a legal requirement to review an existing or potential environmental regulation, or concerns fol- lowing a specific event, such as an oil spill or the siting of a new source of pollution. Regardless of why a decision is needed, when approaching a regulatory decision EPA should first identify and characterize the question 1  Due to a production error, Figure 5-1b was inadvertently left out of the prepublication copy of this report.

OCR for page 147
150 ENVIRONMENTAL DECISIONS IN THE FACE OF UNCERTAINTY or problem that underlies the regulatory decision. In other words, it first needs to perform a problem formulation and scoping. Science and Decisions (NRC, 2009) highlights the importance of the problem formulation phase, which includes identifying the environmental concerns, planning and determining the scope of decision making, and iden- tifying potential regulatory options and the criteria for selecting among those options. This committee agrees with the earlier report that planning for a risk assessment and anticipating issues in advance are key to conducting use- ful and high-quality assessments (such as assessments of human health risks, cost–benefit assessments, and assessments of technology availability), and the committee further emphasizes the importance of identifying uncertainties that affect the decision and determining how those uncertainties should be assessed and considered in the decision-making process. Identifying potential regulatory actions during this first phase will facilitate identifying the uncer- tainty surrounding the consequences of the regulatory actions, in order to plan any assessments of those uncertainties. Although not all stake­ olders h will necessarily agree with a regulatory decision—some will refuse to sup- port any increase in regulation, for example, while others will refuse to s­ pport any decrease in regulation—an enhanced problem formulation will u help to ensure that the different participants are aware of the different per- spectives and that many of the potential uncertainties are identified. Identify- ing and characterizing the problem and potential regulatory options as well as planning for the uncertainty analysis are discussed below. Identifying and Characterizing the Problem and Potential Regulatory Options As discussed in Understanding Risk (NRC, 1996), the assessments of health risks and other factors should be decision driven, that is, driven by the context of the decision. Stakeholders, however, often have differ- ent views and perspectives on what problem underlies or caused the need for a decision, what information is available and should be considered when making a decision, and what uncertainties could affect a decision (Koppenjan and Klijn, 2004). Those different views and perspectives could, in part, determine the most appropriate way to assess the factors in the deci- sion (such as health risks, costs, and technology). All participants, therefore, need to be aware of and understand the views and perspectives of others, as well as have a common understanding of the problem to be addressed, the purpose of the assessments, and the potential regulatory options. Complex decisions that affect multiple stakeholders benefit from a for- mal process that ensures that the problem and the solutions are adequately

OCR for page 147
INCORPORATING UNCERTAINTY INTO DECISION MAKING 151 characterized and agreed upon by all parties. Problem-structuring methods for unstructured problems which—like many of the problems EPA faces, have multiple actors and perspectives, incommensurable or conflicting in- terests, important intangibles, and key uncertainties—provide a “way of representing the situation . . . that will enable participants to clarify their predicament, converge on a potentially actionable mutual problem or issue within it, and agree [on] commitments that will at least partially resolve it” (Mingers and Rosenhead, 2004, p. 531). The interaction among stakehold- ers that occurs with problem-structuring methods typically helps not only to build a consensus about a problem, but also to build social trust (see Chapter 6 for further discussion of social trust). A small but growing literature from operations research provides guid- ance on problem structuring (see Gregory, 2011; Gregory and Keeney, 2002; Gregory et al., 1996; Hammond et al., 1984; Rosenhead, 1996; von Winterfeldt and Fasolo, 2009). According to this literature, in order to structure a problem one should (1) focus on the decision, that is, on the policy or regulatory choices and objectives; (2) maintain a broad perspec- tive, that is, do not narrow down decision alternatives or objectives too early; and (3) involve a broad range of stakeholders to assist in identifying alternatives and objectives, thereby creating a legitimate framing of the policy or regulatory problem. For environmental policy and regulation, the policy or regulatory ob- jectives could include health risk reduction, an improvement of the envi- ronment, minimizing direct implementation costs, minimizing indirect and long-term socioeconomic impacts, or identifying a solution that maximizes the net benefit. Some studies favor structuring the problem in terms of the net benefits (that is, the total benefits minus the total costs from, for example, health improvements) rather than in terms of the risk reduction (Stinnett and Mullahy, 1998). The planning of assessments should include not only assessments of health risks and benefits, but also assessments of the other factors that might be considered in a decision, in particular, technological and economic factors. Keeney and Raiffa (1976) and Keeney (1996) discussed how to generate a comprehensive set of objectives, including identifying which di- rect and indirect costs should be considered part of the objectives. Garber and Phelps’s (1992) work on the near equivalence of benefit–cost analysis and cost-effectiveness analysis, along with the method of cost acceptability curves (Fenwick et al., 2001), lead to a larger framework for analyzing un- certainty in both benefit–cost analysis and cost-effectiveness contexts. The metrics that will be used to measure the objectives should also be defined as part of this phase.

OCR for page 147
152 ENVIRONMENTAL DECISIONS IN THE FACE OF UNCERTAINTY Planning for the Uncertainty Analysis As can be seen in Figure 5-1a, planning for the analyses of uncertainty should begin during the problem-formulation phase. A major challenge is determining whether and how uncertainties should be quantified and how they should be taken into account in a regulatory decision. When consid- ering how to analyze uncertainties, the type and complexity of the uncer- tainty analyses that are appropriate will depend on, among other things, the context of the decision (for example, if it is made in an emergency situation, the level of controversy and scientific disagreement around the decision, and whether the decision would be easily reversible), the nature of the risks and benefits (for example, if the human health risks involve minor adverse events, complex quantitative uncertainty analyses might not be warranted, whereas if they involve a fatal, nonreversible disease, such analyses might be warranted), the factors considered in the decision (for example, economic, technological, or social factors), and the type (for ex- ample, variability, model uncertainty, or deep uncertainty) and magnitude of the uncertainty. In particular, environmental statutes distinguish between decision contexts that are solely based on health considerations and those that consider technological feasibility or availability, cost–benefit trade- offs, or some combination of the three different types of considerations. It is important, therefore, that EPA identify in the problem-formulation phase of its decision-making process those factors it needs to consider in the decision and the nature or type of the uncertainty in those factors. That identification could involve providing a list of items that contribute to un- certainty, such as limited data, alternative models, or disagreements among the experts. For some factors, the process may include providing ranges of estimates from the literature or some preliminary representations of uncer- tainty, such as event trees, influence diagrams, or belief nets (see Box 5-1). There is no “one-size-fits-all” approach for an agency to make deci- sions in the face of uncertainty, nor is a particular approach to uncertainty analysis appropriate for all decisions, but, in general, certain types of approaches and analyses lend themselves to certain types and sources of uncertainty. In Table 5-1, as a guide for EPA, the committee presents a ty- pology of decision situations which indicates when different approaches to handling uncertainty might be appropriate. Those approaches are discussed in more detail in Appendix A. The legal context determines which factors—health, technology, and economics—can be considered in EPA’s decisions and, therefore, should be assessed (shown in the columns in Table 5-1). Each of those three factors can exhibit any or all of the three types of uncertainty to different extents (shown in the rows in Table 5-1), and each combination of factor and type of uncertainty lends itself to a different type of uncertainty analysis.

OCR for page 147
INCORPORATING UNCERTAINTY INTO DECISION MAKING 153 BOX 5-1 Definitions of Preliminary Graphical Representations of Uncertainty Belief nets “represent the causal and noncausal structure of inferences going from data and inference elements to the main hypothesis or event about which an inference is made” (von Winterfeldt, 2007). Event trees start with an initiating event and trace that to the “fault or problem event” (von Winterfeldt, 2007). Influence diagrams are graphical or visual representations of a decision situation. Conventionally, uncertain variables, decision nodes, and value nodes are shown in ellipses, rectangles, and rounded rectangles, respectively (von Winterfeldt, 2007). While the regulatory context specifies which factors EPA can consider in making decisions, many of EPA’s decisions will involve multiple types of uncertainty. EPA’s plans for assessing uncertainty, therefore, will involve multiple analyses and approaches. The committee does not present all pos- sible analytic approaches; rather it presents a number of approaches as a starting point to indicate how EPA should plan its analyses during the first phase of its decision-making process. Looking across the columns in Table 5-1 at the legal or regulatory context, if the context is narrow—such as cases in which only health effects are taken into account (first column, Table 5-1)—then the approaches to uncertainty would typically be limited to versions of using safety or default factors (see Chapter 2 for further discussion); health risk analysis, including extreme value analysis; and scenario analysis, depending on the type or na- ture of the uncertainty. If technological availability—such as the best avail- able or best practicable technology—can be considered (second column, Table 5-1), then health effects analyses can be combined with an assessment of the availability or practicability of the technological option, estimated using direct assessments or technological choice/risk analyses, to reduce health effects. If cost–benefit factors are allowed (third column, Table 5-1), appropriate analytic approaches include cost-effectiveness analysis, cost–benefit analysis, and multiattribute utility analysis. Cost-effectiveness, cost–benefit, multiattribute utility analysis, and decision analysis do not differ if the uncertainties are in variability and heterogeneity or in models and parameters. In the case of deep uncertainty, there is a shift to scenario analysis and robust decision-making tools, but in the case of cost–benefit factors, this analysis would include deep uncertainty about all factors

OCR for page 147
154 ENVIRONMENTAL DECISIONS IN THE FACE OF UNCERTAINTY TABLE 5-1 Influence of the Type and Source of Uncertainty on Incorporating Uncertainty into a Decision REGULATORY CONTEXT: FACTORS CONSIDERED IN THE DECISIONa   Health Effects Only Technology Availability Cost–Benefit Variability and •  se of safety or U •  sing statistics for U • Using statistics for Heterogeneityb default factors o direct  o cost-effectiveness  (using statistics) assessments, and analysis, if little or no data o technological  o cost–benefit  on uncertainties choice/risk analysis, and are available, and analysis o multiattribute  •  nalysis of A utility analysis statistical distributions, including extreme value analysis if data are available Model and •  f little or no data I •  sing formal U •  sing expert U Parameter are available, expert elicitation to judgments for Uncertaintyc using expert assess technology o cost-effectiveness  judgments and availability, and analysis, the use of safety •  sing expert U o cost–benefit  or default factors, judgments for analysis, and TYPE OF UNCERTAINTY and technology choice/ o decision analysis •  f data are I risk analysis available, using expert elicitation and analysis of probability distributions, including extreme value analysis Deep Uncertaintyd Scenario analysis and robust decision-making methods   NOTES: The most appropriate methods to evaluate, analyze, or account for uncertainty often depend on the types and sources of uncertainty that are present. The columns of the matrix show which methods are typically appropriate for different regulatory contexts, that is, what factors environmental laws and execu- tive orders require the Environmental Protection Agency (EPA) to consider in a given decision. The rows of the matrix show the methods that are often appropriate for heterogeneity and variability, model and parameter uncertainty, and deep uncertainty. a The regulatory (or legal) context determines, to a large extent, what factors EPA considers in its regula- tory decisions. b The goal of assessing uncertainty from variability and heterogeneity is to identify different populations (health), technology and facilities (technology), or regulatory options (cost–benefit tradeoffs) and to estimate (with uncertainty) the magnitude of the differences among them. c The goal of assessing model and parameter uncertainty is to estimate (with uncertainty) the effect of model choice and parameter values on assessments of health risks, technological factors, and the cost–ben- efit tradeoffs of different regulatory options. d The goal is to identify deep uncertainties in the assessments, their potential effects on a decision, whether to conduct research to decrease the uncertainties, and when decisions should be revisited in light of those uncertainties. Both variability and heterogeneity, and model and parameter uncertainty, can be deep uncertainty.

OCR for page 147
INCORPORATING UNCERTAINTY INTO DECISION MAKING 155 considered in the analysis. If model or parameter uncertainty is present, expert judgments or elicitations can be helpful in estimating human health risks, technology availability, and costs and benefits. Looking down the columns in Table 5-1 shows that different types of uncertainty lend themselves to different approaches to assessing and considering uncertainty. Statistical methods are appropriate for situations involving large amounts of data that allow uncertainty assessments by fit- ting standard probability distributions to data, that is, when uncertainties are primarily related to statistical variability and population heterogeneity. Expert judgment techniques or safety or default factors are needed when models and their parameters are uncertain and when data are sparse, for example, when the slope or shape of the dose–response function is un- certain or when extrapolation from animal data to humans is necessary. When facing deep uncertainties, probabilistic methods are more limited in use, and scenario analysis, sometimes coupled with robust decision-making methods, can help (see further discussion later in this chapter). Robust decision-making methods are those that provide acceptable outcomes for a range of possible scenarios, including pessimistic ones. The goals in the assessing the different types of uncertainty are also different. For variability or heterogeneity (first row, Table 5-1), the goal of the assessment approach is to identify the subpopulations that are dif- ferentially affected, estimate the magnitude of the differences in results in the different subpopulations and the within-subpopulation variability, and assess the uncertainty in those estimates. For model and parameter uncer- tainty (second row, Table 5-1), the goal of the assessment approach is to compare results based on specifications with different functional forms, and one should compare simulations using different assumptions about the parameters depicting relationships between key explanatory variables and the dependent variables. For deep uncertainty (third row, Table 5-1), the goal or purpose of the assessment approach is fundamentally different; scenarios of various adverse outcomes should be described, and an assess- ment should be made as to whether a proposed solution can eliminate the risks of those outcomes occurring. When accounting for uncertainty in a regulatory decision, each analy- sis or approach is associated with a set of decision rules that identify the “best” regulatory decision if the decision maker were to follow the recom- mendations resulting from the analysis. For example, a decision rule for a cost–benefit analysis would be to select the regulatory option with the highest net social benefit. Further details about the specific approaches to assessing and consider- ing uncertainty in decisions are presented later in this chapter.

OCR for page 147
156 Assessment Once the decision makers, analysts, and stakeholders have a clear understanding of what assessments (human health risk assessments, eco- nomic analysis, or assessments of technology availability) are needed to inform a given decision, how those assessments should be conducted, and the uncertainties that need to be analyzed, the assessment phase begins. Assessment refers to the collection of data, modeling, and the estimation of impacts in order to determine how the regulatory options (including the status quo) perform with respect to the objectives specified in the problem- formulation stage (NRC, 2009). This is the factual part of the decision- making process and provides the analytic basis for the management phase, which involves evaluation, decision making, value-of-information analysis, and implementation. The objective of the assessment phase is to analyze the available data or evidence and provide decision makers with the analyses in a way to inform the decision, including providing information about the uncertain- ties in the data and in the overall assessment. It is crucial that analysts do not lose sight of that objective when conducting uncertainty analyses. For example, they should not use extensive resources to analyze an uncertainty in a parameter or factor that has little relevance to the overall decision. It is also crucial that decision makers understand the implications of choices that analysts might make in the assessment process. For example, decision makers need to be aware of whether any default assumptions or models are embedded in an assessment and how those defaults might affect the assessment. A main objective of EPA’s regulatory decisions is to reduce adverse hu- man health and environmental outcomes. Human health risk assessment is a well-understood and mature activity at EPA and other regulatory agen- cies. As described in Risk Assessment in the Federal Government: Managing the Process (NRC, 1983), it includes hazard identification (determining which health and environmental impacts are pertinent to the decision under consideration, with more specificity than the broader objectives specified in the first phase), an exposure assessment (assessing the levels of exposure to environmental agents), a dose–response assessment (a quantitative analysis of the effect of a unit change in exposure to particular environmental agents on specific health and environmental outcomes), and risk characteriza- tion (the health and environmental outcomes expected at a specific level of exposure to an environmental hazard). Human health risk assessment is conducted for the base case (outcomes at a future date if no change in regulation is implemented) and for one or more regulatory options. Human health risk assessment is the tool that decision makers use to predict the degree of health improvement or protection expected from a decrease in one

OCR for page 147
INCORPORATING UNCERTAINTY INTO DECISION MAKING 157 or more exposures. Such risk assessments do not, however, indicate which intervention to use—that is, which is the best way to decrease exposure. For the assessment phase, most previous NRC reports and EPA risk assessments have focused only on the assessment of health risks and their associated uncertainties. This committee, however, believes that the assess- ment phase should also include examinations of a number of nonhealth factors and their associated uncertainties. In particular, assessments should include technological factors and economic factors. The next section briefly describes where uncertainties can arise in the assessments of factors other than human health risks. For more details of assessments and assessment techniques for human health risks and for the various other factors, readers should refer back to Chapters 2 and 3, respectively. It is worth noting that uncertainties are expressed as probabilities or probability distributions. While there is some discussion in the literature about the use of qualitative (verbal) vs. quantitative (numerical) expressions of probabilities (von Winterfeldt, 2007), most studies of environmental un- certainties use quantitative probabilities because they lend themselves to a wide array of statistical and other analyses. There are also different schools of thought about what these probabilities represent, including the classical or logical view, the frequentistic view, and the subjective or Bayesian view. Without taking a side in the debate among these schools of thought, the committee takes it for granted that probabilities are always based on logic, data, and judgment and that they should be informed and revised as new information is obtained. Typically, EPA’s decisions take into consideration the costs incurred by private parties as well as by public agencies. Private parties bear the cost of mitigation to reduce health and environmental impacts, while the public sector bears the costs of monitoring and enforcement as well as other costs. Both public and private costs are likely to be uncertain for several reasons. On the private side, the costs of mitigation alternatives are often uncertain. Technological development, whose outcome is often uncertain, may be re- quired in many cases, and the uncertainties related to that development add to the uncertainty in the eventual technology costs. On the public side, there are choices to be made concerning the level of regulatory enforcement. An air standard can be enforced with more or less effort devoted to detection or prosecution; each choice implies expending a different amount of public resources. Changes in the level of enforcement are likely to lead to a change in the levels of benefits of the policy. For example, an unenforced standard may be of no benefit except perhaps to signal that some decision maker is sympathetic to a particular cause. At the same time, public expenditures on enforcement may vary from those projected at the time the policy was implemented because individu- als and firms in the private sector respond differently to the policy than

OCR for page 147
170 ENVIRONMENTAL DECISIONS IN THE FACE OF UNCERTAINTY (2) in many situations, more importantly, the cost of delaying action; and (3) the cost of modifying decisions once implemented. Some decisions are, for all practical purposes, irreversible. If the costs of data collection or research and delaying a decision are low and the costs of subsequently modifying a policy decision are high, decision makers may decide to seek further information to reduce uncertainty before making a decision. Business decisions—where the value of information is the difference between the profits with and without the information—illustrate well the concept of VOI analysis (see Box 5-2 for an example). As can be seen in Box 5-2, the value of information is not a fixed number but rather a ran- dom variable that depends on the decision maker’s prior estimate of what the new information will reveal. For this reason, the term “expected value of information” is used, referring to what the additional information is expected to be worth on average before the new information is collected. If there is no possible outcome in which additional information gathering or research would change the decision, then the expected value of information is zero. If any decision is changed for the better after some result, than the value of information is positive. If the costs of obtaining the information (either data-gathering research costs or costs from delaying a decision, as might be the case with some regulatory options) are less than the expected value of information, then it is better to get the information.7 In the theoretical world of Box 5-2, it is assumed that the additional in- formation is perfect. For example, there are no errors in predicting whether or not an event will occur. In practice, however, errors are made. The pre- diction may fail to predict that an event will or will not occur. In the real world of imperfect information, one can use Bayesian updating to incorpo- rate the uncertainty inherent in the new information. In Bayesian updating, a weight is attached to new information, and a second weight is attached to the prior belief; the weights must sum to one. Thus, if the new information is thought to be particularly credible, it will be assigned a higher weight, with a correspondingly lower weight being placed on the prior belief. In the context of VOI, if the weight on placed on the new estimate is low, then it will generally not pay to obtain the additional information (Hunink, 2001; Raiffa, 1968). In a public context, such as EPA’s decision context, the value of infor- mation is calculated in terms of the anticipated net benefits rather than the anticipated profitability calculated in the business example. The calcula- tion of net benefits can be very complex, and for many decisions the only 7  Conventionally the costs of obtaining the information are not part of the value-of-­ information calculations, but are instead compared at the end. In any case, these costs and the cost of revising the decision once made must be considered in choosing whether or not to seek additional information and, if so, what type of information is to be sought.

OCR for page 147
INCORPORATING UNCERTAINTY INTO DECISION MAKING 171 practical way to assess the effect of a policy option, as well as its inherent uncertainty, is to implement the option and pursue an active strategy of monitoring its effects, with the monitoring being done, to the extent pos- sible, in quantitative terms. While conceptually appealing, in the context of EPA’s public decision making VOI has two challenges that do not create as many difficulties in private decisions, such as the one described in Box 5-2. First, expected profit is simpler to estimate than the estimated costs involved in a deci- sion by the EPA. Second, unlike the situation with private decisions, the rationale for EPA postponing a decision and seeking new information may have to be explained to various segments of the public. That explanation will be complicated by the fact that many of the costs and assumptions underlying VOI calculations, including the credibility weights in Bayesian updating analysis, are subjective and difficult to defend. Despite those chal- lenges, VOI can be a useful approach to help determine what information is worth gathering for future decisions. Decision Implementation Implementation of a regulatory decision is an important step in the management process. This step requires significant skills in addressing often competing stakeholder, legal, and political considerations surrounding the proposed decision. Good decision making under uncertainty involves updating informa- tion through research, monitoring the implementation of regulatory action, and periodically revisiting and adapting the decision. A plan should be in place that outlines which uncertainties are being researched and when the decision will be revisited to see if uncertainty has decreased to the point that the decision should be revisited. As discussed earlier in this chapter, when decisions involve deep uncertainty, adaptive management approaches are particularly useful. Those approaches require increased monitoring and a plan for gathering more information and revisiting the decision. OTHER CONSIDERATIONS As discussed in Chapter 1, other factors in addition to human health risks, economic factors, and technology availability play an important role in many of EPA’s decisions. Although not thought of as traditional uncertainties that can be quantified, there is uncertainty in those factors that should be considered in making and communicating about with EPA’s decisions. The roles that some of those factors and the uncertainties in them play in EPA’s decisions are discussed below.

OCR for page 147
172 ENVIRONMENTAL DECISIONS IN THE FACE OF UNCERTAINTY Special Populations and Equity In some cases the regulatory problem is shaped by issues concerning special populations (e.g., the lead exposure of children) or by equity or environmental justice concerns, which have been labeled as priorities by various executive orders,8 although these orders do not have the weight of law. EPA recently issued a report, Plan EJ 2014: Legal Tools, that details the legal tools related to environmental justice that are available to the agency (EPA, 2011b). These special considerations could influence the choice of analytical approaches since approaches that emphasize net aggregate costs and ben- efits do not typically address these concerns. In particular, these factors can add to the variability and heterogeneity in estimates of health risks and economic factors. If the formal approaches described above are used in these contexts, they must be disaggregated so that the impacts they have on special populations can be examined as well as the aggregate effects. In doing so, EPA will be able to see the effects that its decisions could have on different groups and will be able to include the potential effects on those groups in the rationale for its decision. That will allow stakeholders to bet- ter understand the agency’s decision. Geographic Scope The geographic scope of a decision problem may be global, national, regional, or local. Spatial or geographic considerations are likely to intro- duce special problems into assessing and accounting for uncertainty. For example, data on a local area may be inadequate to characterize exposure or the sensitivity of populations to the exposure. Given the inadequacy of data collected on a national basis for use in decisions limited to local areas, decision making may be improved by additional data collection and analy- sis. Furthermore, the preferences of the residents in a community may differ from national averages, and those preferences can affect the values that people assign to outcomes which, in turn, will affect the economic analyses. The goal of an uncertainty analysis is to characterize how these values dif- fer, and doing so may require additional data collection. A characterization of such differences can be qualitative or quantitative. If the scope of a problem is local, such as is the case for a Superfund problem, local stakeholders (including members of the public) may pro- vide input at various times during the analysis phase. It is crucial, there- fore, to obtain stakeholder involvement in the problem-formulation phase, 8  For example, Exec. Order No. 12898. 77 FR 11752 (February 28, 2012) and Exec. Order No. 13045. 78 FR 19884 (April 23, 1997).

OCR for page 147
INCORPORATING UNCERTAINTY INTO DECISION MAKING 173 particularly with regard to decisions about the endpoints to be included in the analyses. On the other hand, if the scope of a problem is national, as is the case when setting an ambient air quality standard, the type of stake- holder involvement will be driven more by the statutory framework and agency procedures. For national issues, the stakeholders who provide input are often representatives of groups with special interests (e.g., industry, or advocacy organizations focused on a particular disease) in addition to—or even rather than—being community members. Decisions applicable to a specific geographic area are well suited to the incorporation of public values. Even when the statutory directive is for the consideration of health effects, the implementation plans will often be of great interest to local communities. For this reason EPA will often solicit input on implementation plans through written comments or at hearings in order to gather public comments at locations across the country (EPA, 2012). Identifying the effects of geographic scope on a decision in the initial, problem-formulation stage will help EPA identify important stakeholders and ensure that the variability in the perspectives can be addressed in the assessment and management phases of the decision. These concerns could affect the assessment of economic factors in particular. Uncertainty analysis and more formal approaches to decision making have not always been applied to these factors in a systematic or rigor- ous way, but some of the analytic techniques described in Chapter 2 and Appendix A could be applied to them. For example, Arvai and Gregory (2003) used multiattribute utility analysis to evaluate different approaches to stakeholder involvement in a decision related to the cleanup of a contam- inated site; one approach involved the presentation of scientific informa- tion, while the other involved the presentation of scientific information and “values-oriented information that seeks to improve the ability of nonexpert participants to make difficult trade-offs across a variety of technical and nontechnical concerns” (p. 1470). The importance of stakeholder engage- ment is discussed further below. STAKEHOLDER ENGAGEMENT Agency decision-making processes that involve stakeholders, includ- ing dialogues with stakeholders about uncertainties, can demonstrate in- tentional transparency and create, maintain, and enhance a relationship of trust between the agency and its stakeholders.9 In addition, a growing 9  The terms used to refer to the parties that can be involved in environmental decision mak- ing are varied and include “stakeholders,” “the public,” “affected parties,” and “interested parties.” The definitions of these terms (i.e., the expertise, affiliations, and perspectives of the

OCR for page 147
174 ENVIRONMENTAL DECISIONS IN THE FACE OF UNCERTAINTY body of research demonstrates that the political aspects of stakeholder processes do not sacrifice decision quality (Beierle, 2000) and that public participation (NRC, 2008) can in fact add information to and improve the quality and legitimacy of agencies’ decisions about the environment.10 Because decisions may ultimately have some impact for the stakeholders, if the decision-making process is to be fair and democratic stakeholders must be given the opportunity to be involved in making those decisions, includ- ing decisions about which uncertainties need better elucidation. Early and continuous involvement of stakeholders can also prevent delays that can occur when stakeholders are not engaged in decision making until later in the process, at which time they might take legal actions. EPA has issued much guidance on public and stakeholder involvement in its programs and activities (EPA, 1998, 2003, 2011a), and there are several regulations that contain public involvement procedures for specific EPA programs and activities.11 The EPA also issued an agency-wide public involvement policy (reissued periodically with updates) that can be applied to all EPA programs and activities (EPA, 2003).12 The agency-wide policy is not mandatory, however. In spite of the existing guidance, there has been repeated concern and criticism over the failure of EPA to engage stakehold- ers more systematically and adequately as part of its various regulatory mandates for environmental decision making (see, for example, NRC, 1996, 2008; Presidential/Congressional Commission on Risk Assessment and Risk Management, 1997). This was the justification for a recommen- dation made in Science and Decisions (NRC, 2009) that EPA adopt formal provisions for stakeholder involvement across a three-phase framework for individuals and organizations they include) have also varied. Unless otherwise specified, in this report we use stakeholder to refer to any parties interested in or affected by a decision-making authority’s activities. Stakeholders may include decision makers, industry groups, communities and community organizations, environmental organizations, scientists and technical special- ists, individuals from the public, and others. 10  For a comprehensive review of research on public participation in environmental assess- ment and decision making, the reader is encouraged to refer to NRC, 2008. 11  See, for example, 40 CFR Part 25—Public Participation in Programs under the Resource Conservation and Recovery Act, the Safe Drinking Water Act, and the Clean Water Act; 40 CFR Part 271—Requirements for Authorization of State Hazardous Waste Programs; 40 CFR Part 300—National Oil and Hazardous Substances Pollution Contingency Plan, Subpart E—Hazardous Substance Response (establishes methods and criteria for determining the ap- propriate extent of response authorized by CERCLA and CWA section 311(c)). 12  According to the guidance, the seven basic steps to effective public involvement are to (1) plan and budget for public involvement activities, (2) identify the interested and affected public, (3) consider providing technical or financial assistance to the public to facilitate in- volvement, (4) provide information and outreach to the public, (5) conduct public consultation and involvement activities, (6) review and use input and provide feedback to the public, and (7) evaluate public involvement activities (EPA, 2003).

OCR for page 147
INCORPORATING UNCERTAINTY INTO DECISION MAKING 175 risk-based decision making (see Figure 5-1).13 This recommendation echoes the point made in other NRC reports (see, for example, NRC, 1996, 2008) that technical and analytical aspects of the decision-making process be bal- ­ anced with adequate involvement by interested and affected parties, and it is a point with which this committee concurs. Concerns about procedural fairness and trust are even more salient when scientific uncertainty is reported (NRC, 2008). Some research has demonstrated that people show a heightened interest in evaluating the credibility of information sources when they perceive uncertainty (Brashers, 2001; Halfacre et al., 2000; van den Bos, 2001), and they are also more likely to challenge the reliability and adequacy of risk estimates and be less accepting of reassurances in such situations (Kroll-Smith and Couch, 1991; Rich et al., 1995). When EPA anticipates more uncertainty in scien- tific aspects of decision making, the need for stakeholder involvement may often be greater. Other research has spoken to the importance of describing the existence of uncertainties in risk assessments as well, both to facilitate transparency and to increase public perceptions of agency honesty (Johnson and Slovic, 1995; Lundgren and McMakin, 2004; Morgan and Henrion, 1990; NRC, 1989). Developing provisions for stakeholder involvement in decision mak- ing, including guidance on discussing with stakeholders the sources of uncertainty and how uncertainty is being managed, could lead to greater transparency and trust and also has the potential to result in better deci- sion making. Stakeholders might be interested in how uncertainty can be dealt with in the analysis, the implications of uncertainties, and what can or cannot be done about the uncertainties. Stakeholders may also suggest new uncertainties not previously under consideration by EPA and, by ex- pressing their values and concerns (cultural, religious, economic, and so on), help decision makers prioritize how the uncertainties are factored into decision making. In discussions with stakeholders about uncertainty, it is important that EPA be proactive in engaging the range of stakeholders for whom a decision may have an impact. Science and Decisions (NRC, 2009) recom- mended that EPA provide incentives to allow for balanced participation of stakeholders, including affected communities and those stakeholders for whom participation is less likely because of competing priorities, fewer resources, a lack of knowledge, or other factors. Boeckmann and 13  The three phases are (1) problem formulation and scoping, (2) planning and conduct of risk assessment, and (3) risk-management phases (see Figure 5-1). As part of the framework, the report also suggests that stakeholder involvement should have time limits so as not to delay decision making and that there should be incentives so that participation is more balanced and includes impacted communities and less advantaged stakeholders (NRC, 2008).

OCR for page 147
176 ENVIRONMENTAL DECISIONS IN THE FACE OF UNCERTAINTY Tyler (2002) found that the public is more likely to participate “in their communities when they feel that they are respected members of those communities” (p. 2067). Showing respect, therefore, is important for stakeholder engagement. The resources required for such an engagement of stakeholders, however, must be weighed against the need for such ac- tions, given the context of the decision, including consideration of the potential health risks, the costs associated with the potential regulatory options, and the magnitude, sources, and nature or type of the uncertainty associated with the decision. KEY FINDINGS • Incorporating uncertainty analysis into a systematic framework, such as a modified version of the decision framework in Science and Decisions (NRC, 2009), provides a process for decision mak- ers, stakeholders, and analysts to discuss the appropriate and nec- essary uncertainty analyses. • Involvement of decision makers in the planning and scoping of uncertainty analyses during the initial, problem-formulation phase will help ensure that the goals of the uncertainty analysis are con- sistent with the needs of the decision makers. • Involvement of stakeholders in the planning and scoping of uncer- tainty analyses during the initial problem-formulation phase will help define analytic endpoints and identify population subgroups as well as heterogeneity and other uncertainties. • Uncertainty analysis must be designed on a case-by-case basis. The choice of uncertainty analysis depends on the context of the deci- sion, including the nature or type of uncertainty (that is, hetero- geneity and variability, model and parameter uncertainty, or deep uncertainty), and the factors that are considered in the decision (that is, health risk, technology availability, and economic, social, and political factors) as well as the data that are available. • When assessing variability and heterogeneity: o  nalyses of statistical distributions, including extreme-value A analyses, are useful for assessing uncertainty in data on health effects (that is, estimates of risks). The use of safety or de- fault factors (using statistics) can also be helpful under certain circumstances. o  irect assessments and technological choice or risk analyses de- D veloped using statistics can be helpful for assessing technological availability.

OCR for page 147
INCORPORATING UNCERTAINTY INTO DECISION MAKING 177 o  ost-effectiveness, cost–benefit analysis, and multiattribute util- C ity analysis developed using statistical methods can be useful for assessing costs and benefits. • When assessing model and parameter uncertainty: o xpert elicitation and the analysis of probability distributions, E including extreme value analyses, can be useful for assessing health effects. Safety or default factors developed using expert judgments can also be helpful. o ormal expert elicitation to assess technology availability, as F well as technology choice and risk analysis using expert judg- ment, can be helpful in assessing technology factors. o  ost-effectiveness, cost–benefit analysis, and multiattribute util- C ity analysis developed using expert judgments can be useful for assessing costs and benefits. • When assessing deep uncertainty: o  Scenario analysis and robust decision-making methods can be helpful for assessing health effects, technology factors, and costs and benefits. • The interpretation and incorporation of uncertainty into environ- mental decisions will depend on a number of characteristics of the risks and the decision. Those characteristics include the distribution of the risks, the decision makers’ risk aversion, and the potential consequences of the decision. • The quality of the analysis and recommendations following from the analysis will depend on the relationship between analyst and the decision maker. The planning, conduct, and results of uncer- tainty analysis should not be isolated from the individuals who will eventually make the decisions. The success of a decision in the face of uncertainty depends on the analysts having a good understand- ing of the context of the decision and the information needed by the decision makers, and the decision makers having a good under- standing of the evidence on which the decision is based, including an understanding of the uncertainty in that evidence. RECOMMENDATION 7 Although some analysis and description of uncertainty is always im- portant, how many and what types of uncertainty analyses are carried out should depend on the specific decision problem at hand. The effort to analyze specific uncertainties through probabilistic risk assessment or quantitative uncertainty analysis should be guided by the ability of those analyses to affect the environmental decision.

OCR for page 147
178 ENVIRONMENTAL DECISIONS IN THE FACE OF UNCERTAINTY REFERENCES Arvai, J., and R. Gregory. 2003. Testing alternative decision approaches for identifying cleanup priorities at contaminated sites. Environmental Science and Technology 37(8):1469–1476. Beierle, T. C. 2000. The quality of stakeholder-based decisions: Lessons from the case study record. Washington, DC: Resources for the Future. Boeckmann, R. J., and T. R. Tyler. 2002. Trust, respect, and the psychology of political engage- ment. Journal of Applied Social Psychology 32(10):2067–2088. Brashers, D. E. 2001. Communication and uncertainty management. Journal of Communica- tion 51(3):477–497. Claxton, K., P. Neumann, S. Araki, and M. Weinstein. 2001. Bayesian value-of-information analysis. An application to a policy model of Alzheimer’s disease. International Journal of Technology Assessment in Health Care 17(1):38–55. EPA (U.S. Environmental Protection Agency). 1998. EPA stakeholder involvement action plan. Washington, DC: Environmental Protection Agency. http://www.epa.gov/publicinvolvement/ siap1298.htm (accessed November 20, 2012). ———. 2003. Public involvement policy of the U.S. Environmental Protection Agency. Wash- ington, DC: Environmental Protection Agency. http://www.epa.gov/publicinvolvement/ pdf/policy2003.pdf (accessed January 3, 2013). ———. 2010. Voluntary Children’s Chemical Evaluation Program (VCCEP). http://www.epa. gov/oppt/vccep (accessed November 20, 2012). ———. 2011a. Expert Elicitation Task Force white paper. Washington, DC: Environmental Protection Agency. http://www.epa.gov/stpc/pdfs/ee-white-paper-final.pdf (accessed Janu- ary 3, 2013). ———. 2011b. Plan EJ 2014: Legal tools. Washington, DC: Environmental Protection Agency. http://www.epa.gov/environmentaljustice/resources/policy/plan-ej-2014/ej-legal-tools.pdf (accessed January 3, 2013). ———. 2012. The plain English guide to the Clean Air Act. http://www.epa.gov/air/caa/peg/ public.html (accessed May 24, 2012). Fenwick, E., K. Claxton, and M. Sculpher. 2001. Representing uncertainty: The role of cost- effectiveness acceptability curves. Health Economics 10(8):779–787. Garber, A. M., and C. E. Phelps. 1992. Economic foundations of cost-effective analysis. NBER working paper series no. 4164. Cambridge, MA: National Bureau of Economic Research. http://www.nber.org/papers/w4164.pdf (accessed January 3, 2013). Gregory, R. 2011. Structured decision making: A practical guide to environmental manage- ment choices. Hoboken, NJ: Wiley-Blackwell. Gregory, R. S., and R. L. Keeney. 2002. Making smarter environmental management decisions. Journal of the American Water Resources Association 38:1601–1612. Gregory, R., T. C. Brown, and J. L. Knetsch. 1996. Valuing risks to the environment. Annals of the American Academy of Political and Social Science 545:54–63. Halfacre, A. C., A. R. Matheny, and W. A. Rosenbaum. 2000. Regulating contested local hazards: Is constructive dialogue possible among participants in community risk manage- ment? Policy Studies Journal 28(3):648–667. Hammitt, J. K., and J. A. K. Cave. 1991. Research planning for food safety: A value of infor- mation approach. Washington, DC: RAND. Hammond, K. R., B. F. Anderson, J. Sutherland, and B. Marvin. 1984. Improving scientists’ judgments of risk. Risk Analysis 4(1):69–78. Howard, R. A. 2007. The foundations of decision analysis revisited. In Advances in decision analysis, edited by W. Edwards, R. F. Miles, and D. Von Winterfeldt. New York: Cam- bridge University Press. Pp. 32–56.

OCR for page 147
INCORPORATING UNCERTAINTY INTO DECISION MAKING 179 Hunink, M. G. M. 2001. Decision making in health and medicine: Integrating evidence and values. Cambridge; New York: Cambridge University Press. IPCC (International Panel on Climate Change). 2007. Climate change 2007: Synthesis report. Contribution of Working Group I, II and III to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change. http://www.ipcc.ch/publications_and_data/ publications_ipcc_fourth_assessment_report_synthesis_report.htm (accessed November 12, 2012). Johnson, B. B., and P. Slovic. 1995. Presenting uncertainty in health risk assessment: Initial studies of its effects on risk perception and trust. Risk Analysis 15(4):485–494. Keeney, R. L. 1996. Value-focused thinking: A path to creative decision making. Cambridge, MA: Harvard University Press. Keeney, R. L., and H. Raiffa. 1976. Decisions with multiple objectives. Cambridge, UK: Cambridge University Press. Koppenjan, J., and E.-H. Klijn. 2004. Managing uncertainties in networks: A network ap- proach to problem solving and decision making. London: Routledge. Kroll-Smith, J. S., and S. R. Couch. 1991. As if exposure to toxins were not enough: The social and cultural system as a secondary stressor. Environmental Health Perspectives 95:61–66. Lundgren, R. E., and A. H. McMakin. 2004. Risk communication: A handbook for commu- nicating environmental, safety, and health risks. Columbus: Battelle Press. Mingers, J., and J. Rosenhead. 2004. Problem structuring methods in action. European Jour- nal of Operational Research 152(3):530–554. Morgan, M. G., and M. Henrion. 1990. Uncertainty: A guide to dealing with uncertainty in quantitative risk and policy analysis. New York: Cambridge University Press. Nordhaus, W. D., and D. Popp. 1997. What is the value of scientific knowledge? An applica- tion to global warming using the price model. Energy Journal 18(1):1–45. NRC (National Research Council). 1983. Risk assessment in the federal government: Manag- ing the process. Washington, DC: National Academy Press. ———. 1989. Improving risk communication. Washington, DC: National Academy Press. ———. 1996. Understanding risk: Informing decisions in a democratic society. Washington, DC: National Academy Press. ———. 2008. Public participation in environmental assessment and decision making. Wash- ington, DC: The National Academies Press. ———. 2009. Science and decisions: Advancing risk assessment. Washington, DC: The Na- tional Academies Press. Presidential/Congressional Commission on Risk Assessment and Risk Management. 1997. Risk assessment and risk management in regulatory decision-making. Final report. Vol- ume 2. Washington, DC: Presidential/Congressional Commission on Risk Assessment and Risk Management. Rabl, A., and B. van der Zwaan. 2009. Cost–benefit analysis of climate change dynamics: Uncertainties and the value of information. Climatic Change 96(3):313–333. Raiffa, H. 1968. Decision analysis: Introductory lectures on choices under uncertainty. Read- ing, MA: Addison-Wesley. Rich, R. C., M. Edelstein, W. K. Hallman, and A. H. Wandersman. 1995. Citizen participa- tion and empowerment: The case of local environmental hazards. American Journal of Community Psychology 23(5):657–676. Rosenhead, J. 1996. What’s the problem? An introduction to problem structuring methods. Interfaces 117–131. Spetzler, C. S. 2007. Building decision competency in organizations. In Advances in decision analysis: From foundations to applications, edited by W. Edwards, R. F. Miles, and D. von Winterfeldt. New York: Cambridge University Press. Pp. 451–468.

OCR for page 147
180 ENVIRONMENTAL DECISIONS IN THE FACE OF UNCERTAINTY Stinnett, A. A., and J. Mullahy. 1998. Net health benefits: A new framework for the analysis of uncertainty in cost-effectiveness analysis. Medical Decision Making 18(2 Suppl.):S68–S80. van den Bos, K. 2001. Uncertainty management: The influence of uncertainty salience on reactions to perceived procedural fairness. Journal of Personality and Social Psychology 80(6):931–941. von Winterfeldt, D. 2007. Defining a decision analytic structure. In Advances in decision analysis: From foundations to applications, edited by W. Edwards, R. Miles, and D. von Winterfeldt. New York: Cambridge University Press. Pp. 81–103. von Winterfeldt, D., and B. Fasolo. 2009. Structuring decision problems: A case study and reflections for practitioners. European Journal of Operational Research 199(3):857–866. Yohe, G. 1996. Exercises in hedging against extreme consequences of global change and the expected value of information. Global Environmental Change 6(2):87–101. Yokota, F., and K. M. Thompson. 2004a. Value of information analysis in environmental health risk management decisions: Past, present, and future. Risk Analysis 24(3):635–650. ———. 2004b. Value of information literature analysis: A review of applications in health risk management. Medical Decision Making 24(3):287–298. Yokota, F., G. Gray, J. K. Hammitt, and K. M. Thompson. 2004. Tiered chemical testing: A value of information approach. Risk Analysis 24(6):1625–1639.