7

Synthesis and Recommendations

Human health risk estimates are a cornerstone of the analyses that the U.S. Environmental Protection Agency (EPA) carries out to help it make its decisions. The uncertainties in those risk estimates have been the subject of much advice from the National Research Council (NRC) and others, many of which have recommended quantitative uncertainty analysis and discussed the use of default assumptions, probabilistic analysis, and the quantitative characterization of uncertainty (NRC, 1983, 1994, 1996, 2009; Presidential/Congressional Commission on Risk Assessment and Risk Management, 1997). EPA has made substantial advances in developing and using analytic techniques for assessing and narrowing the uncertainties in human health risk assessments, and EPA’s assessments have evolved and become more complex over the years (GAO, 2006; NRC, 2009). As a recent publication highlights, however, the quantitative or probabilistic uncertainty analyses that are now sometimes conducted can be more extensive than is needed by the decision maker, and might not contribute to EPA’s public health and environmental objectives (Goldstein, 2011). NRC reports have also emphasized the need to improve the utility of health risk assessments, and of the uncertainty analyses that are a part of the risk-assessment process (NRC, 1996, 2009).

Through its review of the risk assessment, uncertainty analysis, and decision making at EPA and other public health entities, the committee found that, in general, more emphasis should be placed on the uncertainty in the factors that affect EPA’s decisions besides estimates of human health. Uncertainties pervade not only the relationships between hazards and health outcomes, but also other important factors, such as the willingness to pay



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 217
7 Synthesis and Recommendations H uman health risk estimates are a cornerstone of the analyses that the U.S. Environmental Protection Agency (EPA) carries out to help it make its decisions. The uncertainties in those risk estimates have been the subject of much advice from the National Research Council (NRC) and others, many of which have recommended quantitative uncer- tainty analysis and discussed the use of default assumptions, probabilistic analysis, and the quantitative characterization of uncertainty (NRC, 1983, 1994, 1996, 2009; Presidential/Congressional Commission on Risk Assess- ment and Risk Management, 1997). EPA has made substantial advances in developing and using analytic techniques for assessing and narrowing the uncertainties in human health risk assessments, and EPA’s assessments have evolved and become more complex over the years (GAO, 2006; NRC, 2009). As a recent publication highlights, however, the quantitative or probabilistic uncertainty analyses that are now sometimes conducted can be more extensive than is needed by the decision maker, and might not contribute to EPA’s public health and environmental objectives (Goldstein, 2011). NRC reports have also emphasized the need to improve the utility of health risk assessments, and of the uncertainty analyses that are a part of the risk-assessment process (NRC, 1996, 2009). Through its review of the risk assessment, uncertainty analysis, and de- cision making at EPA and other public health entities, the committee found that, in general, more emphasis should be placed on the uncertainty in the factors that affect EPA’s decisions besides estimates of human health. Un- certainties pervade not only the relationships between hazards and health outcomes, but also other important factors, such as the willingness to pay 217

OCR for page 217
218 ENVIRONMENTAL DECISIONS IN THE FACE OF UNCERTAINTY for improvements in health, the pace of technological change, the impor- tance of recreational areas, and the cost to the private sector of implement- ing new rules. As part of a systematic approach to decision making, EPA should plan its analysis of the uncertainty analyses in estimates of health risks and those other factors around the needs of the decision makers beginning at the outset of the decision-making process; in other words, uncertainty analyses should not be an afterthought. Characterizations of health risk estimates, benefits, costs, technological availability, and other factors should reflect those uncertainty analyses. The implications of those different sources of uncertainty are presented in Box 7-1. The appropriate uncertainty analysis depends, in part, on the type of uncertainty (variability and heterogeneity, model and parameter uncertainty, or deep uncertainty). Statistical analyses are often appropriate for assess- ing variability and heterogeneity, expert judgments and elicitations work well for model and parameter uncertainty, and robust decision-making ap- proaches and scenario analysis will be needed for decision making in the presence of deep uncertainty. Good communication among analysts, decision makers, and stakehold- ers is critical to ensuring a high-quality, comprehensive decision-making process and a high-quality, comprehensive decision. A process that includes such communication will help identify stakeholder concerns and potential uncertainties and to build social trust among the participants in the process. Each EPA decision is unique, and there is no universal best approach or tool for communicating uncertainties. The most appropriate strategy for communicating uncertainty will depend on the context of the decision, the purpose of the communication, the type of uncertainty, and the characteris- tics of the audience, including the level of technical expertise, personal and group biases, and the level of social trust. Analyzing and communicating the uncertainty in the various factors that affect EPA’s decisions might require specialized expertise (for example, expertise in the analysis of benefits and cost and the uncertainties in each and also in communicating those uncertainties), and some of the necessary skills may be different from those found among EPA’s current personnel. The committee’s specific findings and recommendations are presented below.

OCR for page 217
SYNTHESIS AND RECOMMENDATIONS 219 BOX 7-1 Implications of Uncertainty Analysis for Decision Making Health Uncertainties Uncertainty analyses in human health risk estimates can help decision makers to • evaluate alternative regulatory options; • assess how credible extreme risk estimates are and how much to rely on them in decision making; • weigh the marginal decrease in risk against the effort made to reduce it; • clarify issues within a decision by using scenarios to characterize very different worlds; and • in the case of scenario analyses for deep uncertainty, identify regulatory solutions that are effective over a broad spectrum of scenarios. Uncertainties About Technology Availability Uncertainty analyses in technology availability can help decision makers to • differentiate between well-established technologies with reasonably well-known costs, and those that have not been used for the purposes at hand; and • consider which technology may be considered “best practicable” or “best available” by providing information about both the likelihood of success of the unproven technologies the time frame for success, and the effectiveness if successful. Uncertainties About Cost–Benefit Analyses Given the highly uncertain estimates of both health benefits and costs, uncertainty analyses in cost–benefit analyses can inform decision makers about • how difficult it is to differentiate among different potential decisions; • the disagreement among experts about the way regulation affects the economy, even when using similar models; and • the ranges and sensitivity of estimates to different variables. FINDINGS AND RECOMMENDATIONS Uncertainties in the Characterization of Human Health Risks Finding 1 Decision documents (such as documents that the technical experts at EPA prepare to explain site-specific decisions) often lack a robust discussion

OCR for page 217
220 ENVIRONMENTAL DECISIONS IN THE FACE OF UNCERTAINTY of the uncertainties identified in the extensive health risk assessments pre- pared by agency scientists. Although decision documents and communi- cations by the agency about its decisions should be succinct, open, and transparent, they should also include information on what uncertainties are present, which uncertainties need to be addressed, and how those uncer- tainties affected a decision. It should be clear from agency communications that uncertainty is inherent in science, including the science that informs EPA decisions. In addition to contributing to full transparency, providing such infor- mation could mitigate attempts to use the existence of uncertainties as a rationale for delayed decision making. A shift in the expectations of those who read the decision documents and other public communications could eventually occur, so that the discussion of uncertainty, even unresolved uncertainty, could eventually come to be considered normal and valuable. The decision maker will be able to use the analyses most responsibly if the known sources of uncertainties are acknowledged and described, even if the uncertainties are large, poorly described, or not currently resolvable. When uncertainties are so extensive and relevant so as to likely undermine the credibility and quality of a decision, it is the responsibility of staff analysts to advise decision makers as soon as possible in order to give the decision makers a chance to revisit the rulemaking schedule, objectives, and resources, which can require a judgment call. RECOMMENDATION 1 To better inform the public and decision makers, U.S. Environmental Protection Agency (EPA) decision documents and other communica- tions to the public should systematically • include information on what uncertainties in the health risk assess- ment are present and which need to be addressed, • discuss how the uncertainties affect the decision at hand, and • include an explicit statement that uncertainty is inherent in science, including the science that informs EPA decisions. Uncertainty in Other Factors That Influence a Decision Finding 2 Although EPA decisions have included discussion and consideration of the uncertainties in the health risk assessment, the agency has gener- ally given less attention to uncertainties in other contributors influenc- ing the regulatory decision. Those contributors include economic, and

OCR for page 217
SYNTHESIS AND RECOMMENDATIONS 221 technological factors, as well as other factors that are not easily quantified, such as environmental justice. A major challenge to decision making in the face of uncertainty is the uncertainty in those other factors. Although every uncertainty does not need be analyzed for every decision, particularly when not important to a decision or when specific values are prescribed by other offices (for example, the value of a life that is set by the Office of Manage- ment and Budget), methods and processes should be available for when such analyses are appropriate and helpful to a decision maker. In general, this might require a research program to develop methods for this new type of uncertainty analysis, changes in decision documents and other analyses, and a program for research on communicating uncertainties. RECOMMENDATION 2 The U.S. Environmental Protection Agency should develop methods to systematically describe and account for uncertainties in decision- relevant factors in addition to estimates of health risks—including technological and economic factors—into its decision-making process. When influential to a decision, those new methods should be subject to peer review. Finding 3 EPA has developed guidance about, and conducted in-depth analyses of, the costs and benefits of major decisions. Some such analyses are con- ducted because of statutory mandate; others are conducted in response to a series of executive orders from the Office of Management and Budget mandating regulatory review. The analytic tools for cost–benefit analyses in health are well developed, and EPA guidance contains appropriate advice about the conduct of these analyses, including the discussion of some un- certainties. The committee, however, noted a lack of transparency regarding uncertainty analyses in the cost–benefit assessments in some EPA decision documents. While economists or modelers could evaluate the technical documents supporting the decision documents, and the economic models do include uncertainty analyses such as sensitivity analyses, the information about these uncertainties is often arcane, hard to locate, and technically very challenging to non-experts. Those analyses often shape regulatory deci- sions, and when they are conducted they should be described in ways that are useful and interpretable for the decision maker and stakeholders. The needs of the two audiences—that is, technical and non-expert audiences— differ, but a given set of decision documents and supporting analyses could include descriptions that explain the sources of uncertainties to the non- expert, and link, either electronically or via text, to more detailed descrip- tions of the economic analyses as appropriate for experts.

OCR for page 217
222 ENVIRONMENTAL DECISIONS IN THE FACE OF UNCERTAINTY RECOMMENDATION 3 Analysts and decision makers should describe in decision documents and other public communications uncertainties in cost–benefit analyses that are conducted, even if not required by statute for decision mak- ing, the analyses should be described at levels that are appropriate for technical experts and non-experts. Finding 4 The role of uncertainty in the costs and benefits and availability and feasibility of control technologies is not well investigated or understood. The evidence base for those factors is not robust. Evaluating case studies of past rulemaking and developing a directed research program on assess- ing the availability of technology might be first steps toward understanding the robustness of technology feasibility and economic assessments, and the potential for technology innovation. RECOMMENDATION 4 The U.S. Environmental Protection Agency (EPA) should fund research, conduct research or both, to evaluate the accuracy and predictive ca- pabilities of past assessments of technologies and costs and benefits for rulemaking in order to improve future efforts. This research could be conducted by EPA staff or else by nongovernmental policy analysts who might be less subject to biases. This research should be used as a learning tool for EPA to improve its analytic approaches to assessing technological feasibility. Finding 5 The committee did not find any specific guidance for assessing the un- certainties in the other factors that affects decision making, such as social factors (for example, environmental justice) and the political context. The committee also did not find examples of systematic consideration of those factors and their uncertainty when exploring the policy implications of strategies to mitigate harms to human health. In response to requirements in statutes or executive orders that require regulations to be based on the open exchange of information and the perspectives of stakeholders, some EPA programs (e.g., Superfund) work to address issues related to public (stakeholder) values and concerns.

OCR for page 217
SYNTHESIS AND RECOMMENDATIONS 223 Ecological risk assessments1 have included contingent valuation to help inform policy development. Economists have similarly explored the values people hold regarding specific health outcomes for purposes of resource allocation or clinical guideline development. More research is needed into methods to appropriately characterize the uncertainty in those other fac- tors, and to communicate that uncertainty to decision makers and the public. RECOMMENDATION 5 The U.S. Environmental Protection Agency should continue to work with stakeholders, particularly the general public, in efforts to iden- tify their values and concerns in order to determine which uncertain- ties in other factors, along with those in the health risk assessment, should be analyzed, factored into the decision-making process, and communicated. Finding 6 The nature of stakeholder participation and input to decision makers depends on the type of stakeholder. The regulated industry, local business communities, and environmental activists (including at the local level if they exist) are more likely to be proactively engaged in providing input on pend- ing regulations. The general public, without encouragement or assistance from EPA (or local environmental regulatory departments), is less likely to participate effectively or at all in such activities. One means to bridge the gap in understanding the values of the public is a formal research program. RECOMMENDATION 6 The U.S. Environmental Protection Agency should fund or conduct methodological research on ways to measure public values. This could allow decision makers to systematically assess and better explain the role that public sentiment and other factors that are difficult to quantify play in the decision-making process. 1  Ecological risk assessment is a “process that evaluates the likelihood that adverse ecologi- cal effects may occur or are occurring as a result of exposure to one or more stressors” (http:// www.epa.gov/raf/publications/pdfs/ECOTXTBX.PDF [accessed January 16, 2013]).

OCR for page 217
224 ENVIRONMENTAL DECISIONS IN THE FACE OF UNCERTAINTY Framework for Incorporating Uncertainty in Decision Making Finding 7 Uncertainty analysis must be designed on a case-by-case basis. The choice of uncertainty analysis depends on the context of the decision, including the nature or type of uncertainty (that is, heterogeneity and variability, model and parameter uncertainty, or deep uncertainty), and the factors that are considered in the decision (that is, health risk, tech- nological and economic factors, as well as other issues such as public sentiment and the political context), as well as the data that are available. Most environmental problems will require the use of multiple approaches to uncertainty analysis. For example, most environmental decisions will involve variability and heterogeneity as well as model and parameter un- certainty. As a result a mix of statistical analyses and expert judgments will be needed. A sensible, decision-driven, and resource-responsible approach to un- certainty analyses is needed. Iterative and deliberative problem formulation and planning using a systematic framework for the decision-making process will help ensure that the nature and extent of uncertainty analysis to be in- cluded in risk characterizations is appropriate to the decision context, and that decision makers are provided a view of uncertainty that is of maximum value to the decision under consideration. Involvement of decision makers and stakeholders in the planning and scoping of uncertainty analyses dur- ing the initial, problem formulation phase of will help ensure that the goals of the uncertainty analysis are consistent with the needs of the decision makers, and will help define analytic endpoints, and identify population subgroups and heterogeneity, and other uncertainties. The quantitative uncertainty analyses that are conducted do not always influence a decision and, therefore, do not always contribute to protection of public health. The committee does not intend to imply that complex uncertainty analyses have no role to play in supporting EPA decision mak- ing; rather the committee believes that such work should only undertaken when it is important and relevant to a given decision. Whether further quantitative uncertainty analysis is needed depends on the ability of these analyses to affect the environmental decision at hand. One way to gauge this is to inquire, whether perfect information would be able to change the decision, for example, whether knowing the exact dose–response function would change the regulatory regime. Clearly, if the environmental decision would stay the same for all states of information and analysis results, then it is not worth conducting the analysis.

OCR for page 217
SYNTHESIS AND RECOMMENDATIONS 225 RECOMMENDATION 7 Although some analysis and description of uncertainty is always im- portant, how many and what types of uncertainty analyses are carried out should depend on the specific decision problem at hand. The effort to analyze specific uncertainties through probabilistic risk assessment or quantitative uncertainty analysis should be guided by the ability of those analyses to affect the environmental decision. Communication Finding 8 A structured format for the public communication of the basis of EPA’s decisions would facilitate transparency and subsequent work with stake- holders, particularly community members. Consistent with findings and analyses in each rulemaking, EPA decision documents should make clear that the identified uncertainties are in line with reasonable expectations pre- sented in EPA guidelines and other sources. This practice would facilitate the goals of the first recommendation of the committee in this report—that EPA decision documents should make clear that uncertainty is inherent in agency risk assessments. The committee intends that the recommendations in this report support full discussion of the difficulties of decision making, including and possibly particularly when social factors (such as environ- mental justice and public values) and political context play a large role. RECOMMENDATION 8.1 U.S. Environmental Protection Agency senior managers should be transparent in communicating the basis of its decisions, including the extent to which uncertainty may have influenced decisions. RECOMMENDATION 8.2 U.S. Environmental Protection Agency decision documents and com- munications to the public should include a discussion of which uncer- tainties are and are not reducible in the near term. The implications of each to policy making should be provided in other communication documents when it might be useful for readers. Finding 9 Given that decision makers vary in their technical backgrounds and experience with highly mathematical depictions of uncertainty, a variety of communication tools should be developed. The ability of the public to assimilate the depictions of uncertainty and risk estimates is even more

OCR for page 217
226 ENVIRONMENTAL DECISIONS IN THE FACE OF UNCERTAINTY diverse. The public increasingly wants, and deserves, the opportunity to understand the decisions of appointed officials in order to manage their own risk, and to hold decision makers accountable. With respect to which uncertainties or aspects of uncertainties to communicate, attention should be paid to the relevance to the audience of the uncertainties, so that the uncertainty information is meaningful to the decision-making process and the audience(s). Those efforts should include different types of decisions and include communication of uncertainty to decision makers and to stake- holders and other interested parties. RECOMMENDATION 9.1 The U.S. Environmental Protection Agency (EPA), alone or in collabo- ration with other relevant agencies, should fund or conduct research on communication of uncertainties for different types of decisions and to different audiences, develop a compilation of best practices, and systematically evaluate their communications. RECOMMENDATION 9.2 As part of an initiative evaluating uncertainties in public sentiment and communication, U.S. Environmental Protection Agency senior manag- ers should assess agency expertise in the social and behavioral sciences (for example, communication, decision analysis, and economics), and ensure it is adequate to implement the recommendations in this report. In summary, the committee was impressed by the technical advances in uncertainty analysis used by EPA scientists in support of EPA’s human health risk assessments, which form the foundation of all EPA decisions. The committee believes that EPA can lead the development of uncertainty analyses in economics and technological assessment that are used for regu- latory purposes, as well as how to characterize and account for public senti- ment and political context. That leadership will require a targeted research program, as well as disciplined attention to how those uncertainties are described and communicated to a variety of audiences, including the role that uncertainties have played in a decision. REFERENCES GAO (Government Accountability Office). 2006. Human health risk assessment: EPA has taken steps to strengthen its process, but improvements needed in planning, data develop- ment, and training. Washington, DC: GAO. Goldstein, B. D. 2011. Risk assessment of environmental chemicals: If it ain’t broke…. Risk Analysis 31(9):1356–1362. NRC (National Research Council). 1983. Risk assessment in the federal government: Manag- ing the process. Washington, DC: National Academy Press.

OCR for page 217
SYNTHESIS AND RECOMMENDATIONS 227 ———. 1994. Science and judgment in risk assessment. Washington, DC: National Academy Press. ———. 1996. Understanding risk: Informing decisions in a democratic society. Washington, DC: National Academy Press. ———. 2009. Science and decisions: Advancing risk assessment. Washington, DC: The Na- tional Academies Press. Presidential/Congressional Commission on Risk Assessment and Risk Management. 1997. Risk assessment and risk management in regulatory decision-making. Final report. Vol- ume 1. Washington, DC.

OCR for page 217