Human health risk estimates are a cornerstone of the analyses that the U.S. Environmental Protection Agency (EPA) carries out to help it make its decisions. The uncertainties in those risk estimates have been the subject of much advice from the National Research Council (NRC) and others, many of which have recommended quantitative uncertainty analysis and discussed the use of default assumptions, probabilistic analysis, and the quantitative characterization of uncertainty (NRC, 1983, 1994, 1996, 2009; Presidential/Congressional Commission on Risk Assessment and Risk Management, 1997). EPA has made substantial advances in developing and using analytic techniques for assessing and narrowing the uncertainties in human health risk assessments, and EPA’s assessments have evolved and become more complex over the years (GAO, 2006; NRC, 2009). As a recent publication highlights, however, the quantitative or probabilistic uncertainty analyses that are now sometimes conducted can be more extensive than is needed by the decision maker, and might not contribute to EPA’s public health and environmental objectives (Goldstein, 2011). NRC reports have also emphasized the need to improve the utility of health risk assessments, and of the uncertainty analyses that are a part of the risk-assessment process (NRC, 1996, 2009).
Through its review of the risk assessment, uncertainty analysis, and decision making at EPA and other public health entities, the committee found that, in general, more emphasis should be placed on the uncertainty in the factors that affect EPA’s decisions besides estimates of human health. Uncertainties pervade not only the relationships between hazards and health outcomes, but also other important factors, such as the willingness to pay
for improvements in health, the pace of technological change, the importance of recreational areas, and the cost to the private sector of implementing new rules.
As part of a systematic approach to decision making, EPA should plan its analysis of the uncertainty analyses in estimates of health risks and those other factors around the needs of the decision makers beginning at the outset of the decision-making process; in other words, uncertainty analyses should not be an afterthought. Characterizations of health risk estimates, benefits, costs, technological availability, and other factors should reflect those uncertainty analyses. The implications of those different sources of uncertainty are presented in Box 7-1.
The appropriate uncertainty analysis depends, in part, on the type of uncertainty (variability and heterogeneity, model and parameter uncertainty, or deep uncertainty). Statistical analyses are often appropriate for assessing variability and heterogeneity, expert judgments and elicitations work well for model and parameter uncertainty, and robust decision-making approaches and scenario analysis will be needed for decision making in the presence of deep uncertainty.
Good communication among analysts, decision makers, and stakeholders is critical to ensuring a high-quality, comprehensive decision-making process and a high-quality, comprehensive decision. A process that includes such communication will help identify stakeholder concerns and potential uncertainties and to build social trust among the participants in the process. Each EPA decision is unique, and there is no universal best approach or tool for communicating uncertainties. The most appropriate strategy for communicating uncertainty will depend on the context of the decision, the purpose of the communication, the type of uncertainty, and the characteristics of the audience, including the level of technical expertise, personal and group biases, and the level of social trust.
Analyzing and communicating the uncertainty in the various factors that affect EPA’s decisions might require specialized expertise (for example, expertise in the analysis of benefits and cost and the uncertainties in each and also in communicating those uncertainties), and some of the necessary skills may be different from those found among EPA’s current personnel.
The committee’s specific findings and recommendations are presented below.
Uncertainty analyses in human health risk estimates can help decision makers to
• evaluate alternative regulatory options;
• assess how credible extreme risk estimates are and how much to rely on them in decision making;
• weigh the marginal decrease in risk against the effort made to reduce it;
• clarify issues within a decision by using scenarios to characterize very different worlds; and
• in the case of scenario analyses for deep uncertainty, identify regulatory solutions that are effective over a broad spectrum of scenarios.
Uncertainties About Technology Availability
Uncertainty analyses in technology availability can help decision makers to
• differentiate between well-established technologies with reasonably well-known costs, and those that have not been used for the purposes at hand; and
• consider which technology may be considered “best practicable” or “best available” by providing information about both the likelihood of success of the unproven technologies the time frame for success, and the effectiveness if successful.
Uncertainties About Cost–Benefit Analyses
Given the highly uncertain estimates of both health benefits and costs, uncertainty analyses in cost–benefit analyses can inform decision makers about
• how difficult it is to differentiate among different potential decisions;
• the disagreement among experts about the way regulation affects the economy, even when using similar models; and
• the ranges and sensitivity of estimates to different variables.
Uncertainties in the Characterization of Human Health Risks
Decision documents (such as documents that the technical experts at EPA prepare to explain site-specific decisions) often lack a robust discussion
of the uncertainties identified in the extensive health risk assessments prepared by agency scientists. Although decision documents and communications by the agency about its decisions should be succinct, open, and transparent, they should also include information on what uncertainties are present, which uncertainties need to be addressed, and how those uncertainties affected a decision. It should be clear from agency communications that uncertainty is inherent in science, including the science that informs EPA decisions.
In addition to contributing to full transparency, providing such information could mitigate attempts to use the existence of uncertainties as a rationale for delayed decision making. A shift in the expectations of those who read the decision documents and other public communications could eventually occur, so that the discussion of uncertainty, even unresolved uncertainty, could eventually come to be considered normal and valuable.
The decision maker will be able to use the analyses most responsibly if the known sources of uncertainties are acknowledged and described, even if the uncertainties are large, poorly described, or not currently resolvable. When uncertainties are so extensive and relevant so as to likely undermine the credibility and quality of a decision, it is the responsibility of staff analysts to advise decision makers as soon as possible in order to give the decision makers a chance to revisit the rulemaking schedule, objectives, and resources, which can require a judgment call.
To better inform the public and decision makers, U.S. Environmental Protection Agency (EPA) decision documents and other communications to the public should systematically
• include information on what uncertainties in the health risk assessment are present and which need to be addressed,
• discuss how the uncertainties affect the decision at hand, and
• include an explicit statement that uncertainty is inherent in science, including the science that informs EPA decisions.
Uncertainty in Other Factors That Influence a Decision
Although EPA decisions have included discussion and consideration of the uncertainties in the health risk assessment, the agency has generally given less attention to uncertainties in other contributors influencing the regulatory decision. Those contributors include economic, and
technological factors, as well as other factors that are not easily quantified, such as environmental justice. A major challenge to decision making in the face of uncertainty is the uncertainty in those other factors. Although every uncertainty does not need be analyzed for every decision, particularly when not important to a decision or when specific values are prescribed by other offices (for example, the value of a life that is set by the Office of Management and Budget), methods and processes should be available for when such analyses are appropriate and helpful to a decision maker. In general, this might require a research program to develop methods for this new type of uncertainty analysis, changes in decision documents and other analyses, and a program for research on communicating uncertainties.
The U.S. Environmental Protection Agency should develop methods to systematically describe and account for uncertainties in decision-relevant factors in addition to estimates of health risks—including technological and economic factors—into its decision-making process. When influential to a decision, those new methods should be subject to peer review.
EPA has developed guidance about, and conducted in-depth analyses of, the costs and benefits of major decisions. Some such analyses are conducted because of statutory mandate; others are conducted in response to a series of executive orders from the Office of Management and Budget mandating regulatory review. The analytic tools for cost–benefit analyses in health are well developed, and EPA guidance contains appropriate advice about the conduct of these analyses, including the discussion of some uncertainties. The committee, however, noted a lack of transparency regarding uncertainty analyses in the cost–benefit assessments in some EPA decision documents. While economists or modelers could evaluate the technical documents supporting the decision documents, and the economic models do include uncertainty analyses such as sensitivity analyses, the information about these uncertainties is often arcane, hard to locate, and technically very challenging to non-experts. Those analyses often shape regulatory decisions, and when they are conducted they should be described in ways that are useful and interpretable for the decision maker and stakeholders. The needs of the two audiences—that is, technical and non-expert audiences—differ, but a given set of decision documents and supporting analyses could include descriptions that explain the sources of uncertainties to the non-expert, and link, either electronically or via text, to more detailed descriptions of the economic analyses as appropriate for experts.
Analysts and decision makers should describe in decision documents and other public communications uncertainties in cost–benefit analyses that are conducted, even if not required by statute for decision making, the analyses should be described at levels that are appropriate for technical experts and non-experts.
The role of uncertainty in the costs and benefits and availability and feasibility of control technologies is not well investigated or understood. The evidence base for those factors is not robust. Evaluating case studies of past rulemaking and developing a directed research program on assessing the availability of technology might be first steps toward understanding the robustness of technology feasibility and economic assessments, and the potential for technology innovation.
The U.S. Environmental Protection Agency (EPA) should fund research, conduct research or both, to evaluate the accuracy and predictive capabilities of past assessments of technologies and costs and benefits for rulemaking in order to improve future efforts. This research could be conducted by EPA staff or else by nongovernmental policy analysts who might be less subject to biases. This research should be used as a learning tool for EPA to improve its analytic approaches to assessing technological feasibility.
The committee did not find any specific guidance for assessing the uncertainties in the other factors that affects decision making, such as social factors (for example, environmental justice) and the political context. The committee also did not find examples of systematic consideration of those factors and their uncertainty when exploring the policy implications of strategies to mitigate harms to human health. In response to requirements in statutes or executive orders that require regulations to be based on the open exchange of information and the perspectives of stakeholders, some EPA programs (e.g., Superfund) work to address issues related to public (stakeholder) values and concerns.
Ecological risk assessments1 have included contingent valuation to help inform policy development. Economists have similarly explored the values people hold regarding specific health outcomes for purposes of resource allocation or clinical guideline development. More research is needed into methods to appropriately characterize the uncertainty in those other factors, and to communicate that uncertainty to decision makers and the public.
The U.S. Environmental Protection Agency should continue to work with stakeholders, particularly the general public, in efforts to identify their values and concerns in order to determine which uncertainties in other factors, along with those in the health risk assessment, should be analyzed, factored into the decision-making process, and communicated.
The nature of stakeholder participation and input to decision makers depends on the type of stakeholder. The regulated industry, local business communities, and environmental activists (including at the local level if they exist) are more likely to be proactively engaged in providing input on pending regulations. The general public, without encouragement or assistance from EPA (or local environmental regulatory departments), is less likely to participate effectively or at all in such activities. One means to bridge the gap in understanding the values of the public is a formal research program.
The U.S. Environmental Protection Agency should fund or conduct methodological research on ways to measure public values. This could allow decision makers to systematically assess and better explain the role that public sentiment and other factors that are difficult to quantify play in the decision-making process.
1 Ecological risk assessment is a “process that evaluates the likelihood that adverse ecological effects may occur or are occurring as a result of exposure to one or more stressors” (http://www.epa.gov/raf/publications/pdfs/ECOTXTBX.PDF [accessed January 16, 2013]).
Framework for Incorporating Uncertainty in Decision Making
Uncertainty analysis must be designed on a case-by-case basis. The choice of uncertainty analysis depends on the context of the decision, including the nature or type of uncertainty (that is, heterogeneity and variability, model and parameter uncertainty, or deep uncertainty), and the factors that are considered in the decision (that is, health risk, technological and economic factors, as well as other issues such as public sentiment and the political context), as well as the data that are available. Most environmental problems will require the use of multiple approaches to uncertainty analysis. For example, most environmental decisions will involve variability and heterogeneity as well as model and parameter uncertainty. As a result a mix of statistical analyses and expert judgments will be needed.
A sensible, decision-driven, and resource-responsible approach to uncertainty analyses is needed. Iterative and deliberative problem formulation and planning using a systematic framework for the decision-making process will help ensure that the nature and extent of uncertainty analysis to be included in risk characterizations is appropriate to the decision context, and that decision makers are provided a view of uncertainty that is of maximum value to the decision under consideration. Involvement of decision makers and stakeholders in the planning and scoping of uncertainty analyses during the initial, problem formulation phase of will help ensure that the goals of the uncertainty analysis are consistent with the needs of the decision makers, and will help define analytic endpoints, and identify population subgroups and heterogeneity, and other uncertainties.
The quantitative uncertainty analyses that are conducted do not always influence a decision and, therefore, do not always contribute to protection of public health. The committee does not intend to imply that complex uncertainty analyses have no role to play in supporting EPA decision making; rather the committee believes that such work should only undertaken when it is important and relevant to a given decision. Whether further quantitative uncertainty analysis is needed depends on the ability of these analyses to affect the environmental decision at hand. One way to gauge this is to inquire, whether perfect information would be able to change the decision, for example, whether knowing the exact dose–response function would change the regulatory regime. Clearly, if the environmental decision would stay the same for all states of information and analysis results, then it is not worth conducting the analysis.
Although some analysis and description of uncertainty is always important, how many and what types of uncertainty analyses are carried out should depend on the specific decision problem at hand. The effort to analyze specific uncertainties through probabilistic risk assessment or quantitative uncertainty analysis should be guided by the ability of those analyses to affect the environmental decision.
A structured format for the public communication of the basis of EPA’s decisions would facilitate transparency and subsequent work with stakeholders, particularly community members. Consistent with findings and analyses in each rulemaking, EPA decision documents should make clear that the identified uncertainties are in line with reasonable expectations presented in EPA guidelines and other sources. This practice would facilitate the goals of the first recommendation of the committee in this report—that EPA decision documents should make clear that uncertainty is inherent in agency risk assessments. The committee intends that the recommendations in this report support full discussion of the difficulties of decision making, including and possibly particularly when social factors (such as environmental justice and public values) and political context play a large role.
U.S. Environmental Protection Agency senior managers should be transparent in communicating the basis of its decisions, including the extent to which uncertainty may have influenced decisions.
U.S. Environmental Protection Agency decision documents and communications to the public should include a discussion of which uncertainties are and are not reducible in the near term. The implications of each to policy making should be provided in other communication documents when it might be useful for readers.
Given that decision makers vary in their technical backgrounds and experience with highly mathematical depictions of uncertainty, a variety of communication tools should be developed. The ability of the public to assimilate the depictions of uncertainty and risk estimates is even more
diverse. The public increasingly wants, and deserves, the opportunity to understand the decisions of appointed officials in order to manage their own risk, and to hold decision makers accountable. With respect to which uncertainties or aspects of uncertainties to communicate, attention should be paid to the relevance to the audience of the uncertainties, so that the uncertainty information is meaningful to the decision-making process and the audience(s). Those efforts should include different types of decisions and include communication of uncertainty to decision makers and to stakeholders and other interested parties.
The U.S. Environmental Protection Agency (EPA), alone or in collaboration with other relevant agencies, should fund or conduct research on communication of uncertainties for different types of decisions and to different audiences, develop a compilation of best practices, and systematically evaluate their communications.
As part of an initiative evaluating uncertainties in public sentiment and communication, U.S. Environmental Protection Agency senior managers should assess agency expertise in the social and behavioral sciences (for example, communication, decision analysis, and economics), and ensure it is adequate to implement the recommendations in this report.
In summary, the committee was impressed by the technical advances in uncertainty analysis used by EPA scientists in support of EPA’s human health risk assessments, which form the foundation of all EPA decisions. The committee believes that EPA can lead the development of uncertainty analyses in economics and technological assessment that are used for regulatory purposes, as well as how to characterize and account for public sentiment and political context. That leadership will require a targeted research program, as well as disciplined attention to how those uncertainties are described and communicated to a variety of audiences, including the role that uncertainties have played in a decision.
GAO (Government Accountability Office). 2006. Human health risk assessment: EPA has taken steps to strengthen its process, but improvements needed in planning, data development, and training. Washington, DC: GAO.
Goldstein, B. D. 2011. Risk assessment of environmental chemicals: If it ain’t broke…. Risk Analysis 31(9):1356–1362.
NRC (National Research Council). 1983. Risk assessment in the federal government: Managing the process. Washington, DC: National Academy Press.
———. 1994. Science and judgment in risk assessment. Washington, DC: National Academy Press.
———. 1996. Understanding risk: Informing decisions in a democratic society. Washington, DC: National Academy Press.
———. 2009. Science and decisions: Advancing risk assessment. Washington, DC: The National Academies Press.
Presidential/Congressional Commission on Risk Assessment and Risk Management. 1997. Risk assessment and risk management in regulatory decision-making. Final report. Volume 1. Washington, DC.