5
Uncertainty

There are several major barriers to broad acceptance of recent EPA health benefits analyses. One barrier is the large amount of uncertainty inherent in these analyses, and another is the manner in which the agency deals with this uncertainty. A third barrier is that projected health benefits are often reported as absolute numbers of avoided death or adverse health outcomes without a context of population size or total numbers of outcomes. Areas of uncertainty include air-quality modeling, population demographics and heterogeneity, intersubject variability, health and exposure baselines, compliance with control measures, effectiveness of controls in reducing pollutant emissions, validity and precision of concentration-response functions and use of alternative models (linear, nonlinear), estimation of these functions as relative effects (relative risks) or absolute effects (risk differences), relative toxicity of mixture components, and applicability of these functions to target populations of regulatory concern. These uncertainties are rooted in incomplete scientific knowledge. When benefits are estimated for future target populations, the cumulative magnitude of the uncertainties can be formidable. Many of them can be reduced by further research, but on the whole, they are likely to remain high.

Even great uncertainty does not imply that action to promote or protect public health should be delayed. Decisions about whether to act, when to act, and how aggressively to act can only be made with some understanding of the likelihood and consequences of alternative courses of action. The potential for improving decisions through research must be balanced against



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations 5 Uncertainty There are several major barriers to broad acceptance of recent EPA health benefits analyses. One barrier is the large amount of uncertainty inherent in these analyses, and another is the manner in which the agency deals with this uncertainty. A third barrier is that projected health benefits are often reported as absolute numbers of avoided death or adverse health outcomes without a context of population size or total numbers of outcomes. Areas of uncertainty include air-quality modeling, population demographics and heterogeneity, intersubject variability, health and exposure baselines, compliance with control measures, effectiveness of controls in reducing pollutant emissions, validity and precision of concentration-response functions and use of alternative models (linear, nonlinear), estimation of these functions as relative effects (relative risks) or absolute effects (risk differences), relative toxicity of mixture components, and applicability of these functions to target populations of regulatory concern. These uncertainties are rooted in incomplete scientific knowledge. When benefits are estimated for future target populations, the cumulative magnitude of the uncertainties can be formidable. Many of them can be reduced by further research, but on the whole, they are likely to remain high. Even great uncertainty does not imply that action to promote or protect public health should be delayed. Decisions about whether to act, when to act, and how aggressively to act can only be made with some understanding of the likelihood and consequences of alternative courses of action. The potential for improving decisions through research must be balanced against

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations the public health costs incurred because of a delay in the implementation of controls. Complete certainty is an unattainable ideal. Health benefits analyses compare alternative scenarios that would result with and without regulatory action. As a consequence, these analyses are inherently speculative and their results unverifiable. Because only one regulatory option can be chosen by decision-makers, the outcomes of the remaining regulatory options, including the baseline with no action (if not chosen), can never be directly observed. Analyses of health benefits should represent the uncertainties in the choices facing decision-makers and society at large (Hattis and Anderson 1999). Analyses should attempt to provide insight into the variability of impacts (among persons, places, and other dimensions of interest) and the extent and sources of uncertainties in the results. The representation of uncertainty requires a good faith appraisal of the imperfection in the state of information about these impacts (Hattis and Burmaster 1994). Uncertainty assessment should not overrepresent or underrepresent the quality and completeness of available information. This chapter discusses EPA’s current approach to assessing uncertainty in health benefits analyses for air pollution control regulations. The agency’s analysis of the health benefits for the final Tier 2 vehicle emissions standards and gasoline sulfur control rule-making (EPA 1999a) is used for illustration. The chapter outlines a revised approach that would reflect overall uncertainty more realistically, in part by using probabilistic expressions of expert judgment. The chapter also briefly reviews the history of probabilistic uncertainty assessment in EPA health benefits analyses under the Clean Air Act. This chapter is confined to uncertainty in the analysis of health benefits expressed solely in terms of health. Although uncertainties in the monetary valuation of health benefits and in the analysis of regulatory costs are not considered, the committee notes that there are great uncertainties in those analyses as well. EPA’S APPROACH TO UNCERTAINTY ANALYSIS EPA uses a two-part approach to assessing uncertainty in health benefits analyses that rely on epidemiological studies as the source of estimated concentration-response functions, although different approaches are some

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations times used, especially when epidemiological evidence is lacking (EPA 1997). The first part is a primary analysis, which produces numerical estimates or projections of each health benefit in the form of a probability distribution. This analysis incorporates only one source of uncertainty: the random sampling error in the epidemiological study or studies that provide the estimated concentration-response function. The second part of the uncertainty assessment is an array of ancillary analyses in which many other sources of uncertainty are considered in several disparate ways. Primary Uncertainty Analysis The primary uncertainty analysis produces a numerical estimate of each health benefit EPA believes to be plausible for a particular regulatory action. Typically, the benefit is expressed as a number of deaths or cases of an adverse health event that will be avoided in the United States in a future year if some regulatory action is taken. The year chosen is often far into the future to allow for the action to be implemented, for the implementation to result in exposure reductions, and for the reduced exposures to result in health benefits. In the Tier 2 analysis, the chosen year was 2030. EPA reports each numerical health benefit estimate in the form of a probability distribution and summarizes the distribution by reporting its mean and 5th and 95th percentiles. The distribution assigns a nonzero probability to every possible value including the null hypothesis of no benefit. The mean of the distribution is interpreted as the expected benefit based upon the analysis performed. The 5th and 95th percentiles are defined as a credible range within which the true benefit value will lie with a 90% probability (EPA 1999a, p. 3-26). The solid line in Figure 5-1 shows the probability distribution from EPA’s primary analysis of avoided mortality for the proposed Tier 2 rule for the year 2030. The mean of the distribution (which is also the median and the 50th percentile) is 4,307 avoided deaths among persons 30 years of age and older. The 5th and 95th percentiles are 2,671 and 5,891 avoided deaths, respectively (EPA 1999a, p. 6-3). The probability models in EPA’s primary analyses incorporate only one of the many sources of uncertainty in these analyses: the random sampling error in the estimated concentration-response function derived from either an epidemiological study or a meta-analytic or pooled aggregation of two or

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations FIGURE 5-1 Probability distributions from primary and alternative analyses of avoided mortality for Tier 2 analysis. Source: Data from EPA 1999a. more such studies. In a meta or pooled analysis of separate studies, a summary estimate of the concentration-response function is produced by averaging study estimates that may include ones that vary in strength and ones that suggest little or no effect. To estimate avoided mortality for the Tier 2 rule, the agency chose an estimated concentration-response function from a log-linear (Poisson regression) analysis of results from a study by the American Cancer Society (Pope et al. 1995). For a change in concentration from 9 to 33.5 • g/m3, the result was an estimated relative risk of 1.17 with a 95% confidence interval of 1.09 to 1.26 (EPA 1999a, p. C-2). The random sampling error represented by this confidence interval is the only source of uncertainty in the agency’s probability distribution for avoided mortality. The incorporation of additional sources of uncertainty would widen the distribution. EPA correctly notes that incorporating only the uncertainty from random sampling error in concentration-response function estimates into its primary health benefits analyses “omits important sources of uncertainty, such as the contribution of air quality changes, baseline population incidences, projected populations exposed, transferability of the concentration-

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations response function to diverse locations, and uncertainty about premature mortality” and “would provide a misleading picture about the overall uncertainty in the estimates” (EPA 1999a, p. 3-26). Ancillary Uncertainty Analyses EPA assesses all other uncertainties in a second part of each health benefits analysis. The agency begins with a list of as many key uncertainties as it can identify. The list compiled for the Tier 2 analysis is given in Table 5-1. Much of this uncertainty results from unavoidable and expected variability or heterogeneity in concentration-response functions estimated by epidemiological studies. Some of it results from baseline statistical variation, as no study has infinite sample size and all study populations differ in their distributions of background causes of health outcomes and in their distributions of susceptibility to toxic agents. Projection of future baselines, such as the death rate to be expected 30 or more years in the future if no action is taken, are particularly uncertain. Important uncertainty is also produced by variation in study design, data collection, and statistical analysis. Although there may be other uncertainties that have not been identified, EPA typically makes no allowance for these unidentified sources of uncertainty. EPA takes a variety of approaches regarding these identified uncertainties. Some are merely mentioned. Other uncertainties are discussed qualitatively with regard to the direction and, sometimes, the magnitude of the impact that they are likely to exert on the mean value of the probability distribution. For example, in the discussion of the epidemiological study providing the estimated concentration-response function for avoided mortality in the Tier 2 analysis, EPA referred to downward biases from the relatively healthy study population and from intercity migration of study participants, which the agency believed would counteract an upward bias associated with historical air-quality trends (EPA 1999a, p. C-1). For selected sources of uncertainty, EPA conducted supplemental calculations, alternative calculations, and sensitivity analyses (EPA 1999a, p. 3-19). These terms have specific meanings in EPA health benefits assessments. Supplemental calculations “provide additional information about specific health effects, but are not suitable for inclusion in the primary or

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations TABLE 5-1 Key Sources of Uncertainty in the Tier 2 Benefits Analysis 1. Uncertainties Associated with Concentration-Response (C-R) Functions – The value of the ozone- or particulate matter (PM)-coefficient in each C-R function. – Application of a single C-R function to pollutant changes and populations in all locations. – Similarity of future year C-R relationships to current C-R relationships. – Correct functional form of each C-R relationship. – Extrapolation of C-R relationships beyond the range of ozone or PM concentrations observed in the study. 2. Uncertainties Associated with Ozone and PM Concentrations – Estimating future-year baseline and hourly ozone and daily PM concentrations. – Estimating the change in ozone and PM resulting from the control policy. 3. Uncertainties Associated with PM Mortality Risk – No scientific literature supports a direct biological mechanism for observed epidemiological evidence. – Direct causal agents within the complex mixture of PM responsible for reported health effects have not been identified. – The extent to which adverse health effects are associated with low level exposures that occur many times in the year versus peak exposures. – Possible confounding in the epidemiological studies of PM2.5 effects with other factors (such as other air pollutants, weather, and indoor and outdoor air). – The extent to which effects reported in the long-term studies are associated with historically higher concentrations of PM rather than the concentrations occurring during the period of study. – Reliability of the limited ambient PM2.5 monitoring data in reflecting actual PM2.5 exposures. 4. Uncertainties Associated with Possible Lagged Effects – What portion of the PM-related long-term exposure mortality associated with changes in annual PM levels would occur in a single year, and what portion might occur in subsequent years. 5. Uncertainties Associated with Baseline Incidence Rates – Some baseline incidence rates are not location-specific (such as those taken from studies) and might not accurately represent the location-specific rates of interest. – Current baseline incidence rates might not approximate baseline incidence rates in the year 2030. – Projected population and current demographics—used to derive incidences—might not approximate future-year populations and demographics.

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations 6. Uncertainties Associated with Aggregation of Monetized Benefits – Health and welfare benefit estimates are limited to the available C-R functions. Thus, unquantified benefit categories will cause total benefits to be underestimated.   Source: Adapted from EPA 1999a, Exhibit 3-3, p. 3-20. alternative estimates due to concerns about double-counting of benefits or the high degree of uncertainty about the estimates” (EPA 1999a, p. 3-21). The supplemental analyses in the Tier 2 report pertained to short-term mortality, infant (postneonatal) mortality, ozone mortality, asthma attacks, restricted-activity days, and ozone-related cardiovascular disease (EPA 1999a, pp. 3-23, 3-24, A-1). In other contexts, both EPA’s alternative calculations and sensitivity analyses would be called sensitivity analyses (Morgan et al. 1990; Greenland 1998). The distinction for EPA lies in its judgment of their plausibility. Alternative calculations “are based on relatively plausible alternatives to the assumptions used in deriving the primary benefit estimates” (EPA 1999a, p. 3-21). Sensitivity analyses “examine the sensitivity of estimated benefits results to less plausible alternatives to the assumptions used in the primary analyses” (EPA 1999a, p. 3-25). For both calculations and analyses, assumptions or sources of uncertain quantities are varied and the mean of the health benefit probability distribution is recomputed. In all cases, the alternative calculations and sensitivity analyses are conducted for only one source of uncertainty at a time. In addition, they are conducted only to determine the sensitivity of the mean of the probability distribution from the primary analysis to modified assumptions and information sources. With one exception, the spread of the health benefit probability distribution, as gauged by the distance of the interval between its 5th and 95th percentiles, is not affected. EPA’s rationale for focusing only on the mean is that an “attempt to assign probabilities to these alternative calculations…would only add to the uncertainty of the analysis or present a false picture about the precision of the results” (EPA 1999a, p. 3-21). EPA does not discuss why adding to the uncertainty of the analysis would be inappropriate. Noting that some analyses of health benefits of air pollution reductions (Lang et al. 1995; Holland et al. 1999) have included the assignment of “probabilities to ranges of parameter values for different endpoints,”

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations EPA argued that “the estimated points on these distributions are themselves highly uncertain and very sensitive to the subjective judgements of the analyst. To avoid these subjective judgements, we choose to allow the reader to determine the weights they would assign to alternative estimates” (EPA 1999a, p. 3-21). For the Tier 2 analysis, alternative calculations were performed for an alternative source of the estimated concentration-response function and for life-years saved rather than avoided deaths as a measure of health benefit (EPA 1999a, pp. 3-21, 23). Sensitivity analyses were conducted for thresholds and alternative lag structures (EPA 1999a, p. 3-25). The one exception to the exclusive focus on the mean of the health benefit probability distribution occurs when an alternative calculation involves the use of a different study to provide the estimated concentration-response function, which has its own standard error estimate. The broken line in Figure 5-1 shows the probability distribution when the concentration-response function from an analysis of the Harvard six cities study (Dockery et al. 1993) is used. This study produced a higher point estimate of the relative risk, so the mean of the probability distribution is higher (10,000 avoided deaths). The alternative study was smaller, however, so its estimate had more random sampling error and the distribution is wider. The 5th and 95th percentiles are 5,000 and 15,000 avoided deaths, respectively. CRITIQUE OF EPA’S CURRENT UNCERTAINTY ASSESSMENTS Numerical projections appear to be essential in health benefits analyses, and probability distributions can be used to describe the uncertainty in these analyses. Issues arise, however, over which sources of uncertainty the distributions should incorporate, how to incorporate them, and how to present the results. EPA’s decision to incorporate only one source of uncertainty, the random sampling error in the estimated concentration-response function, into the probability distributions resulting from its health benefits analyses is worth reconsidering. The committee agrees with the agency’s judgment that its current practice produces health benefits probability distributions that give “a misleading picture about the overall uncertainty in the estimates” (EPA 1999a, p. 3-26). In particular, the distributions suggest that there is less uncertainty, perhaps much less, than is actually present.

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations The committee finds that the mean of the distributions should not be interpreted as “best” estimates, and the intervals between the 5th and 95th percentiles of the distributions should not be interpreted as “90 percent credible intervals,” within which “the true benefit lies with 90 percent probability” (EPA 1999a, p. 3-26). The committee agrees with EPA’s statement that it would require expert judgment to specify probability distributions for many of the uncertain components of the health benefits analyses. In these cases, probability would be used not only in its connotation of variability1 but also in its connotation of subjective uncertainty or lack of complete belief as well (Hacking 1984; Poole 1988; Lindley 2000). EPA is correct that the elicitation of expert opinions in the form of probability distributions is a difficult and uncertain process (Morgan et al. 1990; Cooke 1991; Pate-Cornell 1996). The committee does not agree, however, that these difficulties are sufficient reasons for not trying to obtain such advice. Nor does the committee find any reason to avoid the attempt on the ground that it “would only add to the uncertainty of the analysis or present a false picture about the precision of the results” (EPA 1999a, p. 3-21). On the contrary, by growing wider, the health benefits probability distributions would more accurately depict the uncertainty and lack of precision in the analyses. As difficult and uncertain as these specifications are, they are preferable to EPA’s current practice of treating important and highly uncertain model components as though they were certain. The probability models from which standard errors are estimated for concentration-response-function estimates from observational epidemiological studies are less than certain as well. These models would have a firm theoretical foundation only if study populations were randomly sampled from target populations and exposure concentrations were randomly allocated to study participants (Greenland 1990; Poole 2001). In observational studies such as the American Cancer Society study (Pope et al. 1995) and the 1   In risk analysis, a distinction is often made between characterization of variability (the true variation in a parameter over time, space, or persons) and uncertainty (ignorance about the true value of the parameter). Variability is characterized primarily to provide information about the true distribution of exposure and risk and to suggest opportunities for control or to provide a sense of equity. Uncertainty is characterized primarily to give a sense of the confidence that can be placed in study results and to help in setting priorities.

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations Harvard six cities study (Dockery et al. 1993), neither random sampling nor random exposure allocation was used. Nevertheless, analysts use probability models for these design features in analyzing observational data because larger observational studies have less variability than smaller ones and account for the incomplete enumeration of the population of interest. The applicability of probability models for random variability to observational data is less than perfectly secure, but their use is preferable to assuming that there is no variability related to study size in observational results. The use of probability models for uncertainties involving expert judgment is also preferable to assuming that these uncertainties do not exist. Many of the key uncertainties in these analyses may be characterized only subjectively by reference to expert judgment. The question is not whether to rely on expert judgment but how best to elicit and summarize the views of experts and how to incorporate them into the analysis. Probability distributions are a legitimate and useful way to express the uncertainties in expert judgments. Incorporation of those uncertainties as probability distributions into the primary analysis would likely change the expected value and widen the resulting probability distribution for each health benefit. The result will include more of the uncertainty in the health benefits assessment. The alternative calculations and sensitivity analyses conducted by EPA help to describe the uncertainty in the analyses, but they are not sufficient. The major problems with them are that EPA consigns them to an ancillary status and not to the primary analysis, that the various sources of uncertainty are considered one at a time, and that EPA explicitly offers no judgment as to the relative plausibility of the alternative scenarios considered in these analyses. Without a combined, simultaneous assessment of multiple uncertainty sources, it is impossible to gain an appreciation of the overall magnitude of the uncertainty in the analysis. The committee does not agree with the agency’s decision to have the reader determine the plausibility and relative weighting of alternative assumptions and data sources and integrate these assessments across uncertainty sources. In its current analyses, EPA does not systematically or probabilistically address the extension of results beyond a study population’s age range. The typical assumption is that the health-outcome-rate ratio is constant across age; however, this assumption is seldom tested and seldom has any strong etiological justification, even when compared with a simple alternative, such as a constant-rate difference. For example, the method of extrapolating to additional age groups can be of crucial importance if the study

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations population excludes elderly persons who are at especially high baseline risk and the target population includes a sizable proportion of elderly persons. A large portion of the overall health benefit may then be projected for an age range that has not been studied. In such cases, the mixture of model and data would be tilted heavily toward the model. Two additional illustrative examples are thresholds for adverse effects and lag structures.2 EPA considers implausible any threshold for mortality in the particulate matter (PM) exposure ranges under consideration (EPA 1999a, p. 3-8). Although the agency conducts sensitivity analyses incorporating thresholds, it provides no judgment as to their relative plausibility. In a probabilistic uncertainty analysis, EPA could assign appropriate weights to various threshold models. For PM-related mortality in the Tier 2 analysis, the committee expects that this approach would have resulted in only a slight widening of the probability distribution for avoided mortality and a slight reduction in the mean of that distribution, thus reflecting EPA’s views about the implausibility of thresholds. The committee finds that such formal incorporation of EPA’s expert judgments about the plausibility of thresholds into its primary analysis would have been an improvement. Uncertainty about thresholds is a special aspect of uncertainty about the shape of concentration-response functions. Typically, EPA and authors of epidemiological studies assume that these functions are linear on some scale. Often, the scale is a logarithmic transformation of the risk or rate of the health outcome, but when a rate or risk is low, a linear function on the logarithmic scale is approximately linear on the scale of the rate or risk itself. Increasingly, epidemiological investigators are employing analytic methods that permit the estimation of nonlinear shapes for concentration-response functions (Greenland et al. 1999). As a consequence, EPA will need to be prepared to incorporate nonlinear concentration-response functions from epidemiological studies into the agency’s health benefits analyses. Any source of error or bias that can distort an epidemiological association can also distort the shape of an estimated concentration -response function, as can variation in individual susceptibility (Hattis and Burmaster 1994; Hattis et al. 2001). EPA expressed much less certainty about alternative lag structures than it did about thresholds in the Tier 2 analysis. The lag structure used in the 2   A lag reflects the time course between pollutant exposure and development of clinical disease. A lag structure reflects the variation among the population in the lags experienced by various individuals.

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations FIGURE 5-2 Posterior probability distributions for estimated short-term mortality effects of PM10 with and without adjustment for copollutants. The x-axis is the percent change in daily mortality ([relative risk •1] × 100%) for a 10 •g/m3 increase in PM10 concentration. Thus, 0.2% is a relative risk of 1.002. Source: Adapted from NMMAPS 2002. recommended the use of expert judgment to characterize epistemic uncertainty, while others recommended that such basic scientific uncertainties should be thoroughly described but not quantified. The committee shares the view that proper characterization of uncertainty is essential to good decision-making and agrees that uncertainties are often underestimated, leaving decision-makers with a false sense of security. Having reached this conclusion, the committee shares the view of M. Granger Morgan (Morgan et al. 1990): When the value of an uncertain quantity is needed in policy analysis, and limits in data or understanding preclude the use of conventional statistical techniques to produce probabilistic estimates, about the only remaining option is to ask experts for their best professional judgment.

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations In the committee’s view then, the question is not whether to use expert judgment but how to use expert judgment. The options are to pick a particular model component from a range of varyingly plausible alternatives and treat that one as though it were absolutely certain or to specify a judgment-based probability model for the alternatives that reflects their varying degrees of plausibility and incorporate those probabilities into the primary analysis. The latter option has many difficulties, but it has the potential to portray the existing uncertainty more realistically than the former option does. The committee recommends that EPA begin to incorporate additional sources of uncertainty into the probability models it uses in its primary health benefits analyses. Furthermore, the committee recognizes that decision-makers will need to be informed about how and why uncertainty was added to the health benefits analysis and how, in turn, this uncertainty might be communicated to the public. This process will use probability distributions to replace model components that are treated as known fixed values. Of necessity, the probability distributions for the uncertain model components will have to reflect a combination of empirical observations and expert judgment. This will result in a more realistic picture of the overall uncertainty in the analyses. The mean of the health benefit distribution will reflect the expected magnitude of the health benefit more accurately and, as a consequence, will be more defensible. The mean might shift upward, downward, or not at all as each additional source of uncertainty is added to the core analysis. The effect on the spread of the distribution, as reflected by the interval between its 5th and 95th percentiles for example, will be a predictable widening. There is a large and growing body of literature on the use of expert judgment in risk assessment and quantitative policy analysis (Morgan et al. 1990; Cooke 1991). It has been applied in fields such as climate change (Reilly et al. 2001). There are several applications in health risk assessment (North and Merkhofer 1976; Morgan et al. 1984; Evans et al.1994). In fact, as described above, OAQPS has been a pioneer in the application of these approaches to estimating the health risks due to exposure to air pollutants (Richmond 1981; Feagans and Biller 1981; Whitfield et al. 1991; Rosenbaum et al. 1995). These approaches have also been used in cases of residential radon cancer risks (Krewski et al. 1999) and for stratospheric ozone depletion (NRC 1979a,b). As it incorporates additional sources of uncertainty into its primary

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations health benefits analyses, EPA should consider conducting analyses to determine which uncertainty sources have the greatest influence on the final results. Those impacts should be measured not only on the mean but also on the spread of the health benefit probability distribution. The sources that have the greatest influence on the spread of the distribution of a health benefit should be given priority for future research. Value-of-information analysis, a branch of statistical decision analysis, provides a well-structured approach for estimating the decision-making benefits of information that might be expected to flow from various research strategies (Raiffa 1970; Lindley 1985). EPA should also consider conducting analyses to determine the sensitivity of the final results to the specification of reasonable alternative probability distributions for the uncertainty sources in the primary analyses. The need for sensitivity analyses will be particularly great for distributions that are based solely or largely on expert judgment. EPA should consider comparing predictions from health benefits analysis models with subsequent observations that were not used in deriving or calibrating the models. Ideally, the subsequent observations and comparisons should be made by researchers who are independent from the authors of the original model and the investigators whose observations were used to derive and calibrate it. The results of the comparisons should be presented in future health benefits analyses and used to assess, quantify, and reduce uncertainties in the resulting estimates. As it begins in the transition to incorporate additional sources of uncertainty into its primary health benefits analyses, EPA should continue the sensitivity analyses it has traditionally conducted. These analyses should be expanded, however, to include more than one source of uncertainty at a time. For example, if EPA were to include three additional uncertainty sources into its primary analysis of a health benefit, it might also conduct a traditional sensitivity analysis of these three sources jointly. With three illustrative scenarios for each component, for example, this expansion of the traditional sensitivity analysis would produce mean health benefits estimates for all 27 possible combinations of the scenarios. EPA then would be able to refer to the probability assigned to these combinations in the primary analysis to reflect their varying degrees of plausibility. EPA should consider distinguishing between the uncertainties that arise from difficulties in projecting the future and the uncertainties inherent in estimating health benefits in current populations on the basis of hypothetical

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations changes in current levels of emissions. By providing a preliminary analysis that estimates in current populations the health benefits resulting from hypothetical changes in current levels of emissions, EPA might develop an idea of the lower bound on the uncertainty in any prediction of consequences projected into the distant future. There would be fewer uncertainties in these preliminary analyses than in analyses of the impacts of proposed regulatory actions on future exposures and health outcomes. EPA should continue to strive to present the results of its health benefits analyses in ways that avoid conveying an unwarranted degree of certainty. These alternative approaches should include rounding to fewer significant digits. For example, the mean of the Tier 2 distribution for avoided mortality could have been reported as 4,300 or 4,000 avoided deaths rather than 4,307. Another need is to place less emphasis on a single value, such as the mean of a health benefit probability distribution, and more on ranges, such as the interval between the distribution’s 5th and 95th percentiles. It would also be helpful to increase the use of graphs to display health benefits probability distributions in their entirety. Graphs will be especially helpful as the incorporation of additional uncertainties results in asymmetrical health benefit probability distributions (Read and Morgan 1998). In presenting a probability distribution for each health benefit produced by a primary analysis, EPA should emphasize even more than it has in the past the sources of uncertainty that remain unaccounted for in the primary analysis. Along with depicting the uncertainty in its primary health benefits analyses more realistically, EPA should foster a discussion in which it rebuts explicitly the misperception that such analyses would not be “useful.” That view comes from a mistaken belief that a very high degree of certainty is required before regulatory action can be considered warranted to protect the environment and the public health. The result is needless pressure to make the scientific basis for that regulatory action appear more certain than it is. A more defensible position is that decision-makers can make much better decisions when provided with realistic assessments of the nature and extent of the uncertainty that is present. The correct mix of action and research is a policy decision that can be informed by a full appraisal of the sources, nature, and extent of uncertainty. The committee recommends that formally elicited expert judgment be used in the characterization of uncertainty in estimates of health benefits, although the committee recognizes that a number of issues must be addressed to use this approach responsibly. However, the committee be-

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations lieves approaches that explicitly incorporate judgmental probabilities into EPA estimates of health benefits are preferable to ones that fail to characterize the degree and key sources of uncertainty in estimates of health benefits from regulatory action. Furthermore, the committee recommends that EPA formally acknowledge those experts from whom it elicited judgments on uncertainty issues in the health benefits analysis. CONCLUSIONS In its primary analyses of health benefits, EPA reports the uncertainty as a probability distribution. Only one source of uncertainty, the random sampling variability of the estimated concentration-response function, is given with an emphasis on the mean of the probability distribution. The absence of other sources of uncertainty makes the results of the primary analyses appear more certain than they are. To address other sources of uncertainty, EPA uses ancillary analyses, such as alternative and supplementary calculations and sensitivity analyses. With the exception of concentration-response function estimates, these ancillary analyses usually examine only one source of uncertainty at a time and only for the impact on the mean value of the probability distribution from the primary analysis. As a consequence, though laudable steps in the right direction, these ancillary analyses do not adequately convey the relative or aggregate degree of uncertainty created by the sources of uncertainty addressed in the analyses, nor, of course, do they depict uncertainty from other sources. RECOMMENDATIONS EPA should begin to move the assessment of uncertainties from its ancillary analyses to its primary analyses. This shift will require the specification of a probability distribution for each uncertainty source that is added to the primary analysis and, as necessary, the specification of joint distributions for the uncertainty sources that are not independent of each other. Expert judgment, as well as data, will be required to specify these distributions. Although the effect on the mean of the resulting probability distribution might increase, decrease, or remain the same, the effect on the spread

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations of the distribution will be a predictable widening and, therefore, a more realistic depiction of the overall uncertainty in the analysis. As it incorporates additional sources of uncertainty into its primary health benefits analyses, EPA should consider conducting analyses to determine which uncertainty sources have the greatest influence on the mean and spread of the probability distribution. The need for these sensitivity analyses will be particularly great for distributions that are based on expert judgment. The uncertainty sources that have the greatest consequences for decision-making, including those that have the greatest impact on the spread of the distribution, should be given high priority for additional research. Because the incorporation of expert judgment when data are unavailable will influence the estimates of health benefits as well as the uncertainty analyses, the committee also recommends that EPA clearly distinguish between data-derived estimates of some components—such as the concentration-response function—and expert opinions about other components that are lacking in scientific data—such as the degree of compliance with a particular regulation 30 years into the future. In this way, policy-makers will better understand how existing data and expert judgment combine to produce estimates and where new data would be most valuable. As EPA begins the transition to incorporate additional sources of uncertainty into its primary health benefits analyses, it should continue the sensitivity analyses it has traditionally conducted. These analyses should be expanded, however, to consider sources of uncertainty jointly rather than singly. In presenting the probability distribution for each health benefit produced by a primary analysis, EPA should emphasize even more than it has in the past the sources of uncertainty that remain unaccounted for in the primary analysis. These uncertainties should continue to be described as completely and realistically as possible. EPA should consider providing a preliminary analysis that estimates in current populations the health benefits resulting from hypothetical changes in current levels of emissions. These preliminary analyses would help EPA develop an idea of the lower bound on the uncertainty of future consequences and would have fewer uncertainties than analyses of the impacts of proposed regulatory actions on future exposures and health outcomes. EPA should continue to strive to present the results of its health benefits analyses in ways that avoid conveying an unwarranted degree of

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations certainty. Such ways include rounding to fewer significant digits, increasing the use of graphs, presenting projected baseline along with projected health benefits, and placing less emphasis on single numbers (for example, the mean of the probability distribution for a health benefit) and greater emphasis on ranges (for example, the range between 5th and 95th percentiles of the distribution). There is a common misperception that a high degree of certainty is required for regulatory actions to take place to protect public health. As a result, primary health benefits analyses that more fully and accurately portray the uncertainties might not be considered useful. It is unrealistic for EPA to defer decisions until it can make them on the basis of perfect science. A careful and deliberate balancing of the benefits and costs is required, and this balancing must be informed by a fair assessment of the current levels of uncertainty and a realistic evaluation of the likely reductions in uncertainty attainable through further research. EPA should perform similar detailed analyses of uncertainty in the valuation of health benefits and in the regulatory cost analyses that the committee recommends for the health benefits analyses. REFERENCES CASAC (Clean Air Scientific Advisory Committee). 1986a. Letter to Craig Potter, Assistant Administrator for Air and Radiation, U.S. Environmental Protection, Washington, DC, from Morton Lippmann, Chairman, Clean Air Scientific Advisory Committee. SAB-CASAC-86-024. August 29, 1986. CASAC (Clean Air Scientific Advisory Committee). 1986b. Letter to Lee Thomas, Administrator, U.S. Environmental Protection, Washington, DC, from Morton Lippmann, Chairman, Clean Air Scientific Advisory Committee. SAB-CASAC-86-023. August 29, 1986. Cooke, R.M. 1991. Experts in Uncertainty: Opinion and Subjective Probability in Science. New York: Oxford University Press. Dockery, D.W., C.A. Pope, X. Xu, J.D. Spengler, J.H. Ware, M.E. Fay, B.G. Ferris, and F.E. Speizer. 1993. An association between air pollution and mortality in six U.S. cities. N. Engl. J. Med. 329(24):1753-1759. Dominici, F., A. McDermott, S. Zeger, and J. Samet. 2002. On the use of generalized additive models in time-series studies of air pollution and health . Am. J. Epidemiol. 156(3):193-203. EPA (U.S. Environmental Protection Agency). 1978. A Method for Assessing the Health Risks Associated with Alternative Air Quality Standards for Ozone.

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations Strategies and Air Standards Division, Office of Air Quality Planning and Standards, U.S. Environmental Protection Agency, Research Triangle Park, NC. July 1978. EPA (U.S. Environmental Protection Agency). 1997. Policy for Use of Probabilistic Analysis in Risk Assessment at the U.S. Environmental Protection Agency. Office of Research and Development, U.S. Environmental Protection Agency. May 15, 1997. [Online]. Available: http://www.epa.gov/ncea/mcpolicy.htm [accessed September 10, 2002]. EPA (U.S. Environmental Protection Agency). 1999a. Final Tier 2 Rule: Air Quality Estimation, Selected Health and Welfare Benefits Methods, and Benefit Analysis Results. EPA420-R-99-032. Office of Air Quality Planning and Standards, U.S. Environmental Protection Agency, Research Triangle Park, NC. December 1999. [Online]. Available: http://www.epa.gov/otaq/regs/ld-hwy/tier-2/frm/tsd/r99032.pdf [accessed September 10, 2002]. EPA (U.S. Environmental Protection Agency). 1999b. Final Report to Congress on Benefits and Costs of the Clean Air Act, 1990 to 2010. EPA 410-R-99-001. Office of Air and Radiation, U.S. Environmental Protection Agency. November 1999. Evans, J.S., J.D. Graham, G.M. Gray, and R.L. Sielken. 1994. A distributional approach to characterizing low-dose cancer risk. Risk Anal. 14(1):25-34. Feagans, T.B., and W.F. Biller. 1981. Risk assessment: Describing the protection provided by ambient air quality standards. Environ. Prof. 3(3-4):235-247. Greenland, S. 1990. Randomization, statistics, and causal inference. Epidemiology 1:421-429. Greenland, S. 1998. Basic methods for sensitivity analysis and external adjustment. Pp. 343-357 in Modern Epidemiology, 2nd Ed., K.J. Rothman, and S. Greenland, eds. Philadelphia, PA: Lippincott-Raven. Greenland S., K.B. Michels, J.M. Robins, C. Poole, and W.C. Willett. 1999. Statistical uncertainty in trends and dose-response relations. Am. J. Epidemiol. 149(12):1077-1086. Greenland, S. 2001. Sensitivity analysis, Monte Carlo risk analysis, and Bayesian uncertainty assessment. Risk Anal. 21(4):579-584. Hacking, I. 1984. The Emergence of Probability: A Philosophical Study of Early Ideas About Probability, Induction and Statistical Inference. Cambridge: Cambridge University Press. Hattis, D., and E.L. Anderson. 1999. What should be the implications of uncertainty, variability, and inherent “biases”/”conservatism” for risk management decision making? Risk Anal. 19(1):95-107. Hattis, D., and D.E. Burmaster. 1994. Assessment of variability and uncertainty distributions for practical risk analyses. Risk Anal. 14(5):713-730. Hattis, D., A. Russ, R. Goble, P. Banati, and M Chu. 2001. Human interindividual variability in susceptibility to airborne particles. Risk Anal. 21(4):585-599.

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations Holland, M., D. Forster, and M. Wenborn. 1999. Economic Evaluation of Proposals Under the UNECE Multi-Effects and Multi-Pollutant Protocol. AEAT-4587. Prepared by AEA Technology, Oxfordshire, U.K., for European Commission, DGXI, Brussels and Luxembourg. January 1999. Krewski, D., S.N. Rai, J.M. Zielinski, and P.K. Hopke. 1999. Characterization of uncertainty and variability in residential radon cancer risks. Ann. N.Y. Acad. Sci. 895:245-272. Lang, C.G., G. Yarwood, F. Lalonde, and R. Bloxam. 1995. Environmental and Health Benefits of Cleaner Vehicles and Fuels. Prepared for Canadian Council of Ministers of the Environment Task Force on Cleaner Vehicles and Fuels, Winnipeg, Manitoba. October 1995. Lindley, D.V. 2000. The philosophy of statistics (with discussion). Statistician 49:293-337. Lindley, D.V. 1985. Making Decisions, 2nd Ed. London: John Wiley and Sons. McCurdy, T., and H.M. Richmond. 1983. Description of the OAQPS Risk Program and the Ongoing Lead NAAQS Risk Assessment Project. Paper 83-74.1. Presented at the 76th Annual Meeting of the Air Pollution Control Association, June 19-24, 1983, Atlanta, GA. Morgan, M.G., S.C. Morris, M. Henrion, D.A.L. Amaral, and W.R. Rish. 1984. Technical uncertainty in quantitative policy analysis—a sulfur air pollution example. Risk Anal. 4(3):201-213. Morgan, M.G., M. Henrion, and M. Small. 1990. Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. Cambridge: Cambridge University Press. NAE (National Academy of Engineering). 1972. Perspectives on Benefit-Risk Decision Making. Washington, DC: National Academy of Engineering. NMMAPS (National Morbidity, Mortality, and Air Pollution Study). 2002. Updated NMMAPS results in the National Morbidity, Mortality, and Air Pollution Study. Johns Hopkins University, Bloomberg School of Public Health, Baltimore, MD. [Online]. Available: http://www.biostat.jhsph.edu/biostat/research/update.main.htm [accessed Sept. 9, 2002]. NRC (National Research Council). 1975. Air Quality and Stationary Source Emission Control: A Report. Serial No 94-4. Washington, DC: U.S. Government Printing Office. NRC (National Research Council). 1979a. Protection against Depletion of Stratospheric Ozone by Chlorofluorocarbons. Washington, DC: National Academy Press. NRC (National Research Council). 1979b. Stratospheric Ozone Depletion by Halocarbons: Chemistry and Transport. Washington, DC: National Academy Press. NRC (National Research Council). 1982. Risk and Decision Making: Perspectives and Research. Washington, DC: National Academy Press.

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations NRC (National Research Council). 1983. Risk Assessment in the Federal Government: Managing the Process. Washington, DC: National Academy Press. NRC (National Research Council). 1994. Science and Judgment in Risk Assessment. Washington, DC: National Academy Press. NRC (National Research Council). 1996. Understanding Risk: Informing Decisions in a Democratic Society, P.C. Stern, and H.V. Fineberg, eds. Washington, DC: National Academy Press. North, D.W. and M.W. Merkhofer. 1976. A methodology for analyzing emission control strategies. Comput. Oper. Res. 3:187-207. Pate-Cornell, M.E. 1996. Uncertainties in risk analysis: Six levels of treatment. Reliab. Eng. Syst. Saf. 54(2-3):95-111. Poole, C. 1988. Feelings and frequencies: Two kinds of probability in public health research. Am. J. Public Health 78(12):1531-1533. Poole, C. 2001. Low P-values or narrow confidence intervals: Which are more durable? [Editorial]. Epidemiology 12(3):291-294. Pope, C.A., III, M.J. Thun, M.M. Namboodiri, D.W. Dockery, J.S. Evans, F.E. Speizer, and C.W. Heath Jr. 1995. Particulate air pollution as a predictor of mortality in a prospective study of U.S. adults. Am. J. Respir. Crit. Care Med. 151(3 Pt 1):669-674. Presidential/Congressional Commission on Risk Assessment and Risk Management. 1997. Framework for Environmental Health Risk Management: Final Report. Washington, DC: The Commission. Raiffa, H. 1970. Decision Analysis: Introductory Lectures on Choices under Uncertainty. Reading, MA: Addison-Wesley. Reilly, J., P.H. Stone, C.E. Forest, M.D. Webster, H.D. Jacoby, and R.G. Prinn. 2001. Climate Change. Uncertainty and climate change assessments. Science 293(5529):430-433. Richmond, H.M. 1981. A framework for assessment of health risks associated with national ambient air quality standards. Environ. Prof. 3(3-4):225-234. Read, D., and M.G. Morgan. 1998. The efficacy of different methods for informing the public about the range dependency of magnetic fields from high voltage power lines. Risk Anal. 18(5):603-610. Rosenbaum, A.S., R.L. Winkler, T.S. Wallsten, R.G. Whitfield, and H.M. Richmond. 1995. An assessment of the risk of chronic lung injury attributable to long-term ozone exposure. Operations Research 43(1):19-28. SAB (Science Advisory Board). 1979. Review of A Method of Assessing the Health Risks Associated with Alternative Air Quality Standards for Ozone, Report of the Subcommittee on Health Risk Assessment. EPA/SAB/79/001. Science Advisory Board, U.S. Environmental Protection Agency, Washington, DC. September 1979. SAB (Science Advisory Board). 1980. Approaches to Health Risk Assessment for Alternative National Ambient Air Quality Standards, Report of the Subcommit

OCR for page 126
Estimating the Public Health Benefits of Proposed Air Pollution Regulations tee on Health Risk Assessment. EPA/SAB/80/003. Science Advisory Board, U.S. Environmental Protection Agency, Washington, DC. December 1980. Seigneur, C., E. Constantinou, and T. Permutt. 1994. Uncertainty analysis of health risk estimates, Appendix F. Pp. 453-478 in Science and Judgment in Risk Assessment. Washington, DC: National Academy Press. Whitfield, R.G., T.S. Wallsten, R.L. Winkler, H.M. Richmond, and S.R. Hayes. 1991. Assessing the Risk of Chronic Lung Injury Attributable to Long-Term Ozone Exposure. Report ANL/EAIS-2. NTIS/DE91016814. Argonne National Laboratory, IL. July 1991.