National Academies Press: OpenBook
« Previous: 4 Exposure and Response
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

5
Uncertainty

There are several major barriers to broad acceptance of recent EPA health benefits analyses. One barrier is the large amount of uncertainty inherent in these analyses, and another is the manner in which the agency deals with this uncertainty. A third barrier is that projected health benefits are often reported as absolute numbers of avoided death or adverse health outcomes without a context of population size or total numbers of outcomes. Areas of uncertainty include air-quality modeling, population demographics and heterogeneity, intersubject variability, health and exposure baselines, compliance with control measures, effectiveness of controls in reducing pollutant emissions, validity and precision of concentration-response functions and use of alternative models (linear, nonlinear), estimation of these functions as relative effects (relative risks) or absolute effects (risk differences), relative toxicity of mixture components, and applicability of these functions to target populations of regulatory concern. These uncertainties are rooted in incomplete scientific knowledge. When benefits are estimated for future target populations, the cumulative magnitude of the uncertainties can be formidable. Many of them can be reduced by further research, but on the whole, they are likely to remain high.

Even great uncertainty does not imply that action to promote or protect public health should be delayed. Decisions about whether to act, when to act, and how aggressively to act can only be made with some understanding of the likelihood and consequences of alternative courses of action. The potential for improving decisions through research must be balanced against

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

the public health costs incurred because of a delay in the implementation of controls. Complete certainty is an unattainable ideal.

Health benefits analyses compare alternative scenarios that would result with and without regulatory action. As a consequence, these analyses are inherently speculative and their results unverifiable. Because only one regulatory option can be chosen by decision-makers, the outcomes of the remaining regulatory options, including the baseline with no action (if not chosen), can never be directly observed.

Analyses of health benefits should represent the uncertainties in the choices facing decision-makers and society at large (Hattis and Anderson 1999). Analyses should attempt to provide insight into the variability of impacts (among persons, places, and other dimensions of interest) and the extent and sources of uncertainties in the results. The representation of uncertainty requires a good faith appraisal of the imperfection in the state of information about these impacts (Hattis and Burmaster 1994). Uncertainty assessment should not overrepresent or underrepresent the quality and completeness of available information.

This chapter discusses EPA’s current approach to assessing uncertainty in health benefits analyses for air pollution control regulations. The agency’s analysis of the health benefits for the final Tier 2 vehicle emissions standards and gasoline sulfur control rule-making (EPA 1999a) is used for illustration. The chapter outlines a revised approach that would reflect overall uncertainty more realistically, in part by using probabilistic expressions of expert judgment. The chapter also briefly reviews the history of probabilistic uncertainty assessment in EPA health benefits analyses under the Clean Air Act.

This chapter is confined to uncertainty in the analysis of health benefits expressed solely in terms of health. Although uncertainties in the monetary valuation of health benefits and in the analysis of regulatory costs are not considered, the committee notes that there are great uncertainties in those analyses as well.

EPA’S APPROACH TO UNCERTAINTY ANALYSIS

EPA uses a two-part approach to assessing uncertainty in health benefits analyses that rely on epidemiological studies as the source of estimated concentration-response functions, although different approaches are some

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

times used, especially when epidemiological evidence is lacking (EPA 1997). The first part is a primary analysis, which produces numerical estimates or projections of each health benefit in the form of a probability distribution. This analysis incorporates only one source of uncertainty: the random sampling error in the epidemiological study or studies that provide the estimated concentration-response function. The second part of the uncertainty assessment is an array of ancillary analyses in which many other sources of uncertainty are considered in several disparate ways.

Primary Uncertainty Analysis

The primary uncertainty analysis produces a numerical estimate of each health benefit EPA believes to be plausible for a particular regulatory action. Typically, the benefit is expressed as a number of deaths or cases of an adverse health event that will be avoided in the United States in a future year if some regulatory action is taken. The year chosen is often far into the future to allow for the action to be implemented, for the implementation to result in exposure reductions, and for the reduced exposures to result in health benefits. In the Tier 2 analysis, the chosen year was 2030.

EPA reports each numerical health benefit estimate in the form of a probability distribution and summarizes the distribution by reporting its mean and 5th and 95th percentiles. The distribution assigns a nonzero probability to every possible value including the null hypothesis of no benefit. The mean of the distribution is interpreted as the expected benefit based upon the analysis performed. The 5th and 95th percentiles are defined as a credible range within which the true benefit value will lie with a 90% probability (EPA 1999a, p. 3-26).

The solid line in Figure 5-1 shows the probability distribution from EPA’s primary analysis of avoided mortality for the proposed Tier 2 rule for the year 2030. The mean of the distribution (which is also the median and the 50th percentile) is 4,307 avoided deaths among persons 30 years of age and older. The 5th and 95th percentiles are 2,671 and 5,891 avoided deaths, respectively (EPA 1999a, p. 6-3).

The probability models in EPA’s primary analyses incorporate only one of the many sources of uncertainty in these analyses: the random sampling error in the estimated concentration-response function derived from either an epidemiological study or a meta-analytic or pooled aggregation of two or

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

FIGURE 5-1 Probability distributions from primary and alternative analyses of avoided mortality for Tier 2 analysis. Source: Data from EPA 1999a.

more such studies. In a meta or pooled analysis of separate studies, a summary estimate of the concentration-response function is produced by averaging study estimates that may include ones that vary in strength and ones that suggest little or no effect. To estimate avoided mortality for the Tier 2 rule, the agency chose an estimated concentration-response function from a log-linear (Poisson regression) analysis of results from a study by the American Cancer Society (Pope et al. 1995). For a change in concentration from 9 to 33.5 • g/m3, the result was an estimated relative risk of 1.17 with a 95% confidence interval of 1.09 to 1.26 (EPA 1999a, p. C-2). The random sampling error represented by this confidence interval is the only source of uncertainty in the agency’s probability distribution for avoided mortality. The incorporation of additional sources of uncertainty would widen the distribution.

EPA correctly notes that incorporating only the uncertainty from random sampling error in concentration-response function estimates into its primary health benefits analyses “omits important sources of uncertainty, such as the contribution of air quality changes, baseline population incidences, projected populations exposed, transferability of the concentration-

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

response function to diverse locations, and uncertainty about premature mortality” and “would provide a misleading picture about the overall uncertainty in the estimates” (EPA 1999a, p. 3-26).

Ancillary Uncertainty Analyses

EPA assesses all other uncertainties in a second part of each health benefits analysis. The agency begins with a list of as many key uncertainties as it can identify. The list compiled for the Tier 2 analysis is given in Table 5-1. Much of this uncertainty results from unavoidable and expected variability or heterogeneity in concentration-response functions estimated by epidemiological studies. Some of it results from baseline statistical variation, as no study has infinite sample size and all study populations differ in their distributions of background causes of health outcomes and in their distributions of susceptibility to toxic agents. Projection of future baselines, such as the death rate to be expected 30 or more years in the future if no action is taken, are particularly uncertain. Important uncertainty is also produced by variation in study design, data collection, and statistical analysis. Although there may be other uncertainties that have not been identified, EPA typically makes no allowance for these unidentified sources of uncertainty.

EPA takes a variety of approaches regarding these identified uncertainties. Some are merely mentioned. Other uncertainties are discussed qualitatively with regard to the direction and, sometimes, the magnitude of the impact that they are likely to exert on the mean value of the probability distribution. For example, in the discussion of the epidemiological study providing the estimated concentration-response function for avoided mortality in the Tier 2 analysis, EPA referred to downward biases from the relatively healthy study population and from intercity migration of study participants, which the agency believed would counteract an upward bias associated with historical air-quality trends (EPA 1999a, p. C-1).

For selected sources of uncertainty, EPA conducted supplemental calculations, alternative calculations, and sensitivity analyses (EPA 1999a, p. 3-19). These terms have specific meanings in EPA health benefits assessments. Supplemental calculations “provide additional information about specific health effects, but are not suitable for inclusion in the primary or

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

TABLE 5-1 Key Sources of Uncertainty in the Tier 2 Benefits Analysis

1.

Uncertainties Associated with Concentration-Response (C-R) Functions

The value of the ozone- or particulate matter (PM)-coefficient in each C-R function.

Application of a single C-R function to pollutant changes and populations in all locations.

Similarity of future year C-R relationships to current C-R relationships.

Correct functional form of each C-R relationship.

Extrapolation of C-R relationships beyond the range of ozone or PM concentrations observed in the study.

2.

Uncertainties Associated with Ozone and PM Concentrations

Estimating future-year baseline and hourly ozone and daily PM concentrations.

Estimating the change in ozone and PM resulting from the control policy.

3.

Uncertainties Associated with PM Mortality Risk

No scientific literature supports a direct biological mechanism for observed epidemiological evidence.

Direct causal agents within the complex mixture of PM responsible for reported health effects have not been identified.

The extent to which adverse health effects are associated with low level exposures that occur many times in the year versus peak exposures.

Possible confounding in the epidemiological studies of PM2.5 effects with other factors (such as other air pollutants, weather, and indoor and outdoor air).

The extent to which effects reported in the long-term studies are associated with historically higher concentrations of PM rather than the concentrations occurring during the period of study.

Reliability of the limited ambient PM2.5 monitoring data in reflecting actual PM2.5 exposures.

4.

Uncertainties Associated with Possible Lagged Effects

What portion of the PM-related long-term exposure mortality associated with changes in annual PM levels would occur in a single year, and what portion might occur in subsequent years.

5.

Uncertainties Associated with Baseline Incidence Rates

Some baseline incidence rates are not location-specific (such as those taken from studies) and might not accurately represent the location-specific rates of interest.

Current baseline incidence rates might not approximate baseline incidence rates in the year 2030.

Projected population and current demographics—used to derive incidences—might not approximate future-year populations and demographics.

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

6.

Uncertainties Associated with Aggregation of Monetized Benefits

Health and welfare benefit estimates are limited to the available C-R functions. Thus, unquantified benefit categories will cause total benefits to be underestimated.

 

Source: Adapted from EPA 1999a, Exhibit 3-3, p. 3-20.

alternative estimates due to concerns about double-counting of benefits or the high degree of uncertainty about the estimates” (EPA 1999a, p. 3-21). The supplemental analyses in the Tier 2 report pertained to short-term mortality, infant (postneonatal) mortality, ozone mortality, asthma attacks, restricted-activity days, and ozone-related cardiovascular disease (EPA 1999a, pp. 3-23, 3-24, A-1).

In other contexts, both EPA’s alternative calculations and sensitivity analyses would be called sensitivity analyses (Morgan et al. 1990; Greenland 1998). The distinction for EPA lies in its judgment of their plausibility. Alternative calculations “are based on relatively plausible alternatives to the assumptions used in deriving the primary benefit estimates” (EPA 1999a, p. 3-21). Sensitivity analyses “examine the sensitivity of estimated benefits results to less plausible alternatives to the assumptions used in the primary analyses” (EPA 1999a, p. 3-25). For both calculations and analyses, assumptions or sources of uncertain quantities are varied and the mean of the health benefit probability distribution is recomputed.

In all cases, the alternative calculations and sensitivity analyses are conducted for only one source of uncertainty at a time. In addition, they are conducted only to determine the sensitivity of the mean of the probability distribution from the primary analysis to modified assumptions and information sources. With one exception, the spread of the health benefit probability distribution, as gauged by the distance of the interval between its 5th and 95th percentiles, is not affected. EPA’s rationale for focusing only on the mean is that an “attempt to assign probabilities to these alternative calculations…would only add to the uncertainty of the analysis or present a false picture about the precision of the results” (EPA 1999a, p. 3-21). EPA does not discuss why adding to the uncertainty of the analysis would be inappropriate. Noting that some analyses of health benefits of air pollution reductions (Lang et al. 1995; Holland et al. 1999) have included the assignment of “probabilities to ranges of parameter values for different endpoints,”

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

EPA argued that “the estimated points on these distributions are themselves highly uncertain and very sensitive to the subjective judgements of the analyst. To avoid these subjective judgements, we choose to allow the reader to determine the weights they would assign to alternative estimates” (EPA 1999a, p. 3-21).

For the Tier 2 analysis, alternative calculations were performed for an alternative source of the estimated concentration-response function and for life-years saved rather than avoided deaths as a measure of health benefit (EPA 1999a, pp. 3-21, 23). Sensitivity analyses were conducted for thresholds and alternative lag structures (EPA 1999a, p. 3-25).

The one exception to the exclusive focus on the mean of the health benefit probability distribution occurs when an alternative calculation involves the use of a different study to provide the estimated concentration-response function, which has its own standard error estimate. The broken line in Figure 5-1 shows the probability distribution when the concentration-response function from an analysis of the Harvard six cities study (Dockery et al. 1993) is used. This study produced a higher point estimate of the relative risk, so the mean of the probability distribution is higher (10,000 avoided deaths). The alternative study was smaller, however, so its estimate had more random sampling error and the distribution is wider. The 5th and 95th percentiles are 5,000 and 15,000 avoided deaths, respectively.

CRITIQUE OF EPA’S CURRENT UNCERTAINTY ASSESSMENTS

Numerical projections appear to be essential in health benefits analyses, and probability distributions can be used to describe the uncertainty in these analyses. Issues arise, however, over which sources of uncertainty the distributions should incorporate, how to incorporate them, and how to present the results. EPA’s decision to incorporate only one source of uncertainty, the random sampling error in the estimated concentration-response function, into the probability distributions resulting from its health benefits analyses is worth reconsidering. The committee agrees with the agency’s judgment that its current practice produces health benefits probability distributions that give “a misleading picture about the overall uncertainty in the estimates” (EPA 1999a, p. 3-26). In particular, the distributions suggest that there is less uncertainty, perhaps much less, than is actually present.

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

The committee finds that the mean of the distributions should not be interpreted as “best” estimates, and the intervals between the 5th and 95th percentiles of the distributions should not be interpreted as “90 percent credible intervals,” within which “the true benefit lies with 90 percent probability” (EPA 1999a, p. 3-26).

The committee agrees with EPA’s statement that it would require expert judgment to specify probability distributions for many of the uncertain components of the health benefits analyses. In these cases, probability would be used not only in its connotation of variability1 but also in its connotation of subjective uncertainty or lack of complete belief as well (Hacking 1984; Poole 1988; Lindley 2000). EPA is correct that the elicitation of expert opinions in the form of probability distributions is a difficult and uncertain process (Morgan et al. 1990; Cooke 1991; Pate-Cornell 1996). The committee does not agree, however, that these difficulties are sufficient reasons for not trying to obtain such advice. Nor does the committee find any reason to avoid the attempt on the ground that it “would only add to the uncertainty of the analysis or present a false picture about the precision of the results” (EPA 1999a, p. 3-21). On the contrary, by growing wider, the health benefits probability distributions would more accurately depict the uncertainty and lack of precision in the analyses. As difficult and uncertain as these specifications are, they are preferable to EPA’s current practice of treating important and highly uncertain model components as though they were certain.

The probability models from which standard errors are estimated for concentration-response-function estimates from observational epidemiological studies are less than certain as well. These models would have a firm theoretical foundation only if study populations were randomly sampled from target populations and exposure concentrations were randomly allocated to study participants (Greenland 1990; Poole 2001). In observational studies such as the American Cancer Society study (Pope et al. 1995) and the

1  

In risk analysis, a distinction is often made between characterization of variability (the true variation in a parameter over time, space, or persons) and uncertainty (ignorance about the true value of the parameter). Variability is characterized primarily to provide information about the true distribution of exposure and risk and to suggest opportunities for control or to provide a sense of equity. Uncertainty is characterized primarily to give a sense of the confidence that can be placed in study results and to help in setting priorities.

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

Harvard six cities study (Dockery et al. 1993), neither random sampling nor random exposure allocation was used. Nevertheless, analysts use probability models for these design features in analyzing observational data because larger observational studies have less variability than smaller ones and account for the incomplete enumeration of the population of interest.

The applicability of probability models for random variability to observational data is less than perfectly secure, but their use is preferable to assuming that there is no variability related to study size in observational results. The use of probability models for uncertainties involving expert judgment is also preferable to assuming that these uncertainties do not exist.

Many of the key uncertainties in these analyses may be characterized only subjectively by reference to expert judgment. The question is not whether to rely on expert judgment but how best to elicit and summarize the views of experts and how to incorporate them into the analysis. Probability distributions are a legitimate and useful way to express the uncertainties in expert judgments. Incorporation of those uncertainties as probability distributions into the primary analysis would likely change the expected value and widen the resulting probability distribution for each health benefit. The result will include more of the uncertainty in the health benefits assessment.

The alternative calculations and sensitivity analyses conducted by EPA help to describe the uncertainty in the analyses, but they are not sufficient. The major problems with them are that EPA consigns them to an ancillary status and not to the primary analysis, that the various sources of uncertainty are considered one at a time, and that EPA explicitly offers no judgment as to the relative plausibility of the alternative scenarios considered in these analyses. Without a combined, simultaneous assessment of multiple uncertainty sources, it is impossible to gain an appreciation of the overall magnitude of the uncertainty in the analysis. The committee does not agree with the agency’s decision to have the reader determine the plausibility and relative weighting of alternative assumptions and data sources and integrate these assessments across uncertainty sources.

In its current analyses, EPA does not systematically or probabilistically address the extension of results beyond a study population’s age range. The typical assumption is that the health-outcome-rate ratio is constant across age; however, this assumption is seldom tested and seldom has any strong etiological justification, even when compared with a simple alternative, such as a constant-rate difference. For example, the method of extrapolating to additional age groups can be of crucial importance if the study

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

population excludes elderly persons who are at especially high baseline risk and the target population includes a sizable proportion of elderly persons. A large portion of the overall health benefit may then be projected for an age range that has not been studied. In such cases, the mixture of model and data would be tilted heavily toward the model.

Two additional illustrative examples are thresholds for adverse effects and lag structures.2 EPA considers implausible any threshold for mortality in the particulate matter (PM) exposure ranges under consideration (EPA 1999a, p. 3-8). Although the agency conducts sensitivity analyses incorporating thresholds, it provides no judgment as to their relative plausibility. In a probabilistic uncertainty analysis, EPA could assign appropriate weights to various threshold models. For PM-related mortality in the Tier 2 analysis, the committee expects that this approach would have resulted in only a slight widening of the probability distribution for avoided mortality and a slight reduction in the mean of that distribution, thus reflecting EPA’s views about the implausibility of thresholds. The committee finds that such formal incorporation of EPA’s expert judgments about the plausibility of thresholds into its primary analysis would have been an improvement.

Uncertainty about thresholds is a special aspect of uncertainty about the shape of concentration-response functions. Typically, EPA and authors of epidemiological studies assume that these functions are linear on some scale. Often, the scale is a logarithmic transformation of the risk or rate of the health outcome, but when a rate or risk is low, a linear function on the logarithmic scale is approximately linear on the scale of the rate or risk itself. Increasingly, epidemiological investigators are employing analytic methods that permit the estimation of nonlinear shapes for concentration-response functions (Greenland et al. 1999). As a consequence, EPA will need to be prepared to incorporate nonlinear concentration-response functions from epidemiological studies into the agency’s health benefits analyses. Any source of error or bias that can distort an epidemiological association can also distort the shape of an estimated concentration -response function, as can variation in individual susceptibility (Hattis and Burmaster 1994; Hattis et al. 2001).

EPA expressed much less certainty about alternative lag structures than it did about thresholds in the Tier 2 analysis. The lag structure used in the

2  

A lag reflects the time course between pollutant exposure and development of clinical disease. A lag structure reflects the variation among the population in the lags experienced by various individuals.

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

primary analysis was recommended by the Science Advisory Board (EPA 1999a, pp. 4-6, 4-7), but the agency considered a range of alternative lag structures plausible. Here a probabilistic weighting of alternative lag structures based on expert judgment might have led to a more appreciable widening of the health benefit probability distribution.

Although EPA considered alternative lag structures to vary in plausibility, these variations were not, but could have been, approximately captured by subjective probability distributions. The incorporation of these distributions into the final probability distribution for the primary analysis would have resulted in a more realistic presentation of acknowledged sources of uncertainty.

In principle, many components of the health benefits model need realistic probabilistic models (see Table 5-1 for a listing of such components), in addition to concentration-response thresholds and time lags between exposure and response. For example, additional features of the concentration-response function—such as projection of the results from the study population to the target populations (which may have etiologically relevant characteristics outside the range seen in the study population) and the projection of baseline frequencies of morbidity and mortality into the future—must be characterized probabilistically. Other uncertainties that might affect the probability distributions are the estimations of population exposure (or even concentration) from emissions, estimates of emissions themselves, and the relative toxicity of various classes of particles. Similarly, many aspects of the analysis of the impact of regulation on ambient concentrations and on population exposure involve considerable uncertainty and, therefore, may be beneficially modeled in this way. Depending on the analytic approach used, joint probability distributions will have to be specified to incorporate correlations between model components that are structurally dependent upon each other, or the analysis will have to be conducted in a sequential fashion that follows the model for the data-generating process.

EPA should explore alternative options for incorporating expert judgment into its probabilistic uncertainty analyses. The agency possesses considerable internal expertise, which should be employed as fully as possible. Outside experts should also be consulted as needed, individually or in panels. In all cases, when expert judgment is used in the construction of a model component, the experts should be identified and the rationales and empirical bases for their judgments should be made available.

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

One other potential limitation of the sensitivity analyses and alternative calculations is that the longer and more detailed the ancillary uncertainty analyses are, the less realistic the results of the primary analyses tend to be. Decision-makers and others may be tempted, however, to reach the opposite conclusion. The length and depth of the ancillary uncertainty analyses might give the impression that the mean and 90% “credible interval” from the primary analysis have been rendered more certain and well supported when the opposite is true. The mean of a health benefit probability distribution is in some sense the best single estimate, but no estimate can be considered best if only one of the large number of uncertainties is included in the analysis producing that estimate. Moreover, variable degrees of uncertainty among analyses dictate that some best estimates are better than others.

EPA should present the results of its health benefits analyses in ways that foster an appropriate appreciation of their inherent uncertainty. The reporting of too many significant digits, as in the Tier 2 estimate of 682,898 work-loss days avoided (EPA 1999a, p. 6-3), lends an unwarranted impression of exactitude. EPA’s focus on the mean value of the distribution rather than on the distribution range, such as the interval from the 5th to the 95th percentile of the distribution, also contributes to an impression of undue precision. The absence of graphical displays of the probability distributions, as shown in Figure 5-1, prevents an understanding of how sharply or gradually the probability falls away from its highest values to less plausible health benefits.

EXAMPLES OF UNCERTAINTY ASSESSMENTS

EPA is aware of health benefits analyses for air pollution reduction measures that have used probabilistic uncertainty assessment incorporating expert judgment (Lang et al. 1995; Holland et al. 1999). These assessments use the same methods as those the agency uses to combine health benefits probability distributions with probability distributions for valuation of those benefits as was done for the prospective analysis of the 1990 Clean Air Act Amendments (EPA 1999b). Furthermore, EPA has a limited but promising history of exploring the use of probabilistic uncertainty assessment in air pollution health benefits analysis.

Greenland (2001) illustrated the impact that one additional source of uncertainty can have on an analysis conducted to comply with California

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

regulatory guidelines. The analysis looked at the relative risk of skin cancer in a cohort of patients with severe psoriasis who received topical coal-tarderived therapy. One of the sources of uncertainty pertained to the cohort’s baseline expectation of skin cancer frequency. Typically, this source of uncertainty would be qualitatively described as a “study limitation.” The analysis, which was similar to an EPA health benefits analysis, produced a point estimate of 0.71 and a 95% confidence interval of 0.46 to 1.12. When a probability distribution for this source of uncertainty was incorporated into the analysis, the mean of the probability distribution shifted upward to a relative risk of 0.77, and the 95% probability interval widened to a range of 0.43 to 1.37. Adding further sources of uncertainty to the analysis might cause the mean to rise or fall but would further widen the probability distribution.

In 1994, the National Research Council recommended that EPA conduct formal uncertainty analyses, including probabilistic assessment of uncertainties that “cannot be quantified on the basis of data” (NRC 1994, p. 12) and that therefore require expert judgment to quantify. That committee observed, “Objective probabilities might seem inherently more accurate than subjective probabilities, but this is not always true…. There can be no rule that objective probability estimates are always to be preferred to subjective estimates, or vice versa” (NRC 1994).

The EPA Office of Air Quality Planning and Standards (OAQPS) has demonstrated the potential use of formal expert judgment in support of the development of air quality standards (McCurdy and Richmond 1983). An initial effort in 1977 was made to apply judgmental probabilities elicited from health and air-quality experts to evaluate the health risks associated with alternative air-quality standards for ozone (EPA 1978). In 1979, the EPA Science Advisory Board reviewed this work, commended EPA for its initiative, critiqued certain elements of the approach used, and recommended further work in this area (SAB 1979). The major criticism of this initial work was that rather than focusing on point estimates of subjective probabilities, EPA attempted to elicit secondary probabilities (interval estimates of the probabilities).3 The SAB (1980) endorsed the concept but asked EPA to develop approaches that did not involve secondary probability.

In 1980, OAQPS held a public meeting involving six groups of experts

3  

Discussion. Presented by H.M. Richmond at the American Statistical Association–EPA conference, Washington, DC, October 2, 1986.

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

from the fields of decision analysis, biostatistics, and behavioral psychology to discuss alternative approaches for the elicitation of expert judgment. Two approaches were selected for further development. OAQPS began work to illustrate how these approaches could be applied to estimate the health risks from carbon monoxide and also began to conduct an illustrative probabilistic risk assessment for lead.

By 1983, the OAQPS effort moved from the developmental stage to initiation of its lead National Ambient Air Quality Standard (NAAQS) risk assessment. Probabilistic dose-response functions for two health outcomes of lead exposure were elicited from 10 nationally recognized experts. This work was reviewed favorably by the EPA’s Clean Air Scientific Advisory Committee (CASAC 1986a,b). In 1990-1991, a second effort was made to apply expert judgment to assess the risks of ozone. In view of the lack of adequate human data, EPA developed a risk assessment for chronic lung damage from ozone based on formally elicited expert judgment (Rosenbaum et al. 1995). However, the risk assessment for chronic lung injury was not formally used in support of the 1997 NAAQS revision for ozone. By the time the standard was set, the risk assessment was out of date, and the experts elicited had been told that their judgments would not be used for standard setting.

The committee recommends that the EPA offices responsible for health benefits analysis build on OAQPS’ experience. Although the specific methods for selection and elicitation of experts may need to be modified somewhat, the protocols that have been developed and tested by OAQPS provide a solid foundation for future work in this area. EPA may also consider having its approaches reviewed and critiqued by decision analysts, biostatisticians, and psychologists from other fields where expert judgment has been applied (for example, nuclear-power-plant-accident-consequence risk assessment). Much has been learned in this area since EPA’s last formal review of methods in the late 1970s.

An approach for the analysis of uncertainty is the 1994 NRC report, which included a case study of a probabilistic uncertainty analysis in an assessment of cancer risk from coal-fired power plant emissions of chromium, arsenic, cadmium, and benzene. The authors identified 49 uncertain parameters, which they reduced to 22 on the basis of a preliminary assessment of their degree of uncertainty and potential influence on the final results. “Evaluation of the probability distributions of the 22 influential parameters of the model was performed on the basis of available statistical data,

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

literature value ranges, and personal judgment” (Seigneur et al. 1994). This analysis is mentioned not for its substantive implications, but because it illustrates how the propagation of the uncertainties in many individual parameters can be analyzed and how the results of such an analysis can be used to identify the most influential sources of uncertainty.

A recent analysis of estimated short-term mortality of PM2.5 provides another example (Dominici et al. 2002). Figure 5-2 shows the effect of adjusting for ozone, carbon monoxide, and sulfur dioxide. The mean of the resulting distribution does not change very much, but the spread of the distribution appropriately widens. Although not the result of a formal probabilistic uncertainty analysis, this widening is an approximate reflection of existing uncertainty that these copollutants might affect mortality and be associated with PM2.5, thus confounding the PM2.5 effect estimate.

RECOMMENDED APPROACH TO UNCERTAINTY ASSESSMENT

Other NRC reports addressed the issue of uncertainty in risk assessment and benefits estimation (NAE 1972; NRC 1975, 1982, 1983, 1994, 1996; Presidential Commission 1997). Without exception, they found that proper characterization of uncertainty is essential. Almost all expressed concern that most risk assessments and health benefits analyses tend to underestimate uncertainties and leave decision-makers with a false sense of confidence in estimates of risk. In addition, most of the reports suggested that failure to address model uncertainty adequately is often a major issue.

Despite this broad agreement on the importance of honest characterization of uncertainty and the common view that model uncertainty in particular tends to be understated or ignored, the NRC reports reached somewhat different conclusions about how best to resolve the problem. All agreed that sources of uncertainty should be listed and described, and most recommended that parameter uncertainties (and variability) be quantitatively characterized. Furthermore, they recommended that formal approaches for uncertainty, such as Monte Carlo analysis, be applied to understand the cumulative uncertainty and to provide insight into the dominant sources of parameter uncertainty. However, on the question of characterization of model uncertainty, differences in opinion emerged—some discussed and

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

FIGURE 5-2 Posterior probability distributions for estimated short-term mortality effects of PM10 with and without adjustment for copollutants. The x-axis is the percent change in daily mortality ([relative risk •1] × 100%) for a 10 •g/m3 increase in PM10 concentration. Thus, 0.2% is a relative risk of 1.002. Source: Adapted from NMMAPS 2002.

recommended the use of expert judgment to characterize epistemic uncertainty, while others recommended that such basic scientific uncertainties should be thoroughly described but not quantified.

The committee shares the view that proper characterization of uncertainty is essential to good decision-making and agrees that uncertainties are often underestimated, leaving decision-makers with a false sense of security. Having reached this conclusion, the committee shares the view of M. Granger Morgan (Morgan et al. 1990):

When the value of an uncertain quantity is needed in policy analysis, and limits in data or understanding preclude the use of conventional statistical techniques to produce probabilistic estimates, about the only remaining option is to ask experts for their best professional judgment.

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

In the committee’s view then, the question is not whether to use expert judgment but how to use expert judgment. The options are to pick a particular model component from a range of varyingly plausible alternatives and treat that one as though it were absolutely certain or to specify a judgment-based probability model for the alternatives that reflects their varying degrees of plausibility and incorporate those probabilities into the primary analysis. The latter option has many difficulties, but it has the potential to portray the existing uncertainty more realistically than the former option does.

The committee recommends that EPA begin to incorporate additional sources of uncertainty into the probability models it uses in its primary health benefits analyses. Furthermore, the committee recognizes that decision-makers will need to be informed about how and why uncertainty was added to the health benefits analysis and how, in turn, this uncertainty might be communicated to the public. This process will use probability distributions to replace model components that are treated as known fixed values. Of necessity, the probability distributions for the uncertain model components will have to reflect a combination of empirical observations and expert judgment. This will result in a more realistic picture of the overall uncertainty in the analyses. The mean of the health benefit distribution will reflect the expected magnitude of the health benefit more accurately and, as a consequence, will be more defensible. The mean might shift upward, downward, or not at all as each additional source of uncertainty is added to the core analysis. The effect on the spread of the distribution, as reflected by the interval between its 5th and 95th percentiles for example, will be a predictable widening.

There is a large and growing body of literature on the use of expert judgment in risk assessment and quantitative policy analysis (Morgan et al. 1990; Cooke 1991). It has been applied in fields such as climate change (Reilly et al. 2001). There are several applications in health risk assessment (North and Merkhofer 1976; Morgan et al. 1984; Evans et al.1994). In fact, as described above, OAQPS has been a pioneer in the application of these approaches to estimating the health risks due to exposure to air pollutants (Richmond 1981; Feagans and Biller 1981; Whitfield et al. 1991; Rosenbaum et al. 1995). These approaches have also been used in cases of residential radon cancer risks (Krewski et al. 1999) and for stratospheric ozone depletion (NRC 1979a,b).

As it incorporates additional sources of uncertainty into its primary

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

health benefits analyses, EPA should consider conducting analyses to determine which uncertainty sources have the greatest influence on the final results. Those impacts should be measured not only on the mean but also on the spread of the health benefit probability distribution. The sources that have the greatest influence on the spread of the distribution of a health benefit should be given priority for future research. Value-of-information analysis, a branch of statistical decision analysis, provides a well-structured approach for estimating the decision-making benefits of information that might be expected to flow from various research strategies (Raiffa 1970; Lindley 1985).

EPA should also consider conducting analyses to determine the sensitivity of the final results to the specification of reasonable alternative probability distributions for the uncertainty sources in the primary analyses. The need for sensitivity analyses will be particularly great for distributions that are based solely or largely on expert judgment.

EPA should consider comparing predictions from health benefits analysis models with subsequent observations that were not used in deriving or calibrating the models. Ideally, the subsequent observations and comparisons should be made by researchers who are independent from the authors of the original model and the investigators whose observations were used to derive and calibrate it. The results of the comparisons should be presented in future health benefits analyses and used to assess, quantify, and reduce uncertainties in the resulting estimates.

As it begins in the transition to incorporate additional sources of uncertainty into its primary health benefits analyses, EPA should continue the sensitivity analyses it has traditionally conducted. These analyses should be expanded, however, to include more than one source of uncertainty at a time. For example, if EPA were to include three additional uncertainty sources into its primary analysis of a health benefit, it might also conduct a traditional sensitivity analysis of these three sources jointly. With three illustrative scenarios for each component, for example, this expansion of the traditional sensitivity analysis would produce mean health benefits estimates for all 27 possible combinations of the scenarios. EPA then would be able to refer to the probability assigned to these combinations in the primary analysis to reflect their varying degrees of plausibility.

EPA should consider distinguishing between the uncertainties that arise from difficulties in projecting the future and the uncertainties inherent in estimating health benefits in current populations on the basis of hypothetical

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

changes in current levels of emissions. By providing a preliminary analysis that estimates in current populations the health benefits resulting from hypothetical changes in current levels of emissions, EPA might develop an idea of the lower bound on the uncertainty in any prediction of consequences projected into the distant future. There would be fewer uncertainties in these preliminary analyses than in analyses of the impacts of proposed regulatory actions on future exposures and health outcomes.

EPA should continue to strive to present the results of its health benefits analyses in ways that avoid conveying an unwarranted degree of certainty. These alternative approaches should include rounding to fewer significant digits. For example, the mean of the Tier 2 distribution for avoided mortality could have been reported as 4,300 or 4,000 avoided deaths rather than 4,307. Another need is to place less emphasis on a single value, such as the mean of a health benefit probability distribution, and more on ranges, such as the interval between the distribution’s 5th and 95th percentiles. It would also be helpful to increase the use of graphs to display health benefits probability distributions in their entirety. Graphs will be especially helpful as the incorporation of additional uncertainties results in asymmetrical health benefit probability distributions (Read and Morgan 1998).

In presenting a probability distribution for each health benefit produced by a primary analysis, EPA should emphasize even more than it has in the past the sources of uncertainty that remain unaccounted for in the primary analysis. Along with depicting the uncertainty in its primary health benefits analyses more realistically, EPA should foster a discussion in which it rebuts explicitly the misperception that such analyses would not be “useful.” That view comes from a mistaken belief that a very high degree of certainty is required before regulatory action can be considered warranted to protect the environment and the public health. The result is needless pressure to make the scientific basis for that regulatory action appear more certain than it is. A more defensible position is that decision-makers can make much better decisions when provided with realistic assessments of the nature and extent of the uncertainty that is present. The correct mix of action and research is a policy decision that can be informed by a full appraisal of the sources, nature, and extent of uncertainty.

The committee recommends that formally elicited expert judgment be used in the characterization of uncertainty in estimates of health benefits, although the committee recognizes that a number of issues must be addressed to use this approach responsibly. However, the committee be-

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

lieves approaches that explicitly incorporate judgmental probabilities into EPA estimates of health benefits are preferable to ones that fail to characterize the degree and key sources of uncertainty in estimates of health benefits from regulatory action. Furthermore, the committee recommends that EPA formally acknowledge those experts from whom it elicited judgments on uncertainty issues in the health benefits analysis.

CONCLUSIONS

  • In its primary analyses of health benefits, EPA reports the uncertainty as a probability distribution. Only one source of uncertainty, the random sampling variability of the estimated concentration-response function, is given with an emphasis on the mean of the probability distribution. The absence of other sources of uncertainty makes the results of the primary analyses appear more certain than they are.

  • To address other sources of uncertainty, EPA uses ancillary analyses, such as alternative and supplementary calculations and sensitivity analyses. With the exception of concentration-response function estimates, these ancillary analyses usually examine only one source of uncertainty at a time and only for the impact on the mean value of the probability distribution from the primary analysis. As a consequence, though laudable steps in the right direction, these ancillary analyses do not adequately convey the relative or aggregate degree of uncertainty created by the sources of uncertainty addressed in the analyses, nor, of course, do they depict uncertainty from other sources.

RECOMMENDATIONS

  • EPA should begin to move the assessment of uncertainties from its ancillary analyses to its primary analyses. This shift will require the specification of a probability distribution for each uncertainty source that is added to the primary analysis and, as necessary, the specification of joint distributions for the uncertainty sources that are not independent of each other. Expert judgment, as well as data, will be required to specify these distributions. Although the effect on the mean of the resulting probability distribution might increase, decrease, or remain the same, the effect on the spread

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

of the distribution will be a predictable widening and, therefore, a more realistic depiction of the overall uncertainty in the analysis.

  • As it incorporates additional sources of uncertainty into its primary health benefits analyses, EPA should consider conducting analyses to determine which uncertainty sources have the greatest influence on the mean and spread of the probability distribution. The need for these sensitivity analyses will be particularly great for distributions that are based on expert judgment. The uncertainty sources that have the greatest consequences for decision-making, including those that have the greatest impact on the spread of the distribution, should be given high priority for additional research.

  • Because the incorporation of expert judgment when data are unavailable will influence the estimates of health benefits as well as the uncertainty analyses, the committee also recommends that EPA clearly distinguish between data-derived estimates of some components—such as the concentration-response function—and expert opinions about other components that are lacking in scientific data—such as the degree of compliance with a particular regulation 30 years into the future. In this way, policy-makers will better understand how existing data and expert judgment combine to produce estimates and where new data would be most valuable.

  • As EPA begins the transition to incorporate additional sources of uncertainty into its primary health benefits analyses, it should continue the sensitivity analyses it has traditionally conducted. These analyses should be expanded, however, to consider sources of uncertainty jointly rather than singly.

  • In presenting the probability distribution for each health benefit produced by a primary analysis, EPA should emphasize even more than it has in the past the sources of uncertainty that remain unaccounted for in the primary analysis. These uncertainties should continue to be described as completely and realistically as possible.

  • EPA should consider providing a preliminary analysis that estimates in current populations the health benefits resulting from hypothetical changes in current levels of emissions. These preliminary analyses would help EPA develop an idea of the lower bound on the uncertainty of future consequences and would have fewer uncertainties than analyses of the impacts of proposed regulatory actions on future exposures and health outcomes.

  • EPA should continue to strive to present the results of its health benefits analyses in ways that avoid conveying an unwarranted degree of

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

certainty. Such ways include rounding to fewer significant digits, increasing the use of graphs, presenting projected baseline along with projected health benefits, and placing less emphasis on single numbers (for example, the mean of the probability distribution for a health benefit) and greater emphasis on ranges (for example, the range between 5th and 95th percentiles of the distribution).

  • There is a common misperception that a high degree of certainty is required for regulatory actions to take place to protect public health. As a result, primary health benefits analyses that more fully and accurately portray the uncertainties might not be considered useful. It is unrealistic for EPA to defer decisions until it can make them on the basis of perfect science. A careful and deliberate balancing of the benefits and costs is required, and this balancing must be informed by a fair assessment of the current levels of uncertainty and a realistic evaluation of the likely reductions in uncertainty attainable through further research.

  • EPA should perform similar detailed analyses of uncertainty in the valuation of health benefits and in the regulatory cost analyses that the committee recommends for the health benefits analyses.

REFERENCES

CASAC (Clean Air Scientific Advisory Committee). 1986a. Letter to Craig Potter, Assistant Administrator for Air and Radiation, U.S. Environmental Protection, Washington, DC, from Morton Lippmann, Chairman, Clean Air Scientific Advisory Committee. SAB-CASAC-86-024. August 29, 1986.

CASAC (Clean Air Scientific Advisory Committee). 1986b. Letter to Lee Thomas, Administrator, U.S. Environmental Protection, Washington, DC, from Morton Lippmann, Chairman, Clean Air Scientific Advisory Committee. SAB-CASAC-86-023. August 29, 1986.

Cooke, R.M. 1991. Experts in Uncertainty: Opinion and Subjective Probability in Science. New York: Oxford University Press.


Dockery, D.W., C.A. Pope, X. Xu, J.D. Spengler, J.H. Ware, M.E. Fay, B.G. Ferris, and F.E. Speizer. 1993. An association between air pollution and mortality in six U.S. cities. N. Engl. J. Med. 329(24):1753-1759.

Dominici, F., A. McDermott, S. Zeger, and J. Samet. 2002. On the use of generalized additive models in time-series studies of air pollution and health . Am. J. Epidemiol. 156(3):193-203.


EPA (U.S. Environmental Protection Agency). 1978. A Method for Assessing the Health Risks Associated with Alternative Air Quality Standards for Ozone.

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

Strategies and Air Standards Division, Office of Air Quality Planning and Standards, U.S. Environmental Protection Agency, Research Triangle Park, NC. July 1978.

EPA (U.S. Environmental Protection Agency). 1997. Policy for Use of Probabilistic Analysis in Risk Assessment at the U.S. Environmental Protection Agency. Office of Research and Development, U.S. Environmental Protection Agency. May 15, 1997. [Online]. Available: http://www.epa.gov/ncea/mcpolicy.htm [accessed September 10, 2002].

EPA (U.S. Environmental Protection Agency). 1999a. Final Tier 2 Rule: Air Quality Estimation, Selected Health and Welfare Benefits Methods, and Benefit Analysis Results. EPA420-R-99-032. Office of Air Quality Planning and Standards, U.S. Environmental Protection Agency, Research Triangle Park, NC. December 1999. [Online]. Available: http://www.epa.gov/otaq/regs/ld-hwy/tier-2/frm/tsd/r99032.pdf [accessed September 10, 2002].

EPA (U.S. Environmental Protection Agency). 1999b. Final Report to Congress on Benefits and Costs of the Clean Air Act, 1990 to 2010. EPA 410-R-99-001. Office of Air and Radiation, U.S. Environmental Protection Agency. November 1999.

Evans, J.S., J.D. Graham, G.M. Gray, and R.L. Sielken. 1994. A distributional approach to characterizing low-dose cancer risk. Risk Anal. 14(1):25-34.

Feagans, T.B., and W.F. Biller. 1981. Risk assessment: Describing the protection provided by ambient air quality standards. Environ. Prof. 3(3-4):235-247.


Greenland, S. 1990. Randomization, statistics, and causal inference. Epidemiology 1:421-429.

Greenland, S. 1998. Basic methods for sensitivity analysis and external adjustment. Pp. 343-357 in Modern Epidemiology, 2nd Ed., K.J. Rothman, and S. Greenland, eds. Philadelphia, PA: Lippincott-Raven.

Greenland S., K.B. Michels, J.M. Robins, C. Poole, and W.C. Willett. 1999. Statistical uncertainty in trends and dose-response relations. Am. J. Epidemiol. 149(12):1077-1086.

Greenland, S. 2001. Sensitivity analysis, Monte Carlo risk analysis, and Bayesian uncertainty assessment. Risk Anal. 21(4):579-584.


Hacking, I. 1984. The Emergence of Probability: A Philosophical Study of Early Ideas About Probability, Induction and Statistical Inference. Cambridge: Cambridge University Press.

Hattis, D., and E.L. Anderson. 1999. What should be the implications of uncertainty, variability, and inherent “biases”/”conservatism” for risk management decision making? Risk Anal. 19(1):95-107.

Hattis, D., and D.E. Burmaster. 1994. Assessment of variability and uncertainty distributions for practical risk analyses. Risk Anal. 14(5):713-730.

Hattis, D., A. Russ, R. Goble, P. Banati, and M Chu. 2001. Human interindividual variability in susceptibility to airborne particles. Risk Anal. 21(4):585-599.

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

Holland, M., D. Forster, and M. Wenborn. 1999. Economic Evaluation of Proposals Under the UNECE Multi-Effects and Multi-Pollutant Protocol. AEAT-4587. Prepared by AEA Technology, Oxfordshire, U.K., for European Commission, DGXI, Brussels and Luxembourg. January 1999.

Krewski, D., S.N. Rai, J.M. Zielinski, and P.K. Hopke. 1999. Characterization of uncertainty and variability in residential radon cancer risks. Ann. N.Y. Acad. Sci. 895:245-272.


Lang, C.G., G. Yarwood, F. Lalonde, and R. Bloxam. 1995. Environmental and Health Benefits of Cleaner Vehicles and Fuels. Prepared for Canadian Council of Ministers of the Environment Task Force on Cleaner Vehicles and Fuels, Winnipeg, Manitoba. October 1995.

Lindley, D.V. 2000. The philosophy of statistics (with discussion). Statistician 49:293-337.

Lindley, D.V. 1985. Making Decisions, 2nd Ed. London: John Wiley and Sons.


McCurdy, T., and H.M. Richmond. 1983. Description of the OAQPS Risk Program and the Ongoing Lead NAAQS Risk Assessment Project. Paper 83-74.1. Presented at the 76th Annual Meeting of the Air Pollution Control Association, June 19-24, 1983, Atlanta, GA.

Morgan, M.G., S.C. Morris, M. Henrion, D.A.L. Amaral, and W.R. Rish. 1984. Technical uncertainty in quantitative policy analysis—a sulfur air pollution example. Risk Anal. 4(3):201-213.

Morgan, M.G., M. Henrion, and M. Small. 1990. Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. Cambridge: Cambridge University Press.


NAE (National Academy of Engineering). 1972. Perspectives on Benefit-Risk Decision Making. Washington, DC: National Academy of Engineering.

NMMAPS (National Morbidity, Mortality, and Air Pollution Study). 2002. Updated NMMAPS results in the National Morbidity, Mortality, and Air Pollution Study. Johns Hopkins University, Bloomberg School of Public Health, Baltimore, MD. [Online]. Available: http://www.biostat.jhsph.edu/biostat/research/update.main.htm [accessed Sept. 9, 2002].

NRC (National Research Council). 1975. Air Quality and Stationary Source Emission Control: A Report. Serial No 94-4. Washington, DC: U.S. Government Printing Office.

NRC (National Research Council). 1979a. Protection against Depletion of Stratospheric Ozone by Chlorofluorocarbons. Washington, DC: National Academy Press.

NRC (National Research Council). 1979b. Stratospheric Ozone Depletion by Halocarbons: Chemistry and Transport. Washington, DC: National Academy Press.

NRC (National Research Council). 1982. Risk and Decision Making: Perspectives and Research. Washington, DC: National Academy Press.

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

NRC (National Research Council). 1983. Risk Assessment in the Federal Government: Managing the Process. Washington, DC: National Academy Press.

NRC (National Research Council). 1994. Science and Judgment in Risk Assessment. Washington, DC: National Academy Press.

NRC (National Research Council). 1996. Understanding Risk: Informing Decisions in a Democratic Society, P.C. Stern, and H.V. Fineberg, eds. Washington, DC: National Academy Press.

North, D.W. and M.W. Merkhofer. 1976. A methodology for analyzing emission control strategies. Comput. Oper. Res. 3:187-207.

Pate-Cornell, M.E. 1996. Uncertainties in risk analysis: Six levels of treatment. Reliab. Eng. Syst. Saf. 54(2-3):95-111.

Poole, C. 1988. Feelings and frequencies: Two kinds of probability in public health research. Am. J. Public Health 78(12):1531-1533.

Poole, C. 2001. Low P-values or narrow confidence intervals: Which are more durable? [Editorial]. Epidemiology 12(3):291-294.

Pope, C.A., III, M.J. Thun, M.M. Namboodiri, D.W. Dockery, J.S. Evans, F.E. Speizer, and C.W. Heath Jr. 1995. Particulate air pollution as a predictor of mortality in a prospective study of U.S. adults. Am. J. Respir. Crit. Care Med. 151(3 Pt 1):669-674.

Presidential/Congressional Commission on Risk Assessment and Risk Management. 1997. Framework for Environmental Health Risk Management: Final Report. Washington, DC: The Commission.


Raiffa, H. 1970. Decision Analysis: Introductory Lectures on Choices under Uncertainty. Reading, MA: Addison-Wesley.

Reilly, J., P.H. Stone, C.E. Forest, M.D. Webster, H.D. Jacoby, and R.G. Prinn. 2001. Climate Change. Uncertainty and climate change assessments. Science 293(5529):430-433.

Richmond, H.M. 1981. A framework for assessment of health risks associated with national ambient air quality standards. Environ. Prof. 3(3-4):225-234.

Read, D., and M.G. Morgan. 1998. The efficacy of different methods for informing the public about the range dependency of magnetic fields from high voltage power lines. Risk Anal. 18(5):603-610.

Rosenbaum, A.S., R.L. Winkler, T.S. Wallsten, R.G. Whitfield, and H.M. Richmond. 1995. An assessment of the risk of chronic lung injury attributable to long-term ozone exposure. Operations Research 43(1):19-28.


SAB (Science Advisory Board). 1979. Review of A Method of Assessing the Health Risks Associated with Alternative Air Quality Standards for Ozone, Report of the Subcommittee on Health Risk Assessment. EPA/SAB/79/001. Science Advisory Board, U.S. Environmental Protection Agency, Washington, DC. September 1979.

SAB (Science Advisory Board). 1980. Approaches to Health Risk Assessment for Alternative National Ambient Air Quality Standards, Report of the Subcommit

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×

tee on Health Risk Assessment. EPA/SAB/80/003. Science Advisory Board, U.S. Environmental Protection Agency, Washington, DC. December 1980.

Seigneur, C., E. Constantinou, and T. Permutt. 1994. Uncertainty analysis of health risk estimates, Appendix F. Pp. 453-478 in Science and Judgment in Risk Assessment. Washington, DC: National Academy Press.

Whitfield, R.G., T.S. Wallsten, R.L. Winkler, H.M. Richmond, and S.R. Hayes. 1991. Assessing the Risk of Chronic Lung Injury Attributable to Long-Term Ozone Exposure. Report ANL/EAIS-2. NTIS/DE91016814. Argonne National Laboratory, IL. July 1991.

Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 126
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 127
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 128
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 129
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 130
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 131
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 132
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 133
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 134
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 135
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 136
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 137
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 138
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 139
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 140
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 141
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 142
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 143
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 144
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 145
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 146
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 147
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 148
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 149
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 150
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 151
Suggested Citation:"5 Uncertainty." National Research Council. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC: The National Academies Press. doi: 10.17226/10511.
×
Page 152
Next: 6 Using, Presenting, and Reviewing Health Benefits Analyses »
Estimating the Public Health Benefits of Proposed Air Pollution Regulations Get This Book
×
Buy Paperback | $55.00 Buy Ebook | $44.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

EPA estimates that thousands of premature deaths and cases of illnesses may be avoided by reducing air pollution. At the request of Congress, this report reviews the scientific basis of EPA's methods used in estimating the public health benefits from its air pollution regulations.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!