In Chapter 1 the committee specified three factors that affect decisions made by the U.S. Environmental Protection Agency (EPA): estimates of human health risks, technology availability, and economics (see Figure 1-2). As outlined in Chapter 1, the legal context within which a decision is made determines, to a large extent, which of those factors are considered by EPA. EPA’s analyses of uncertainty have traditionally focused on the uncertainties in human health risk estimates (discussed in Chapter 2), but uncertainties in technological and economic factors can affect EPA’s decisions. In this chapter the committee discusses the uncertainty in those factors. Although these three factors are not independent—for example, the estimates of human health risks influence the economic analysis, and technology availability contributes to economic analyses—the committee discusses them separately.
In addition to those three factors, other factors, including the political climate and social factors such as environmental justice and public sentiment, can also affect EPA’s decisions. Although there is uncertainty about such factors, that uncertainty is difficult, if not impossible, to quantify. Those factors, therefore, are discussed separately at the end of this chapter.
Congress recognizes that technological considerations—including the feasibility, impacts, and range of risk-management options—are key to
many of EPA’s regulatory issues and decisions, and many of the statutes that grant EPA its regulatory authority require the evaluation of the technology available to implement a proposed regulation. Exactly how Congress directs EPA to consider technological factors differs from statute to statute and, in some cases, from section to section in a single statute (see Box 3-1 for a description of different technology standards). Statutes typically require a consideration of the technologies likely to be feasible within a given time frame, and some require engineering costs (that is, the costs of purchasing, installing, and operating the technologies to meet the standard) to be considered when assessing feasibility.1 EPA typically considers—and court decisions have supported that consideration2—“technologies that are currently available” to include technologies that can reasonably be anticipated to be developed in the future.
The section of the Clean Air Act (CAA) related to mobile emission sources requires EPA to evaluate the “availability of technology, including costs”3 when considering revised tailpipe emission standards (Section 202), but to evaluate the “best technology that can reasonably be anticipated to be available at the time such measures are to be implemented”4 when considering standards for urban buses (Section 219). When EPA is deciding whether to list a pollutant as a hazardous air pollutant (HAP) under the CAA, it is only allowed to consider “adverse effects on human health or adverse environmental effects,”5 not cost or technical feasibility. When it sets standards for HAPs under the CAA, however, in addition to health risk assessments, it is required to take the maximum achievable control technology into account. The CAA also requires EPA to establish National Ambient Air Quality Standards solely on the basis of health risks, and it is up to individual states to set implementation plans to achieve those standards.
Under the Clean Water Act (CWA), EPA is required to develop effluent standards for sources of water pollution based on the “best practicable control technology currently available”6 (BPT), or the “best available technology economically achievable”7 (BAT). The CWA establishes the framework for determining the BAT for conventional water pollution.8 In general, BAT effluent limits represent the best available economically achievable
1 The initial estimation of engineering costs could be considered part of the technology assessment or part of the benefit–cost assessment. Regardless, those costs will be included in the economic analysis in the regulatory impact assessment.
2 See, for example, Portland Cement Assn. v. Ruckelshaus, 486 F.2s 375 (D.C. Circ. 1973).
3 42 U.S.C. § 7521 (i)(2)(A)(i) (2012).
4 42 U.S.C. § 7554 (a) (2012).
5 42 U.S.C. § 7412 (b)(2)(B) (2012).
6 33 U.S.C. § 1311 (b)(1)(A) (2010).
7 33 U.S.C. § 1314 (b)(1)(B) (2010).
8 CWA, Pub. L. No. 107-377, Section 304(b)(2) (1972).
Best Practicable Control Technology Currently Available
“BPT is defined at Section 304(b)(1) of the [Clean Water Act (CWA)]. EPA sets Best Practicable Control Technology Currently Available (BPT) effluent limitations for conventional, toxic, and non-conventional pollutants. Section 304(a)(4) designates the following as conventional pollutants: biochemical oxygen demand (BOD5), total suspended solids, fecal coliform, pH, and any additional pollutants defined by the Administrator as conventional. The Administrator designated oil and grease as an additional conventional pollutant on July 30, 1979 (see 44 FR 44501).”
Best Conventional Pollutant Control Technology
“Best Conventional Pollutant Control Technology (BCT) is defined at Section 304(b)(4) of the CWA. The 1977 amendments to the CWA required EPA to identify effluent reduction levels for conventional pollutants associated with BCT for discharges from existing industrial point sources. In addition to the other factors specified in section 304(b)(4)(B), the CWA requires that EPA establish BCT limitations after consideration of a two part “cost-reasonableness” test. EPA explained its methodology for the development of BCT limitations in a Federal Register notice.”
Best Available Technology Economically Achievable
“Best Available Technology Economically Achievable (BAT) is defined at Section 304(b)(2) of the CWA. In general, Best Available Technology Economically Achievable (BAT) represents the best available economically achievable performance of plants in the industrial subcategory or category. The factors considered in assessing BAT include the cost of achieving BAT effluent reductions, the age of equipment and facilities involved, the process employed, potential process changes, non-water quality environmental impacts, including energy requirements and other such factors as the EPA Administrator deems appropriate. EPA retains considerable discretion in assigning the weight according to these factors. BAT limitations may be based on effluent reductions attainable through changes in a facility’s processes and operations. Where existing performance is uniformly inadequate, BAT may reflect a higher level of performance than is currently being achieved within a particular subcategory based on technology transferred from a different subcategory or category. BAT may be based upon process changes or internal controls, even when these technologies are not common industry practice.”
SOURCE: EPA, 2012.
performance of plants in the industrial subcategory or category. If the variability of technologies within a category is too large, EPA can subdivide an industrial sector into more narrowly defined categories in order to examine available technologies in a more granular way.
Under the Safe Drinking Water Act (SDWA),9 EPA must determine a nonenforceable maximum contaminant level goal (MCLG) for contaminants solely on the basis of human health risks.10 In contrast, the enforceable drinking-water standard promulgated under the SDWA—termed the maximum contaminant level (MCL)—is the “level that may be achieved with the use of the best available technology, treatment techniques, and other means that EPA finds are available (after examination for efficiency under field conditions, not solely under laboratory conditions), taking cost into consideration.”11 Thus, EPA must consider technological and economic factors when setting the enforceable drinking water standards (the MCLs) but not the MCLGs.
Uncertainties in Technology Assessments
There is inherent uncertainty in the analyses of both current and future control technologies. When assessing current technologies to establish a BAT, for example, EPA must consider such parameters as the cost of achieving BAT effluent reductions, the age of the equipment and facilities involved, the process employed, potential process changes, and non-water-quality environmental impacts. There may be only limited data available about any or all of those parameters; for instance, EPA might have the facility age for only a subset of facilities within a sector or subcategory of a sector. There can also be variability in those parameters within a sector or subcategory of a sector, which contributes to the uncertainty in EPA’s decisions. Using current technology for rulemaking, however, ignores that uncertainty, and it can lead to an underestimate of the level of control technologies that could be implemented. For example, mobile source control technology has steadily improved over the past 30 years. A 2005 review of technology innovation included information about how the technologies for catalyst and fuels preparation continued to improve over time with progressively more stringent tail pipe emission standards (ICF Consulting, 2005).
9 SDWA, Pub. L. No. 93-523.
10 When the statutory framework requires that decisions be made solely on the basis of health effects, the focus is likely to be on protecting maximally exposed or sensitive individuals. For such decisions there is no opportunity for other factors (for example, cost of mitigation, loss to property, loss of employment, and social factors) to influence the selection of management options.
11 SDWA, Pub. L. No. 93-523.
The challenge is even greater when trying to predict what technologies might be available or in widespread use in the future and also to predict the effectiveness and costs of those technologies. For those predictions, analysts not only must estimate what technologies are currently available and the costs and amounts of emission reductions associated with those technologies, but also must model or somehow predict future developments in control technologies. The rate of innovation varies among sectors (Pavitt, 1984). An additional challenge is that past innovation curves might not reflect future rates of innovation. As businesses begin to implement technologies, there is a learning force at play that changes the rate of innovation. A regulation itself can also lead to an increase in the market size anticipated for a given technology, which provides a stimulus for investment in research and development for the technology. In other words, having regulations in place or on the horizon can lead to innovation in control technologies. In some sectors different businesses or entities will compete based on how efficiently they achieve a performance standard. This stimulates innovation and, as discussed later in this chapter, some analyses suggest that the innovation leads to costs that turn out to be less than estimated at the time of rulemaking (Morgenstern, 1997, 2011). Because there has been little study of innovation rates and the effects that EPA’s regulations have on those innovation rates, there are insufficient data with which to develop models that account for those effects. Learning how technologies develop over time and how EPA’s regulatory decisions affect that development could help improve the agency’s models of technological factors and decrease the uncertainty in agency decisions.
EPA sometimes has the statutory authority, or the precedent, to consider both available technologies and technologies that are anticipated to be available. There are greater inherent unknowns associated with anticipating technologies—which in some cases could be considered deep uncertainty. In some cases, however, the agency will select a current technology but build into the rule a review or evaluation at a future date to update the rule as technology advances (see Chapter 2 for more discussion). An alternative to this approach is the development of performance standards rather than specific technologies.
Decision in the Face of Technology Uncertainties: EPA’s Highway Heavy-Duty Engine Rule
In 1996 EPA was faced with uncertainties surrounding the technologies available to control emissions from highway heavy-duty engines. When considering regulations to control emissions of air pollution from highway heavy-duty engines (Control of emissions of air pollution from highway heavy-duty engines, 1996), EPA recognized that high concentrations of
sulfur in diesel fuel led to corrosion caused by emission-control technologies that employ exhaust gas recirculation and that these sulfur concentrations were a major limiting factor to implementing emission-control technology. In its rulemaking EPA was faced with uncertainty about when and by what extent the concentrations of sulfur in fuel would decrease and about how low the concentration of sulfur would have to be in order not to affect emission-control technologies adversely.
Given those uncertainties, EPA took a more adaptive strategy to rulemaking. Rather than establishing a standard for emissions immediately, it established a timeline for future implementation of a standard and for evaluation of the effects of sulfur on control technologies in the interim. Specifically, in its rule EPA stated that “fuel changes could reduce the amount of emission control necessary for the engine, but … are probably not necessary to meet the proposed standards. However, this remains an area of uncertainty and is one of the issues which would be addressed further in the proposed 1999 review of the feasibility of the standard” (Control of emissions of air pollution from highway heavy-duty engines, 1996, p. 33455). In other words, EPA identified the potential effects of changing fuel composition as an uncertainty in the sense that it was not known to what extent decreasing the sulfur content and other changes in diesel fuel would affect the ability meet proposed emission standards, and the agency indicated that it would further evaluate those effects in a 1999 review of the standard. EPA’s subsequent July 2000 document, Regulatory Impact Analysis: Control of Emissions of Air Pollution from Highway Heavy-Duty Engines (EPA, 2000c), reported that much lower concentrations of sulfur dioxide (a maximum of 30 ppm compared to then concentrations up to 500 ppm) would limit the corrosion caused by exhaust gas recirculation emission-control technologies.
In a separate rule EPA later required a significant reduction in the amount of sulfur in diesel fuels beginning with the 2007 model year. During the decade between when the diesel engine standards were set in the 1990s and when the standards took effect in 2007, EPA developed companion regulations to change fuel composition and implemented demonstration programs, while engine manufacturers developed innovative emission controls as they were preparing for the regulatory change. EPA was faced with an uncertainty about how sulfur content affects emission controls and, therefore, allowed the industry time for additional research, development, and commercialization of control technologies before having a regulation become effective. Such an approach allowed for the evaluation of recent technological innovations and updated standards that reflected those innovations. Scheduling such reviews of standards several years in the future can also motivate research and development.
As discussed above, a number of the statutes and executive orders12 under which EPA operates require it to consider economics and economic analyses in its regulatory decisions. As described in EPA’s Guidelines for Preparing Economic Analysis (EPA, 2010), economic analyses combine various types of information, including information from the assessments of other factors discussed in this and the previous chapter, to provide “a means to organize information and to comprehensively assess alternative actions and their consequences” (pp. 1–2). The analyses inform decision makers of the costs associated with the various risks, the benefits of reducing those risks, the costs associated with risk mitigation or remediation options, and the distributional effects (see Box 3-2 for EPA’s definitions related to economic analysis). As with the other factors that affect EPA’s decisions, economic analyses have uncertainties. Those uncertainties contribute to the overall uncertainty in a decision, and EPA should consider them its decision-making process.
This section discusses economic analysis and its uncertainty. The section begins with a brief overview of economic analysis in the regulatory setting. A number of texts and reports describe the use of economics and economic analyses in decision making in general (Gold et al., 1996; IOM, 2006; Sloan and Hsieh, 2012) and in environmental decision making in particular (Atkinson and Mourato, 2008; EPA, 2010; Pearce et al., 2006); the reader is referred to those sources for more detailed discussions. The committee then describes the uncertainties associated with those analyses, using examples to illustrate how EPA has evaluated and characterized uncertainty in economic analyses, followed by a discussion of the assessment of those uncertainties and reporting of those uncertainties.
Economic Analysis Approaches
Two of the main types of economic analysis are benefit–cost analysis (BCA; also called cost–benefit analysis) and cost-effectiveness analyses (CEA) (see Box 3-2 for definitions). In the context of environmental
12 For example, Executive Order 12866 requires analyses of the costs and benefits of “significant regulatory actions.” A “significant regulatory action” is defined as “any regulatory action that is likely to result in a rule that may: (1) Have an annual effect on the economy of $100 million or more or adversely affect in a material way the economy, a sector of the economy, productivity, competition, jobs, the environment, public health or safety, or State, local, or tribal governments or communities; (2) Create a serious inconsistency or otherwise interfere with an action taken or planned by another agency; (3) Materially alter the budgetary impact of entitlements, grants, user fees, or loan programs or the rights and obligations of recipients thereof; or (4) Raise novel legal or policy issues arising out of legal mandates, the President’s priorities, or the principles set forth in this Executive order.”
Benefit–cost analysis (BCA)
A BCA evaluates the favorable effects of policy actions and the associated opportunity costs of those actions. It answers the question of whether the benefits are sufficient for the gainers to potentially compensate the losers, leaving everyone at least as well off as before the policy. The calculation of net benefits helps ascertain the economic efficiency of a regulation.
A benefit–cost ratio is the ratio of the net present value (NPV) of benefits associated with a project or proposal relative to the NPV of the costs of the project or proposal. The ratio indicates the benefits expected for each dollar of costs. Note that this ratio is not an indicator of the magnitude of net benefits. Two projects with the same benefit–cost ratio can have vastly different estimates of benefits and costs.
Benefits are the favorable effects society gains due to a policy or action. Economists define benefits by focusing on changes in individual well-being, referred to as welfare or utility. Willingness to pay is the preferred measure of these changes, as it theoretically provides a full accounting of individual preferences across trade-offs between income and the favorable effects.
Cost-effectiveness analysis (CEA)
CEA examines the costs associated with obtaining an additional unit of an environmental outcome. It is designed to identify the least expensive way of achieving a given environmental quality target or the way of achieving the greatest improvement in some environmental target for a given expenditure of resources.
regulations, the overall objective of both BCA and CEA is to compare different regulatory options,13 different combinations of regulatory options, or the value of different regulatory options.14 In BCA both benefits and costs are expressed in monetary units, whereas CEA is intended to identify
13 Potential regulatory options include the option of taking no regulatory action.
14 That comparison, even in cost-effectiveness analysis, requires characterizing value in a common unit; one convenient measure of common value is money (Gold et al., 1996; Sloan and Hsieh, 2012).
Costs are the dollar values of resources needed to produce a good or service; once allocated, these resources are not available for use elsewhere. Private costs are the costs that the buyer of a good or service pays the seller. Social costs, also called externalities, are the costs that people other than the buyers are forced to pay, often through nonpecuniary means, as a result of a transaction. The bearers of social costs can be either particular individuals or society at large.
Distributional analysis assesses changes in social welfare by examining the effects of a regulation across different subpopulations and entities. Two types of distributional analyses are the economic impact analysis (EIA) and the equity assessment.
Economic impact analysis (EIA)
An EIA examines the distribution of monetized effects of a policy, such as changes in industry profitability or in government revenues, as well as nonmonetized effects, such as increases in unemployment rates or numbers of plant closures.
From a regulatory standpoint, social cost represents the total burden a regulation will impose on the economy. It can be defined as the sum of all opportunity costs incurred as a result of the regulation. These opportunity costs consist of the value lost to society of all the goods and services that will not be produced and consumed if firms comply with the regulation and reallocate resources away from production activities and towards pollution abatement. To be complete, an estimate of social cost should include both the opportunity costs of current consumption that will be foregone as a result of the regulation and also the losses that may result if the regulation reduces capital investment and thus future consumption.
Total cost is defined as the sum of all costs associated with a given activity.
SOURCE: EPA, 2010.
the most effective use of resources without requiring the monetization of all benefits or costs.
In 2003 the Office of Management and Budget (OMB) issued guidance on the development of regulatory analyses and the use of BCA and CEA. In that guidance OMB states that “major rulemaking should be supported by both types of analysis wherever possible” (OMB, 2003, p. 9). In 2011 EPA issued guidelines for economic analyses (EPA, 2010). In contrast to the OMB guidance, EPA’s guidelines focus on conducting BCAs for economic
analyses, although they mention that CEA can be used to help identify the least costly approach to achieving a specific goal.
BCAs are used to determine whether the benefits of a particular regulatory option justify its costs. When making decisions using a strict benefit–cost rule, a regulatory agency will adopt only those projects or implement only those regulations for which the present value of the net benefit (benefit minus cost) is non-negative or the benefit–cost ratio (present value of benefits divided by present value of cost) is one or greater. Under that strict rule, all projects for which the net benefit is negative or the benefit-to-cost ratio is less than one would be rejected. Alternatively, BCA can be used to rank regulatory options by the size of their net benefits; regulators can then choose the option or options with the largest net benefit. That approach, however, places small-scale projects at a disadvantage relative to larger ones. To account for project size, some economic analysts prefer to use a benefit-to-cost (B/C) ratio; if the B/C ratio is greater than one, the project is accepted, and otherwise it is rejected.
Description of Uncertainties in Economic Analyses
Uncertainty in economic analyses can stem from determining what costs and benefits should be including in the analyses, in the estimates of the costs and benefits themselves, and in adjusting the costs and benefits to reflect that they will occur in the future (that is, discounting).
The outcome of an economic analysis can vary greatly depending on the boundaries of that analysis, that is, on what is included in the analysis (Meltzer, 1997). Analyses can include mental health care costs and other health care costs, costs from lost employment, and a variety of other costs, such as the costs from the increased domestic violence that results from lost employment. In its guidelines, EPA (2010) considers social cost—which is described as the “total burden a regulation will impose on the economy” and “the sum of all opportunity costs incurred as a result of the regulation” (pp. xiv–xv)—to be the most comprehensive and appropriate measure of cost for a BCA. “Opportunity costs consist of the value lost to society of all the goods and services that will not be produced and consumed if firms comply with the regulation and reallocate resources away from production activities and toward pollution abatement” (p. xv). Social cost is narrower than the “total cost,” which is considered in a number of regulatory impact analyses and includes costs beyond the social costs.
At the time of an economic analysis, the mean estimated values of anticipated benefits and costs and the corresponding net benefit or the benefit–cost ratio might indicate that a project is worth undertaking or that a rule is worth adopting, but after implementation the actual benefits and costs seen in retrospect can differ considerably from the estimated mean values
because of uncertainties at the time of the analysis. For example, if there is an unanticipated increase in energy prices after a decision, the actual cost of a decision could exceed the anticipated cost, and what was thought to be a project of positive net benefit may turn out to have a substantial negative net benefit. That uncertainty, however, is rarely detailed in a BCA or CEA, let alone in the rationale for the decisions that use the BCA or CEA (NRC, 2002a).
We discuss below the uncertainties associated with the estimates of costs—categorized broadly as compliance costs and the costs imposed across the entire economy—and benefits as well as the uncertainties related to discounting those costs and benefits.
Compliance costs are the costs incurred in complying with a proposed regulatory rule, and they include those costs incurred by parties complying with the regulations (for example, the costs to install emission-control technologies in an industrial facility). These compliance costs are borne by regulated entities, of which many, if not most, are in the private sector (EPA, 2010). Additional compliance costs include the government’s costs to monitor and enforce the rule and, more generally, activities to ensure compliance (Harrington et al., 1999). The increased costs borne by regulated entities and public agencies charged with enforcement may also be associated with increased costs borne by other private-sector entities and public agencies at different levels of government. An increase in enforcement effort will generally raise compliance rates and, subsequently, the costs borne by the regulated parties (Shimshack and Ward, 2005, 2008). Some research indicates, however, that the costs of “government administration of environmental statues and regulations” are “rarely considered in regulatory cost estimates” (Harrington et al., 1999, p. 9).
Measuring the capital costs and operating costs incurred by private parties in order to estimate compliance costs can be a difficult task for agencies with the responsibility of implementing a regulation. For example, under a cost-of-service regulation, an agency sets a price per unit of output and must gauge whether the regulated price it sets is sufficient to yield an adequate return to the regulated entity. At the same time, the agency must ensure that the price is not set so high that returns are excessive (Breyer, 1995).
Compliance costs are often estimated using engineering models, with the models often being based on expert opinions of the relationships between input use and outputs for a particular industry or application within
an industry. It is not always made clear, however, from what settings the estimates were drawn. There may often be substantial variability in compliance costs depending on the characteristics of the regulated entity, including its scale and the age of its plants and equipment. For example, a plant may be so old that it would be replaced anyway, and the newer plant’s design might already incorporate a technology that is consistent with the proposed rule. In other cases, the plant might be relatively new and at a much earlier point in the company’s capital replacement cycle. In still other cases, if the cost of compliance is too great, facilities may be closed. It is also possible that, unknown to the EPA, particular facilities may have already been targeted for closure by company management even without the arrival of the new environmental requirements. A lack of such information is a common problem in the area of regulation and adds to the total uncertainty. EPA typically conducts surveys of facilities to determine the different types of technologies that are in place (EPA, 1995, 2004b). Cost estimates are usually based on estimates of the changes that would be required in the different types of facilities to comply with the standards.
EPA’s decision documents rarely present a range of costs that represents the uncertainty in estimates of engineering costs,15 and often it is not clear what assumptions underlay the computation of those cost estimates. For example, the summary of a 2000 regulatory impact analysis for arsenic in drinking water included tables listing the monetized health benefits from avoided cases of bladder and lung cancers and containing estimated compliance costs (EPA, 2000b). The table with the monetized benefits contains lower and upper estimates of benefits, which were based on the lower and the upper estimates of bladder cases avoided. No estimates that took sources of uncertainty other than human health risks into account are displayed. Estimates of the costs were provided in the summary table for two discount rates (3 percent and 7 percent), for two different plant categories, and for four different maximum contaminant levels. No analysis of other factors was displayed. As a result, the estimates did not reflect the overall variability in the cost of complying with the rule. A first step in dealing with this source of uncertainty would have been greater transparency in how the estimates were derived.
In a report that includes separate evaluations by different authors of three of EPA’s regulatory impact analyses, Harrington et al. (2009) also cited a lack of consideration of the uncertainty in many parameters that affected the regulatory impact analysis.
The use of engineering models for estimating costs raises a number of other technical issues as well. Conceptually, the relevant compliance
15 Engineering costs include the costs of purchasing, installing, and operating the technologies to meet a standard.
costs are marginal costs—that is, the incremental costs of complying with the rule—rather than average costs. There are issues of how joint costs or products are treated in the determination of engineering cost estimates. For example, the removal of one type of contaminant may be much less costly if industries have already installed treatment processes for other contaminants; similarly, if a control technology will decrease the emissions of a number of pollutants, it is difficult to know what portion of the costs of installing and maintaining that control technology should be attributed to regulating just one of those pollutants. In such cases the marginal cost of removing the contaminant can be substantially overstated if the other pollution control activities are not considered. EPA often estimates marginal costs and accounts for spillover effects, such as joint costs, in its analyses (EPA, 2011b; NAPEE, 2008). For example, EPA uses a model that accounts for the control of multiple pollutants (sulfur oxides, nitrogen oxides, directly emitted particulate matter, and carbon dioxide) in its regulatory impact analysis for mercury (EPA, 2011b).
There is also likely to be uncertainty concerning the number of households, firms, or systems (for example, water systems)16 that may be affected by a rule and also concerning the methods that the regulated entities will use to comply with the rule. Uncertainty is even greater when EPA sets a national standard and agencies at a lower level of government, such as state agencies, implement the rule. In such instances, in addition to the issue of how firms will actually change to meet the new standards, there is additional uncertainty concerning how other units of government will implement the new standard. Once again, however, systematic inaccuracy is unlikely to occur, except in those cases in which a problem with compliance is known or anticipated.
Other sources of uncertainty that are sometimes relevant are the level of enforcement, the productivity of such enforcement efforts, and, subsequently, the compliance with the rule. Further increasing the uncertainty associated with compliance is the fact that in some cases lower levels of government enforce EPA’s regulations. The simplest approach for dealing with such uncertainty is to assume complete compliance—in other words, 100 percent enforcement. EPA’s guidelines recommend that when conducting regulatory impact analyses, analysts should, as a general rule, assume full compliance (100 percent) with EPA regulations (EPA, 2010). The guidelines recommend departure from using the “default” of full compliance only when there is sufficient data to calculate the true compliance rate (EPA, 2010). This level of enforcement may be higher than either the level that is socially optimal (that is, the one at which the marginal cost of
16 See, e.g., Federal Register, November 22, 2001, p. 47, regarding an estimate of water systems affected by a proposal rule.
enforcement equals the marginal benefit of enforcement) or the level that occurs in practice. Estimates made with this assumption, therefore, should be considered to be high estimates of enforcement cost. Alternatively, EPA could estimate a range of costs using different percentages of firms complying with the laws. Whatever level of enforcement is assumed should be assumed throughout the analysis, including in the computation of benefits.
A few studies have compared the compliance costs estimated in regulatory impact analyses to estimates of actual compliance costs incurred after a regulation has been put into effect (Harrington, 2006; Harrington et al., 1999; OMB, 2005). Those comparisons indicate that compliance costs are often overestimated. OMB (2005) also concluded, however, that benefits are overestimated to a greater extent than costs, so that economic analyses typically predict that the performance of a regulation will be better than actually occurs. Harrington (2006) also found that although total costs were overestimated, unit costs were not, and he found “no bias in estimates of benefit–cost ratios.” Regardless of which analysis is more accurate, all of those analyses demonstrate the uncertainty that is inherent in predictions of compliance costs (and benefits). And as discussed by Harrington (2006), these experiences also demonstrate the importance of conducting analyses after the implementation of a regulation (so-called ex post analyses) to evaluate and improve the methods used for predicting costs and benefits.
When a rule is to be implemented over a number of years, additional uncertainties arise. For example, input prices (that is, the price of inputs to a process, such as the price of low-sulfur coal) might vary with time. Moreover, as discussed earlier, the costs of the technological changes and equipment necessary for a plant to comply with the rule might change. For example, the promulgation of a rule on a national basis might increase the market size for an innovation that improves environmental quality. Such new technologies may be more productive in achieving a particular environmental goal, and in some cases the purchase prices of equipment incorporating the new technologies might be lower. At the time the rule is being considered, however, there is considerable uncertainty about how innovators will respond to the rule (that is, the amount of investment in research and development that will be forthcoming in response to promulgation of the rule), the yield from such investment, the time frame within which any yield will occur (that is, when the innovation will occur), and at which price the new technology will be marketed. A practical solution is to assume a worst case in which no innovation takes place, but such an assumption might underestimate the net benefit of innovation to the extent that there is an overestimation of the costs.
Estimates of compliance costs often must be made by estimating the number of facilities currently not in compliance with a proposed standard or rule, the magnitude by which those facilities would be out of compliance,
and the current and future costs to bring those facilities into compliance under the new rule. All of those estimates are associated with an uncertainty that is difficult to accurately quantify. That uncertainty can, however, be qualitatively described, and potential ranges of costs can be used to provide decision makers with information on the effects of potential uncertainty on the estimates of the cost of different regulatory options.
Costs Imposed Economy Wide
The second broad category of costs consists of those costs imposed on other parties by increased prices (EPA, 2010). To the extent that prices increase, the quantities of output in other sectors are affected, which in turn affects the output for the economy as a whole (i.e., the gross domestic product). For example, if an environmental regulation increases the cost of coal mining, the price of coal is likely to increase, which in turn could lead to a decrease in coal consumption and an increase in the use of other energy sources. The increased price of coal would likely lead to an overall increase in energy costs, adding to the cost of manufacturing various products, which could in turn lower national production and employment.
When evaluating the broader costs of regulations, a distinction is often made between partial and general equilibrium analysis. Partial equilibrium analysis examines the effects of a regulatory change on a single firm. For example, a partial equilibrium analysis might assess the effect of a particular environmental regulation on the capital spending decisions of an individual firm. By contrast, general equilibrium analysis considers the effects of a regulatory change on all participants in a market or even in the economy as a whole. Individual sectors do not operate in a vacuum; if regulated firms increase the prices of their products, it may affect outputs (and prices) in other sectors. An analysis of those economy-wide effects, therefore, is often appropriate. Increased prices and reductions in output impose costs on society at large. However, if the potential effects of the regulatory rule are small or localized, there is little reason to assess its impacts on the economy as a whole.
It would be impractical to attempt to assess the economy-wide impacts of individual regulatory rules de novo. Instead it is necessary to employ models that have been developed for more general purposes. One useful tool that EPA has used for these purposes is a computable general equilibrium (CGE) model (EPA/RTI International, 2008). CGE models are a class of economic models that use actual economic data to estimate how an economy might react to changes in policy, technology, or other external factors. CGE models can also be used to compute the distributional impacts of regulatory changes. EPA has used a CGE model for a recent retrospective analysis of the benefits of the CAA (EPA, 2011a).
The starting point for an analysis with a CGE model is a set of assumptions about the impacts of the proposed rule on the output prices of the firms directly affected by the rule (RTI International, 2008). With the CGE model, the analyst computes prices and outputs of goods and services in various sectors and calculates the gross domestic product once the simulated economy has returned to a new equilibrium following implementation of the rule. CGE models are based on myriad assumptions about the underlying relationships among economic sectors—that is, about the substitutability of various goods and services in the economy. Those assumptions about these interrelationships, as well as the assumptions about price changes that are the essential inputs in the calculations, are sources of uncertainty in CGE models (RTI International, 2008).
There are also dynamic versions of CGE models that consider a broader range of longer-term effects, including technological changes, which have the potential to capture the long-term effects of regulatory rules on labor supply, savings, the growth of classes of inputs, and input productivity (RTI International, 2008). Structural changes in the economy occur over time as a result of a policy change. For example, a policy offering financial incentives to purchase energy-efficient appliances may lead to more demand for such appliances in the short run. In the long run, new appliances are developed because there is a greater financial incentive for firms to engage in research and development to develop new even more efficient projects. Because such long-term analyses rely on future projections, however, outcomes are far more uncertain than those obtained from use of static models.
One problem with CGE models is a lack of transparency. As discussed in the examples in Chapter 2, it is generally the case that few details are provided about either the baseline assumptions that were the starting points of the calculation or the key assumptions that went into development of the CGE model used for the economy-wide calculations. Thus it seems necessary for the decision maker to either accept the results at face value or to reject the exercise in its entirety.
To project the increase in benefits attributable to an intervention, one must have a measure relating inputs to endpoints. In economics, that measure is termed a production function. The production function expresses a technical or scientific relationship, and the analyst typically obtains the production function from the scientific literature. For example, the production function could describe the effect that removing a carcinogen from the water supply has on the rates of particular forms of cancer. Given estimates of the production function parameters, it is possible to calculate the changes
in health and other endpoints that are attributable to the intervention. Again, in a BCA the analyst must attach a pecuniary value to each endpoint.
Once the parameters of an assessment are established, estimating benefits requires establishing baseline values for the endpoints of interest, estimating the changes that would occur in those endpoints with different regulatory options (that is, the marginal effects of the policy or rule), and attaching a monetary value to a given endpoint (that is, valuing those endpoints). There are uncertainties inherent in each of these steps in a benefits assessment. An important source of uncertainty is the decision about which endpoints to include in the assessment, that is, determining the parameters the assessment will look at. Those three steps and the uncertainty associated with them are discussed below.
Establishing Baseline Values
The effect of a particular policy or rule on outcomes or endpoints of interest will depend, in part, on the original levels of what is being monitored. One exception is when the relationship between relevant effects of particular pollution levels is linear (EPA, 2010). Thus, the first step in calculating the benefit of a proposed rule is to calculate the values of a particular endpoint at a particular point in time when the change originates—that is, calculate the baseline benefits. For example, when estimating the benefits from a potential air pollution regulation, the rate of respiratory problems at a baseline point in time would be estimated. That point might be the time when the regulation is announced, if the industry is expected to reduce emissions in advance of the implementation of the law in order to prepare for the implementation, or at the time when the regulation is implemented if no changes in air pollution are anticipated in advance of implementation (NRC, 2002a).
Estimating the baseline involves a series of calculations, each with inherent uncertainties. The factors being measured could include human health as well as other factors with value to society, such as the preservation of specific species or habitats, atmospheric visibility, and pollution’s effects on recreational use of various resources (EPA, 2010). The effects on those factors depend, in part, on the magnitude of pollutants which, in turn, depend on various output levels (EPA, 2010). For example, the effect of car emission regulations depends in part on the magnitude of the emissions from automobiles, which would depend in part on the number of cars on the road and the number of miles driven, and both of those outputs would vary by geographic location. The baseline variables are subject to scientific uncertainty as well as to uncertainties in activity levels in various sectors, which in turn can depend on exogenous factors (EPA, 2010). The number of miles driven, for instance, can depend on the price of gasoline, demographic
changes, behavioral changes, and the existence of other regulatory rules and their levels of enforcement (EPA, 2010). In general, longer-range projections are subject to more uncertainty than shorter-range projections. Technological change can be a source of uncertainty in long-range projections (Moss and Schneider, 2000). For example, baseline air pollution is highly dependent on innovations in motor vehicle technology, such as the development of electric or hydrogen-powered automobiles. If baseline benefits are being estimated for a rule under consideration for automobile emissions and there are many years between when the proposed rule is developed and when it is implemented, uncertainty can come from trying to estimate improvements in the technology for electric cars and changes in consumer adoption of electric cars over time. Further complicating those estimates and creating more uncertainty is the fact that the proposed rule itself can encourage such innovation (EPA, 2010).
Multiple baselines can be used to indicate the range of potential baseline estimates, but they can make calculations very complex if all subsequent calculations need to account for multiple estimates of baseline benefits (EPA, 2010). Discussing baseline benefits with policy makers prior to estimating the baseline benefits can narrow the possible scenarios and decrease the number of baseline calculations that need to be performed. Many of the uncertainties associated with estimates of baseline benefits are deep uncertainties which will not be able to be resolved within the time-frame needed. In the face of such a high degree of uncertainty, transparency in the assumptions and analytic methods used to estimate baseline benefits is important in order to allow decision makers and stakeholders to understand how those benefits are estimated. If multiple baselines are not used, that fact should be clearly stated to make it clear that there is a source of uncertainty that is not being represented. Baseline estimates are the source of many errors in cost and benefit estimates (Harrington et al., 1999).
Marginal Effects of Policies and Rules
Once baseline benefits are estimated, the next step in estimating the overall benefits from a policy or rule is estimating the marginal effects of the policy or rule. In other words, one computes the anticipated changes from the baseline benefits that are attributable to the policy or rule in question and compares those estimates to estimates of the costs and benefits that would have occurred in the absence of the policy or rule. The same endpoints used in estimating the baseline benefits should be used to estimate the marginal effects of the rule.
The effects of a policy or rule on benefits are, in part, a function of the effects of the policy or rule on the exposures of a population to the harmful substance or substances being controlled by the policy or rule. Uncertainties
about the effects of the rule on the exposure of a population to a harmful substance or substances come from uncertainties about rule enforcement and compliance, about the dose–response relationship, and about the time path of the response.
The final step in evaluating benefits is to attach a monetary value to each endpoint. Those endpoints could be related to the effects that a regulation had on human health or on such things as access to a park or clean air or community health. The monetary value placed on these things reflects society’s maximum willingness to pay and is generally expressed per unit change in the endpoint—for example, how much society is willing to pay for each life-year saved or each day of hospitalization averted (EPA SAB, 2009; NRC, 2004). There can be large variability in how different endpoints are valued, which adds to the uncertainty in the economic analysis.
There are essentially two broad approaches for valuing the effects of policies: the revealed-preference approach and the stated-preference approach (Adamowicz et al., 1997; EPA SAB, 2009; Sloan and Hsieh, 2012; Williams, 1994). The advantages and disadvantages of those two methods are reviewed in detail elsewhere (EPA SAB, 2009; Freeman, 2003). The uncertainties in the approaches are discussed below.
Revealed-Preference Approach The revealed-preference approach bases valuations on actual decisions that people make, for example, the additional wage rate that compensates workers for taking a job with a higher fatality risk (Viscusi and Aldy, 2003). Revealed-preference studies are available for some, but by no means all, endpoints that need to be valued for environmental policy decisions (Boyd and Krupnick, 2009). Although there are many revealed preference–based studies of the value of life and the value of life-years, few analyses of morbidity and disability exist. Furthermore, many of the endpoints that are valued in benefit analyses for environmental policy decision making do not have inherent commercial or market value. For such endpoints, it is necessary to rely on valuations obtained from stated preference approaches (Yao and Kaval, 2011).
The major source of uncertainty is that there are few valid, reliable, and representative estimates of societal willingness to pay for reductions in morbidity and disability. Given the paucity of relevant studies, valuations tend to be based on a few studies of local populations that may not be nationally representative. It is not the existence of uncertainty in the valuation that is problematic, however, but rather it is when those uncertainties are ignored—that is, not quantified or somehow accounted for in decisions—that problems arise.
Furthermore, several potentially important dimensions of benefit are not valued at all. For example, in an economic analysis of a rule governing arsenic in drinking water, several factors that could influence the value of the benefits from removing arsenic from the drinking water—for example, not having to accept an involuntary risk of being exposed to arsenic in the water supply—are not currently given a value (EPA, 2000a). The Arsenic Benefits Review Panel of EPA’s Science Advisory Board (EPA, 2001), in its review of EPA’s economic analysis of the arsenic rule (EPA, 2000b), noted that “some people may value the existence of lower levels of arsenic in drinking water, possibly for psychological reasons (e.g., dread of being exposed)” (p. 3). That is, aside from the health risk of being exposed to arsenic, arsenic exposure has a variety of other costs. Having arsenic in the water supply can be a source of anxiety for those individuals directly exposed to it, for example, and the fact that some persons are exposed to arsenic in the water supply can be seen as unfair, a negative feature over and above any adverse health effects arsenic might have.
Stated-Preference Approach The stated-preference approach, also called contingent valuation, bases valuations on surveys designed to determine the willingness of a household to pay for a policy that will produce benefits for that household. Contingent valuation has been used to place value on nonmarket items, such as the worth of access to a park or of clean air or of community health. Many studies that use contingent valuation address ecological issues, such as the value of preserving bald eagles, wetlands, forests, or visibility at national parks (Breedlove, 1999), and they can identify priorities among various types of concerns, such as air quality; trash, illegal dumping, and abandoned housing; economic development; parks and surface water quality; and community health. The surveys are a mechanism for stakeholder input into the decision-making process and can be used when estimating the benefits of a regulatory decision related to human health. They are also a source of uncertainty in regulatory decisions, including from variability in the community’s views and a lack of knowledge about that variability, and uncertainty in the techniques used to assess those views.
EPA’s Science Advisory Board (SAB) has published a number of reports on ecological valuation methods, how EPA could apply those methods in its decision-making process, and the value that communities place on ecology. In 2006 the board held a workshop on ecological risk assessment and environmental decision making, and the following year it issued a report on that subject (EPA, 2007). The report demonstrates the recognition by the SAB that EPA needs a broader approach to ecological risk assessment and decision making. In particular, the SAB stated, “Local and regional regulatory processes are conditioned by community values and economic objectives as well as by ecological conditions. Therefore, aligning the decision and
the supporting risk and economic analyses with ‘what matters to people’ is essential to achieve acceptable risk solutions that can be easily and effectively communicated to the public” (EPA SAB, 2007, p. ii). To achieve that alignment the SAB recommends that EPA “increase its understanding of and capacity to utilize ecosystem valuation methods in conjunction with such decisions” (EPA SAB, 2007, p. ii).
The SAB has extended this work in a report on valuing ecological systems and services which includes a summary of the contingent valuation literature (EPA, 2009). Some of the relevant recommendations on implementation involve seeking information about public concerns and needs using a variety of methods, including interactive processes to elicit public values; describing the valuation measures in terms that are meaningful and understandable to the public; and providing information to decision makers about the level of uncertainty in the valuation efforts.
The Superfund site cleanup program provides EPA with the opportunity and mandate to consider public values, and it is one for which EPA has evaluated community concerns. For example, the Hudson River PCBs Superfund Site New York Record of Decision (EPA, 2002a) details community concerns about proposed remediation activities. Community concerns included traffic, noise, construction lighting, air quality, odor, aesthetics, and a loss of recreational activities on the river. In 2004 EPA released a document on performance standards for quality-of-life concerns (EPA, 2004a) that addressed odor, noise, construction lighting, and navigation. Other quality-of-life considerations (aesthetics, road traffic, and recreational activities on the river) were reviewed as part of the development of performance standards, but it was determined that they did not need a performance standard. No quality-of-life standard for water quality was issued because other standards and regulatory requirements dealt with water quality.
While the quality-of-life issues were not part of the decision by EPA to remediate the Hudson River site per se, these issues were considered in decisions about the day-to-day process for remediation. For example, EPA assessed the levels of noise from remediation activities. The agency stated that it “has determined that the noise associated with construction and continuous operation of the sediment process/transfer facilities and hydraulic and mechanical dredging operations is not expected to be a significant concern” (EPA, 2004a, pp. 4-3, 4-4). The basis for that assessment, however, is not clear in the record of decision, the responsiveness summary, or the white papers that make up the lengthy documentation for the Hudson River cleanup (EPA, 2002b).
Under the Superfund Law, the Department of the Interior and the National Oceanic and Atmospheric Administration have used contingent valuation to calculate damages to natural resources (Lipton et al., 1995).
Contingent valuation has also been used, for example, to assess people’s preferences among three endangered species (Wallmo and Lew, 2011).
In 2005 the Irish Environmental Protection Agency undertook a nationwide survey of citizen’s views on litter, illegal dumping, and the remediation of illegal dumpsites (IEPA, 2006). The survey examined the level of concern, demand for greater enforcement, and willingness to pay among citizens. The survey results suggested that waste management was by far the most important environmental issue facing that country—56 percent of the 1,500 respondents chose that answer compared to, for example, the 2 percent who identified factory emissions or the 9 percent who identified planning and green spaces as the most important environmental issue. While the committee did not identify or explore how the Irish agency used this information, the survey results shed light on how the people of that nation value environmental issues and on how those values can be assessed.
Tools and surveys are available for use in contingent valuation. Managing Risks to the Public: Appraisal Guidance (HM Treasury, 2005) presents a concern assessment tool that provides “a framework for understanding people’s concerns in order that they can be considered in policy development and in the development of related consultation agreements and communication strategies” (p. 33). The framework is based on findings related to risk perception (Fischhoff, 1995; Slovic, 1987) and includes questions about familiarity and experience with a hazard, understanding of cause and effect, the fairness of the distribution of risks and benefits, the fear or dread of a risk, and the trust that people have in the agency (HM Treasury, 2005).
Survey techniques and contingent valuation have both critics and supporters (Diamond and Hausman, 1994; Epstein, 2003; Hanemann, 1994; Portney, 1994). For example, Yeager and colleagues (2011) demonstrated the sensitivity of responses to survey question design and argued that caution should be exercised when interpreting surveys. There is also evidence that responses to such surveys reflect what people are willing to pay for something in general, but not necessarily what they are willing to pay for the specific item they are being questioned about—an observation called the list paradox. For example, once a person has indicated a willingness to pay $5 for clean air, if asked about valuing another item, such as Yosemite National Park or the other 57 national parks, he or she will also often say $5. The “willingness” to pay $5 is actually a signal that the person cares about the environment, and it does not indicate that the person, upon reflection, would actually be willing to pay $5. In addition, Kahneman (2011) pointed out that because people tend to think only about what is in front of them and neglect the opportunity costs, people make better decisions if they think more broadly. Showing people a full list of problems shows the opportunity costs clearly and can help balance the responses. Brookshire and Coursey (1987) demonstrated that an individual’s valuation can change as he or she
become more familiar with the survey methods and “the degree in which values are measured in a market or nonmarket environment.”
An issue that is often not addressed directly is the fact that arguments about uncertainty are often a proxy for other public concerns. As discussed further in Chapters 5 and 6, when regulators, stakeholders, and the public work together in a decision-making process and disagreements occur, consideration should be given to what might be motivating the public’s concerns, questions, and oppositions to decisions. Concerns about a decision and its uncertainties sometimes stem not from the specific uncertainties, but rather from dissatisfaction about the process that led to the decision (Covello et al., 2004; Hance et al., 1988). The scientific and technical disagreements can be more readily identified and resolved if people have an opportunity to air their concerns and if everyone has a common understanding of what the regulatory options and legal context are, of the science and the economics, and of what technologies are available (IOM, 2012). Mistrust of government is often at the root of what may appear at first to be scientific or technical disagreements.17 For example, Santos and Edwards (1990) explored the underlying issues in disagreements about citing a nuclear power plant. They concluded, “The bottom line was that the public did not trust the government’s and nuclear industry’s ability to control human error. Unfortunately, there appears to have been no direct dialogue about this. Instead, the risk assessment itself [for the nuclear power plant siting] became the target for challenge” (p. 60).
Current costs and benefits accrued in the present are typically considered to be worth more than costs and benefits accrued in the future, based on the fact that individuals generally prefer to enjoy benefits now rather than later (see, for example, Gold et al., 1996). So when assessing the costs and benefits associated with an environmental regulation, future costs and benefits are discounted compared to current costs and benefits. Because the discount rate accounts for the preference for current benefits, there is no “correct” discounting rate, which creates uncertainty about how to select the appropriate discount rate and which also adds to the variability in the cost and benefit assessments. Differences of opinions about the appropriate discounting rate are unlikely to be resolved, moving it toward being a deep uncertainty. Therefore, it is less important to choose a discount rate than it is to provide information about how different discount rates affect the analysis of a regulation.
17 Social trust is discussed further in Chapter 6.
In practice, discount rates of 3 and 7 percent are in widespread use for projects of only a few decades in duration (EPA/RTI International, 2008). Some research indicates that long-term rates are less than short-term rates, and therefore, for projects affecting more than one generation, such as those involving climate change, a lower discount rate or varying discount rates that are given by a schedule may be appropriate (Newell and Pizer, 2003, 2004). These “default” assumptions indicate the uncertainty inherent in this component of economic analyses. The most direct way of accounting for such uncertainties is to evaluate the benefits versus the costs of a particular decision using alternative discount rates and to judge how estimates of net benefits or benefit-to-cost ratios are affected and if they are affected sufficiently to affect the decision about project or rule adoption. As the committee indicated in Chapter 1, a sole focus on uncertainties in the risk assessment ignores possibly even greater uncertainties in key factors in decision making.
Assessment of Uncertainties in Economic Analyses
There are several methods for accounting for the uncertainty in economic analyses in a decision, ranging from the very simple to the highly complex. Simple solutions include inflating costs, deflating benefits, or both, and establishing a threshold for a regulatory action, such as only adopting those projects for which the projected benefit is 1.25 times the cost. That approach is similar to the use of defaults in estimating health risks and, similarly, has inherent uncertainties that EPA should consider when weighing economic factors. An example of a more complex approach to dealing with uncertainty in economic analyses can be found in OMB’s guidelines on the development of regulatory analysis (OMB, 2003). Those guidelines recommend specifying the entire probability distribution of benefits and costs by, for example, using Monte Carlo simulations to assess the degree of uncertainty in estimates of benefits, costs, and benefit–cost ratios (OMB, 2003). Monte Carlo simulations are widely used in business applications and also, as mentioned in Chapter 2, for estimating human health risks (Marshall et al., 2009; Thompson et al., 1992).
People’s decisions about when they can afford to retire provide an example of how Monte Carlo analyses can aid decision making. By running multiple simulations using typical future consumption levels and the probability of different rates of return on investments and longevity, both of which are considered to be uncertain, Monte Carlo simulations can estimate the probability that an individual near retirement will outlive his or her assets, given the person’s initial asset mix. For a Monte Carlo simulation of a BCA or CEA, the input would be the range of potential benefits and the likelihood that each benefit would turn out to be the “true benefit” that
is actually seen. Similarly, the range of potential costs and the likelihood that each cost is the “true” or “actual” cost would be used to determine the probable cost. OMB’s guidelines (OMB, 2003) and the NRC report Estimating the Public Health Benefits of Proposed Air Pollution Regulations (NRC, 2002b) both recommend probabilistic modeling of uncertainty for BCAs and CEAs. Although such models are not commonplace in EPA’s estimates of benefits and costs (Jaffee and Stavins, 2007), the agency has used them at times. In a Monte Carlo analysis of benefits and costs, one calculates the probability that the actual net benefit will be negative or else that the net benefit will be less than a prespecified threshold amount.
Reporting of Uncertainty
A common problem when decision makers are faced with uncertainty related to a decision is how the uncertainty should be presented to the decision makers. Estimates of uncertainty in BCA and CEA are often presented as aggregate numbers. Aggregates, however, do not by themselves help a decision maker identify the individual sources of uncertainty, but rather they indicate the overall level of certainty in the analyses. Knowing the specific sources of uncertainty can be as important as—if not more important than—documenting uncertainties in the aggregate. If the sources of uncertainty are identified, it may be possible to decrease the uncertainty by conducting further research or to refine the policy option in order to reduce the impact of the uncertainty. For example, if it is known that the uncertainty in a benefit valuation is due to heterogeneity, then the identification of the groups or stakeholders who would bear the burden of a higher-than-anticipated cost or of a lower-than-anticipated benefit because of the uncertainty might allow decision makers to design the initial proposed regulations to address or cope with those potential outcomes in advance. Although such analyses are possible, they are rarely done for BCAs and CEAs (NRC, 2002a). If the individual sources of uncertainty that contribute to the overall uncertainty can be determined, uncertainty analyses in BCAs and CEAs can incorporate graphic representations—such as Tornado plots where the relative importance of the different sources of uncertainty are displayed sequentially—to provide an easily interpretable graphic display of the sources of uncertainty (Krupnick et al., 2006).
Arsenic in Drinking Water
As discussed in Chapter 2, in 2001 EPA evaluated a recently enacted standard that would decrease the allowable concentration of arsenic in drinking water. When estimating the cost associated with implementing the proposed arsenic rule, EPA used a Monte Carlo simulation to forecast “a
distribution of costs around the mean compliance cost expected for each system size category” (EPA, 2000a).18 The uncertainty analysis included only treatment costs, and it assumed a single commercial discount rate of 5 percent. In reviewing EPA’s cost estimates for the arsenic rule, the Arsenic Cost Working Group of the National Drinking Water Advisory Council noted that the “value of existing national cost estimates is now limited by the large uncertainty associated with the estimated outcomes” and recommended that the EPA “clearly explain the limitations of each estimate and quantify the uncertainty associated with the Arsenic Rule estimates” (Arsenic Cost Working Group to the National Drinking Water Advisory Council, 2001, p. 2). The advisory council also noted the need for “a more representative methodology to assess compliance cost,” noting the advantages of “an approach based on aggregated county, regional or state costs, coupled with extensive individual case analysis” (p. 2). Furthermore, the council recommended the use of a distribution of flows rather than the mean or median flow and noted the “significant uncertainty associated with EPDS [the number of entry points to the distribution system] determination” (p. 3), and it recommended how EPA should examine information given that uncertainty.
The relative lack or small amount of uncertainty analyses performed for the analysis of cost and benefits in the arsenic rule contrasts sharply with the extensive uncertainty analyses that were conducted for the human health risk estimates. EPA and the National Research Council conducted extensive analyses of the health data in order to estimate health risks, and that work included an extensive discussion of inherent uncertainties as well as quantitative assessments of how many of those uncertainties might affect the health risk estimates.
Some of those limitations were outlined by the Arsenic Rule Benefits Review Panel of EPA’s SAB (EPA, 2000b). That panel noted that in the estimates for the number of cancer cases avoided there was nothing said about the uncertainty in the assumptions made about the lag time between the reduction of arsenic exposures and the reduction in risk. It also noted that the benefits and costs should be summarized in a manner that would indicate the variability in the benefits and costs associated with different sizes of water treatment facilities. It recommended that the age distribution of cancer cases avoided should be presented in order to allow readers to know the age distribution of those benefits, and it discussed the limitations of EPA’s valuation of avoided cancer morbidity and mortality, recommending that the agency conduct more uncertainty analyses around those valuations, using sensitivity analyses or Monte Carlo analyses.
18 In that report EPA noted that, “Historically, most drinking water regulatory impact analyses used point estimates to describe the average system-level costs” (EPA, 2000a, pp. 6–17).
A number of factors other than human health risks, technological availability, and economics affect EPA’s decisions. Some of those factors and the uncertainty in estimates of them can be accounted for in estimates of human health risks (for example, the adverse health effects on sensitive populations such as infants and children) or in economic analyses (for example, the value that the public places on having access to recreational space is often accounted for in BCAs). Other factors—such as environmental justice, political climate, and public sentiment—and their uncertainty, however, are not taken into account in the analyses of human health risk, economics, and technological availability, despite their influence on decisions.
Executive Order 13563, issued in 2011, states that regulations are “to be based, to the extent feasible and consistent with law, on the open exchange of information and perspectives among State, local, and tribal officials, experts in relevant disciplines, affected stakeholders in the private sector, and the public as a whole.”19 As defined by EPA, public values “reflect broad attitudes of society about environmental risks and risk management” (EPA, 2000d, p. 52). Public values can be specific to a certain geographic area or can apply to the nation as a whole. As such, public values affect both local- and national-level decisions and can affect EPA’s regulations. Several programs within EPA take community concerns into consideration, as required by either statute or executive order. The Resource Conservation and Recovery Act and the Comprehensive Environmental Response, Compensation, and Liability Act, commonly known as Superfund, address several issues related to public values.
A community, for example, has an interest when it is chosen as a location for a hazardous waste facility or when an existing facility is permitted. EPA provides guidance for industries and agencies working with communities likely to be affected by a hazardous waste site location; in that guidance the agency describes potential community concerns, such as concerns about the effects on quality of life, including concerns about preserving the community’s use of space, enjoyment and value of property, and sense of belonging and security, as well as promoting the economically sound protection of resources (EPA, 2000e). The guidance does not provide advice on integrating those concerns into the decision-making process. Others define public participation as “the process by which public concerns, needs and values are incorporated into governmental and corporate decision making” (Creighton, 2005, p. 7). The broader aspect of public participation is recommended by others, including a National Research Council committee
19 Exec. Order No. 13563. Improving regulation and regulatory review. 76 FR 3821 (January 21, 2011).
(NRC, 2008) and EPA’s SAB Committee on Valuing the Protection of Ecological Systems and Services (EPA, 2009).20
The political climate can also affect EPA’s decisions. Although EPA is a scientific regulatory agency, its decisions take place in a broader context that includes more than just the scientific issues. As described by EPA (2000d), political factors that can affect decisions are “based on the interactions among branches of the Federal government, with other Federal, state, and local government entities, and even with foreign governments; these may range from practices defined by Agency policy and political administrations through inquiries from members of Congress, special interest groups, or concerned citizens” (p. 52). Thus the agency is influenced by a complex set of forces, not the least of which are the values, priorities, and direction of the President. No decision is based absolutely or purely on scientific analyses. Regardless of statutory directives, agencies often have broad discretion about when to act in the face of a risk, and a decision of when to act is influenced by executive branch leadership, congressional intention and attention, and the possibility of judicial intervention.
Although social considerations such as environmental justice and the political climate affect EPA’s decisions and there is uncertainty in those factors and how they influence decisions, there is seldom any discussion concerning just how those factors and their uncertainty affect a decision.
• EPA considers a number of factors in addition to human health risks when making regulatory decisions. EPA’s legal authority and requirements predominately determine what factors it considers when making regulatory decisions.
• Uncertainty is present in the assessment of all of the factors EPA that considers when making regulatory decisions, including technology availability and economic factors. Those uncertainties, however, are rarely analyzed or explicitly accounted for in EPA’s regulatory decisions. Similarly, factors such as public sentiment, environmental justice, and the political climate influence EPA’s decisions, but the uncertainty in those factors is rarely accounted for in EPA’s decisions. EPA also does not discuss the uncertainty in any of those factors in its decision documents as thoroughly as it does the uncertainty in human health risk estimates.
Although different estimates of technology availability are sometimes used, the uncertainties that stem from those estimates are
20 Chapter 6 further discusses the importance of stakeholder engagement and communication approaches for that participation.
not often carried through to the final outputs in a regulatory impact assessment.
Uncertainties in economic analyses are sometimes conducted, but they are not necessarily presented.
Uncertainties in public sentiment and social factors, such as environmental justice, are rarely accounted for explicitly in decisions, and their effects are rarely discussed.
The political climate can affect assessments of regulatory options. That effect of that climate on decisions is not always transparent, adding to the uncertainty in decisions.
• The methods and use of uncertainty analyses for technological, economic, and other factors are not as well established as for the uncertainties in human health risk estimates. The committee’s review of the literature indicated that uncertainty analyses for economic analyses have been studied more than uncertainty analyses for the technological and social factors.
The U.S. Environmental Protection Agency (EPA) should develop methods to systematically describe and account for uncertainties in decision-relevant factors in addition to estimates of health risks—including technological and economic factors—in its decision-making process. When influential in a decision, those new methods should be subject to peer review.
Analysts and decision makers should describe in decision documents and other public communications uncertainties in cost–benefit analyses that are conducted, even if not required by statute for decision making, and the analyses should be described at levels that are appropriate for technical experts and non-experts.
The U.S. Environmental Protection Agency should fund research, conduct research, or both to evaluate the accuracy and predictive capabilities of past assessments of technologies and costs and benefits for rulemaking in order to improve future efforts. This research could be conducted by EPA staff or else by nongovernmental policy analysts, who might be less subject to biases. This research should be used as a learning tool for EPA to improve its analytic approaches to assessing technological feasibility.
The U.S. Environmental Protection Agency should continue to work with stakeholders, particularly the general public, in efforts to identify their values and concerns in order to determine which uncertainties in other factors, along with those in the health risk assessment, should be analyzed, factored into the decision-making process, and communicated.
The U.S. Environmental Protection Agency should fund or conduct methodological research on ways to measure public values. This could allow decision makers to systematically assess and better explain the role that public sentiment and other factors that are difficult to quantify play in the decision-making process.
Adamowicz, W., J. Swait, P. Boxall, J. Louviere, and M. Williams. 1997. Perceptions versus objective measures of environmental quality in combined revealed and stated preference models of environmental valuation. Journal of Environmental Economics and Management 32(1):65–84.
Arsenic Cost Working Group to the National Drinking Water Advisory Council. 2001. Report of the Arsenic Cost Working Group to the National Drinking Water Advisory Council. Washington, DC: Environmental Protection Agency.
Atkinson, G., and S. Mourato. 2008. Environmental cost–benefit analysis. Annual Review of Environment and Resources 33:317–344.
Boyd, J., and A. Krupnick. 2009. The definition and choice of environmental commodities for nonmarket valuation. Washington, DC: Resources for the Future.
Breedlove, J. 1999. Natural resources: Assessing non-market values through contingent valuation. Washington, DC: Congressional Research Service.
Breyer, S. 1995. Breaking the vicious circle: Toward effective risk regulation. Cambridge, MA: Harvard University Press.
Brookshire, D. S., and D. L. Coursey. 1987. Measuring the value of a public good: An empirical comparison of elicitation procedures. American Economic Review 554–566.
Control of emissions of air pollution from highway heavy-duty engines. 1996. Federal Register 61(125):33421–33469.
Covello, V. T., D. B. McCallum, and M. T. Pavlova. 2004. Effective risk communication: The role and responsibility of government and nongovernment organizations. New York: Plenum Press.
Creighton, J. L. 2005. The public participation handbook: Making better decisions through citizen involvement. San Francisco, CA: Jossey-Bass.
Diamond, P. A., and J. A. Hausman. 1994. Contingent valuation: Is some number better than no number? Journal of Economic Perspectives 8(4):45–64.
EPA (U.S. Environmental Protection Agency). 1995. Survey of control technologies for low concentration organic vapor gas streams. Washington, DC: EPA.
———. 2000a. Arsenic in Drinking Water Rule economic analysis. Washington, DC: Office of Ground Water and Drinking Water, EPA.
———. 2000b. Proposed Arsenic in Drinking Water Rule regulatory impact analysis. Washington, DC: Office of Ground Water and Drinking Water, EPA.
———. 2000c. Regulatory impact analysis: Control of emissions of air pollution from highway heavy-duty engines. Washington, DC: PA.
———. 2000d. Social aspects of siting RCRA hazardous waste facilities. Washington, DC: EPA.
———. 2001. Arsenic rule benefits analysis: An SAB review. A review by the Arsenic Rule Benefits Review Panel (ARBRP) of the U.S. EPA Science Advisory Board (SAB). Washington, DC: EPA Science Advisory Board.
———. 2002a. Hudson River PCBs site New York record of decision. Washington, DC: Environmental Protection Agency. http://www.epa.gov/hudson/RecordofDecision-text.pdf (accessed November 21, 2012).
———. 2002b. Responsiveness summary: Hudson River PCBs site record of decision—White papers. Washington, DC: Environmental Protection Agency. http://www.epa.gov/hudson/d_rod.htm#response (accessed November 21, 2012).
———. 2004a. Hudson River PCBs Superfund site quality of life performance standards. Washington, DC: Environmental Protection Agency. http://www.epa.gov/hudson/quality_of_life_06_04/full_report.pdf (accessed November 21, 2012).
———. 2004b. Survey of technologies for monitoring containment liners and covers. Washington, DC: EPA.
———. 2007. Advice to EPA on advancing the science and application of ecological risk assessment in environmental decision making: A report of the U.S. EPA Science Advisory Board. Washington, DC: Environmental Protection Agency, Science Advisory Board.
———. 2009. Valuing the protection of ecological systems and services: A report of the EPA Science Advisory Board. Washington, DC: EPA Science Advisory Board.
———. 2010. Guidelines for preparing economic analyses. http://yosemite.epa.gov/ee/epa/eerm.nsf/vwAN/EE-0568-51.pdf/$file/EE-0568-51.pdf (accessed November 21, 2012).
———. 2011a. The benefits and costs of the Clean Air Act from 1990 to 2020. Final report. Washington, DC: Environmental Protection Agency, Office of Air and Radiation.
———. 2011b. Regulatory impact analysis for the final mercury and air toxics standards. Research Triangle, NC: Environmental Protection Agency, Office of Air Quality Planning and Standards, Health and Environmental Impacts Division.
———. 2012. Water: Industry effluent guidelines. Washington, DC: EPA. http://water.epa.gov/scitech/wastetech/guide/questions_index.cfm (accessed November 21, 2012).
EPA/RTI International. 2008. EMPAX–CGE model documentation interim report. Research Triangle Park, NC: EPA.
EPA SAB (EPA Science Advisory Board). 2007. Advice to the EPA. http://yosemite.epa.gov/sab/sabproduct.nsf/7140dc0e56eb148a8525737900043063/$file/sab-08-002.pdf (accessed November 21, 2012).
———. 2009. Valuing the protection of ecological systems and services. http://yosemite.epa.gov/sab/sabproduct.nsf/F3DB1F5C6EF90EE1852575C500589157/$File/EPA-SAB-09-012-unsigned.pdf (accessed November 21, 2012).
Epstein, R. A. 2003. The regrettable necessity of contingent valuation. Journal of Cultural Economics 27(3):259–274.
Fischhoff, B. 1995. Ranking risk. Risk: Health, Safety, and Environment 6:189–200.
Freeman, A. M. 2003. The measurement of environmental and resource values: Theory and methods. Washington, DC: Resources for the Future.
Gold, M. R., J. E. Siegel, L. B. Russell, and M. C. Weinstein. 1996. Cost effectiveness in health and medicine. New York: Oxford University Press.
Hance, B., C. Chess, and P. Sandman. 1988. Improving dialogue with communities: A risk communication manual for government. Trenton: New Jersey Department of Environmental Protection.
Hanemann, W. M. 1994. Valuing the environment through contingent valuation. Journal of Economic Perspectives 8(4):19–43.
Harrington, W. 2006. Grading estimates of the benefits and costs of federal regulation. Washington, DC: Resources for the Future.
Harrington, W., R. Morganstern, and P. Nelson. 1999. On the accuracy of regulatory cost estimates. http://www.rff.org/documents/RFF-DP-99-18.pdf (accessed November 21, 2012). Harrington, W., L. Heinzerling, and R. D. Morgenstern. 2009. Reforming regulatory impact analysis. Washington, DC: Resources for the Future.
HM Treasury. 2005. Managing risks to the public: Appraisal guidance. London: Crown.
ICF Consulting. 2005. The Clean Air Act amendments: Spurring innovation and growth while cleaning the air. Fairfax, VA: ICF Consulting.
IEPA (Irish Environmental Protection Agency). 2006. Public perceptions, attitudes and values on the environment—A national survey. Wexford, Ireland: Irish Environmental Protection Agency.
IOM (Institute of Medicine). 2006. Valuing health for regulatory cost-effectiveness analysis. Washington, DC: The National Academies Press.
———. 2012. Ethical and scientific issues in studying the safety of approved drugs. Washington, DC: The National Academies Press.
Jaffee, J., and R. N. Stavins. 2007. On the value of formal assessment of uncertainty in regulatory analysis. Regulation and Governance 1:154–171.
Kahneman, D. 2011. Thinking, fast and slow. New York: Farrar, Straus, Giroux.
Krupnick, A., R. Morgenstern, M. Batz, P. Nelson, D. Burtraw, J. Shih, and M. McWilliams. 2006. Not a sure thing: Making regulatory choices under uncertainty. Washington, DC: Resources for the Future.
Lipton, D. W., K. Wellman, I. C. Sheifer, and R. F. Weiher. 1995. Economic valuation of natural resources: A handbook for coastal resource policymakers. Silver Spring, MD: U.S. Department of Commerce, National Oceanic and Atmospheric Administration—Coastal Ocean Office.
Marshall, C. M., R. W. Kolb, and J. A. Overdahl. 2009. Monte Carlo techniques in pricing and using derivatives. In Financial derivatives. New York: John Wiley & Sons, Inc. Pp. 423–440.
Meltzer, D. 1997. Accounting for future costs in medical cost-effectiveness analysis. Journal of Health Economics 16(1):33–64.
Morgenstern, R. 1997. Economic analyses at EPA: Assessing regulatory impact. Washington, DC: Resources for the Future.
———. 2011. Reflections on the conduct and use of regulatory impact analysis at the U.S. Environmental Protection Agency. Discussion paper prepared for an RFF Conference, April 7. http://www.rff.org/RFF/Documents/RFF-DP-11-17.pdf (accessed November 21, 2012).
Moss, R., and S. Schneider. 2000. Uncertainties in the IPCC TAR: Recommendations to lead authors for more consistent assessment and reporting. In Guidance papers on the cross cutting issues of the Third Assessment Report of the IPCC, edited by R. Pachauri, T. Taniguchi, and K. Tanaka. Geneva: World Meteorological Organization. Pp. 33–51.
NAPEE (National Action Plan for Energy Efficiency). 2008. National action plan for energy efficiency vision for 2025: A framework for change. Washington, DC: Environmental Protection Agency, Department of Energy.
Newell, R. G., and W. A. Pizer. 2003. Discounting the distant future: How much do uncertain rates increase valuations? Journal of Environmental Economics and Management 46(1):52–71.
———. 2004. Uncertain discount rates in climate policy analysis. Energy Policy 32(4):519–529. NRC (National Research Council). 2002a. Estimating the public health benefits of proposed air pollution regulations. Washington, DC: The National Academies Press.
———. 2002b. Estimating the public health benefits of proposed air pollution regulations. Washington, DC: The National Academies Press.
———. 2004. Valuing ecosystem services: Toward better environmental decision-making. Washington, DC: The National Academies Press.
———. 2008. Public participation in environmental assessment and decision making. Washington, DC: The National Academies Press.
OMB (Office of Management and Budget). 2003. Regulatory analysis: Circular A–4 to the heads of executive agencies and establishments. http://www.whitehouse.gov/omb/circulars/a004/a-4.pdf (accessed November 21, 2012).
———. 2005. Validating regulatory analysis: 2005 report to Congress on the costs and benefits of federal regulations and unfunded mandates on state, local, and tribal entities. Washington, DC: Office of Management and Budget.
Pavitt, K. 1984. Sectoral patterns of technical change: Towards a taxonomy and a theory. Research Policy 13(6):343–373.
Pearce, D., G. Atkinson, and S. Mourato. 2006. Cost–benefit analysis and the environment: Recent developments. Paris: OECD Publishing.
Portney, P. R. 1994. The contingent valuation debate: Why economists should care. Journal of Economic Perspectives 8(4):3–17.
RTI International. 2008. EMPAX–CGE model documentation interim report. Research Triangle Park, NC: EPA.
Santos, S., and S. Edwards. 1990. Comparative study of risk assessment and risk communication practices in western Europe: A report prepared for the German Marshall Fund of the United States.
Shimshack, J. P., and M. B. Ward. 2005. Regulator reputation, enforcement, and environmental compliance. Journal of Environmental Economics and Management 50(3):519–540.
———. 2008. Enforcement and over-compliance. Journal of Environmental Economics and Management 55(1):90–105.
Sloan, F. A., and C.-R. Hsieh. 2012. Health economics. Cambridge, MA: The MIT Press.
Slovic, P. 1987. Perception of risk. Science 236(4799):280–285.
Thompson, K. M., D. E. Burmaster, and E. A. C. Crouch. 1992. Monte Carlo techniques for quantitative uncertainty analysis in public health risk assessments. Risk Analysis 12(1):53–63.
Viscusi, W., and J. Aldy. 2003. The value of a statistical life: A critical review of market estimates throughout the world. Journal of Risk and Uncertainty 27(1):5–76.
Wallmo, K., and D. K. Lew. 2011. Valuing improvements to threatened and endangered marine species: An application of stated preference choice experiments. Journal of Environmental Management 9(27):1793–1801.
Williams, M. 1994. Combining revealed and stated preference methods for valuing environmental amenities. Journal of Environmental Economics and Management 26:271–292.
Yao, R., and P. Kaval. 2011. Non-market valuation in New Zealand: 1974 to 2005. Annals of Leisure Research 14(1):60–83.
Yeager, D. S., S. B. Larson, J. A. Krosnick, and T. Tompson. 2011. Measuring Americans’ issue priorities: A new version of the most important problem question reveals more concern about global warming and the environment. Public Opinion Quarterly 75(1):125–138.