Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
16 Challenges in Evaluating Voluntary Environmental Programs Kathryn Harrison Novel environmental policy instruments, including tradable permits, vol- untary programs, and more flexible approaches to regulation, have been embraced by governments throughout the world in recent years. Rigor- ous evaluation of these diverse approaches is clearly essential if we are to draw any conclusions about which approaches offer the most effective basis for future policy. The focus of this chapter is on the particular challenges associated with evaluation of voluntary programs. Because it is not possible to provide a sum- mary of the effectiveness of the wide variety of voluntary programs that have emerged in recent years within the scope of this brief chapter, I propose instead to focus on some of the challenges that arise in evaluation of voluntary pro- grams, illustrated with three case studies. The discussion that follows focuses on conceptual issues in evaluation of voluntary programs, setting aside for the mo- ment questions of political will and adequacy of administrative resources to conduct program evaluations (Gormley, 2000~. Policy evaluation is a difficult task, one that is too seldom undertaken for traditional regulatory as well as novel voluntary programs. However, rigorous evaluation of voluntary programs is arguably particularly important, because the historical failure of markets and voluntarism to address environmental problems and resource depletion creates a heavy burden of proof for those who advocate voluntary alternatives to regulation (Andrews, 1998; Leiss and Associates, 1996~. Unfortunately, evaluation of voluntary programs is simultaneously especially difficult, because it is plagued by problems of data availability, credibility, and self-selection. I will argue that as a result of these factors, there has been a tendency to overstate the effectiveness of voluntary programs. In the chapter that follows, I discuss three questions that arise in any evalu- 263
264 CHALLENGES IN EVALUATING VOLUNTARY ENVIRONMENTAL PROGRAMS ation: (1) What is being evaluated? (2) What criteria should be used in evalua- tion? (3) What baseline should be used for the evaluation? Then I illustrate issues that arise in that discussion with case studies of the U.S. 33/50 program, the Canadian Accelerated Reduction/Elimination of Toxics (ARET) program, and the Canadian National Pollutant Release Inventory. WHAT IS BEING EVALUATED? A TYPOLOGY OF VOLUNTARY PROGRAMS The Organization for Economic Co-operation and Development (OECD) (1999) distinguishes among three types of voluntary programs: negotiated vol- untary agreements, public voluntary programs (also sometimes referred to as "voluntary challenge" programs), and unilateral (i.e., nongovernmental) pro- grams. Among these, voluntary agreements are closest to regulation. Voluntary agreements are characterized by strong expectations on the part of government that nongovernmental parties to the negotiated agreements (typically industry) will comply, and such agreements often are prompted by an explicit or implied threat of regulation should the voluntary approach fail. Voluntary agreements typically are negotiated by government and industry, whether an individual firm or an entire sector, though other stakeholders also may be involved. Although most voluntary agreements are nonbinding, some take the form of legally bind- ing contracts.) In contrast to voluntary agreements, governmental efforts to persuade target groups to change their behavior via voluntary challenges or public voluntary programs typically involve less arm twisting. Although voluntary agreements have been prominent in Europe in recent years,2 voluntary challenge programs are more common in North America. Examples include the U.S. Environmental Protection Agency's (EPA's) 33/50 program and various Partners for the Envi- ronment programs, and, in Canada, the ARET Challenge and the Voluntary Chal- lenge and Registry for greenhouse gases.3 The central difference between a voluntary agreement and a voluntary challenge is that the latter is an open-ended public challenge that applies widely, but that no particular actor is expected to accept. Thus, the EPA 33/50 program called on any and all firms releasing 17 specified toxic substances to reduce their discharges of those substances. In practice, requirements of participation in challenge programs tend to be very flexible. For instance, in both the 33/50 and ARET programs, participants were expected to commit to make some reductions of the listed substances, but they were not required in practice to meet any particular performance standard. Information dissemination is often central to voluntary challenge programs, which typically offer participants an incentive of public recognition. A related approach, and one that has buttressed both the 33/50 and ARET programs, is the mandatory pollutant release or "community right-to-know" inventory (discussed also by Herb and colleagues, this volume, Chapter 15~. Such inventories require
KATHRYN HARRISON 265 that polluters publicly report their discharges, but do not require reductions of those discharges per se. Rather, the intent of the approach is to facilitate public pressure on firms to make voluntary reductions. Inventories can also document changes in releases reported to complementary voluntary challenge programs. In addition to these government-sponsored voluntary programs, a wide vari- ety of nongovernmental or unilateral programs exist (many of which are re- viewed by Nash, this volume, Chapter 14~. In many respects, nongovernmental programs parallel the state-based voluntary programs described earlier, but with someone other than government doing the coercing or encouraging. Participants in nongovernmental voluntary programs vary from trade associations to nonpar- tisan third parties such as the International Organization for Standardization (ISO) to environmental groups. A critical distinction among nongovernmental programs (to be discussed) is whether they focus directly on environmental per- formance objectives or rather on management systems as a tool to improve envi- ronmental performance. No single unidimensional typology can capture all the relevant differences among voluntary programs. There is thus substantial variation within each of the categories listed.4 However, accepting OECD's three-way typology as a useful starting point, it is apparent that the kind of voluntary program being evaluated has several implications for the evaluation. The first issue concerns the credibility of the evaluation, which can be especially problematic in the case of nongovernmental programs sponsored by firms or trade associations. In such cases, evaluations by the program's sponsors understandably may be viewed with the same skepticism that would greet students not only grading their own exam papers, but also writing the exam questions in the first place. Verification by independent third parties is thus essential to lend credibility to the evaluation. That is not to suggest that program evaluations by government are immune to credibility challenges. Government evaluations of voluntary agreements ne- gotiated bilaterally by industry and the state may be met with particular skepti- cism because as Ayres and Braithwaite (1992:55) note, "The very conditions that foster the evolution of cooperation are also the conditions that promote the evolution of capture and indeed corruption."5 Openness of the evaluation pro- cess and, ideally, participation by environmentalists and other third parties in negotiating voluntary programs in the first place will be critical in establishing credibility (Gunningham and Grabosky, 1998~. Distinctions between voluntary programs also are relevant to program evalua- tion because conclusions about the effectiveness of one voluntary program may not apply to another very different program. In this regard, the typology offered earlier is at best a starting point, given the diversity of voluntary approaches within each category. For example, lessons learned from legally binding Dutch cove- nants negotiated to achieve environmental targets previously set by the state may not apply to nonbinding "gentlemen's agreements" in which the goals themselves
266 CHALLENGES IN EVALUATING VOLUNTARY ENVIRONMENTAL PROGRAMS may be subject to negotiation, let alone to voluntary challenges or nongovernmen- tal programs.6 CRITERIA FOR EVALUATION An obvious question for any program evaluation is what evaluative criteria should be used. Although it is clear that evaluation of any environmental policy instrument should consider its effectiveness in achieving environmental objec- tives, policymakers also routinely consider other societal objectives in devising and evaluating public policies. Evaluations of voluntary programs thus have tended to apply multiple criteria, including environmental effectiveness, cost- effectiveness for both government and business, equity, democratic accountabil- ity and public participation, promotion of innovation, and adaptability to future challenges (European Environmental Agency, 1997; Davies and Mazurek, 1996; Harrison, 1998~. Many studies of voluntary approaches also have emphasized "soft effects," such as improvement of the business-government relationship, building trust among stakeholders, and promotion of a more environmentally sensitive business culture (European Environment Agency, 1997; European Commission, 1996; Nash and Ehrenfeld, 1997~. Among these criteria, it is useful to draw a distinction between ultimate objec- tives, such as equity, efficiency, and reduction of environmental impacts, and in- termediate or instrumental objectives, such as fostering innovation and improving business-government relationships. It is striking that even ambitious attempts to evaluate voluntary programs often have been able to say little about attainment of objectives such as reduction of environmental impacts and cost-effectiveness, and thus have focused on instrumental goals (e.g., European Environment Agency, 1997~. Indeed, some analysts of voluntary programs focus almost exclusively on these "soft" objectives.7 This emphasis on intermediate objectives is not unique to voluntary programs. Gormley (2000) reports that in evaluating regulatory pro- grams, government agencies also tend to focus on easier-to-measure "outputs" (such as inspections and permitting activity) to the exclusion of "outcomes" (such as emissions) or "impacts" on human health or the environment. The focus on intermediate objectives arguably is more problematic with respect to voluntary than regulatory programs, however. The emphasis on soft effects often has been necessitated by a lack of data on environmental outcomes. This can be particularly problematic with respect to industry-sponsored pro- grams, where if data on environmental performance do exist, they tend to be proprietary. However, it also can be an issue for government-sponsored volun- tary programs which, unlike regulatory programs, typically do not rely on coer- cion to compel disclosure of monitoring nor threaten inspections to verify firms' reports. The extent of this problem was demonstrated by a study of 137 volun- tary agreements by the European Commission in 1997, which found that 118 had no requirement for firms to report the results of their compliance monitoring,
KATHRYN HARRISON 267 and 47 had no requirements for monitoring at all (European Commission Direc- torate, 1997~. Based on a review of 88 industry-sponsored voluntary initiatives worldwide, Borkey and Levesque (1998) concluded that industry initiatives typ- ically made no provisions for monitoring or public reporting. In other words, evaluations of voluntary codes of practice often have focused on intermediate objectives simply because they have no data with which to assess performance relative to environmental objectives. Although lack of environmental quality data is often problematic for evaluation of regulatory programs as well, at least the latter typically require monitoring and reporting by industry and also include provisions for independent verification by the state. That might be acceptable if there was a sufficient basis for the assumption that "soft effects," such as business-government cooperation, will in fact deliver "hard" environmental benefits. As Mazurek speculates (this volume, Chapter 13), coop- eration may not only foster success in voluntary programs, but may spill over and improve business-government relations in traditional regulatory programs. How- ever, in the absence of evidence, it seems premature to draw that conclusion. I have argued elsewhere that business-government cooperation also can be detri- mental to the environment (Harrison, 1995), while the fine line between coopera- tion and capture already has been noted (Ayres and Braithwaite, 1992:55~. Closely related to this distinction between ultimate and intermediate criteria is the distinction between voluntary programs that establish environmental per- formance objectives and "voluntary environmental management systems," which leave the setting of performance objectives up to individual participants and focus instead on management approaches to ensure that those goals are achieved. The latter approach is most common among industry-sponsored programs (Nash, this volume, Chapter 14~. For example, the Canadian Responsible Care codes of practice commit participants to "be aware of all effluents and emissions to the environment, monitor those for which it is necessary, and implement plans for their control when necessary," but do not specify which emissions should be monitored nor what degree of control should be applied. Although the program provides for independent audits of firms' compliance, the audit focuses exclu- sively on whether the facility has all the elements of the environmental manage- ment system in place, not on its environmental performance per se.8 It might be argued that this emphasis on intermediate objectives is quite appropriate for nongovernmental codes of practice intended for use as internal management tools and that it is thus unfair to hold such programs to the broader societal standards appropriate for public policy. That is certainly true to the extent that environmental management systems are limited to use as internal tools. However, the complaint seems disingenuous in light of firms' efforts to publicize to customers and regulators their compliance with management sys- tems such as ISO 14001 and Responsible Care. More importantly, to the extent that nongovernmental voluntary programs are being considered as a complement to or substitute for government policies, it is entirely appropriate for the state to
268 CHALLENGES IN EVALUATING VOLUNTARY ENVIRONMENTAL PROGRAMS hold them to the same standards, even if firms set less ambitious objectives for themselves. BENCHMARKS FOR EVALUATION A critical question in evaluating voluntary programs concerns the appropri- ate baseline against which to measure performance. One can, in fact, envision several different levels of analysis of environmental effectiveness (European Environment Agency, 1997~. The first, and simplest to achieve, is a measure of change (e.g., in environmental performance, costs to government, costs to the private sector, and so on) relative to a reference year. A second and more rigor- ous standard is to ask to what degree the observed changes are attributable to the program in question, acknowledging that actors' behavior might have changed anyway in response to other factors, including market forces and other concur- rent government policies. This standard necessitates the difficult task of devel- oping a counterfactual "business-as-usual" baseline that would have prevailed in the absence of the voluntary program. Finally, the third and most difficult standard of analysis is to compare the impact of the voluntary program to what would have happened if an alternative approach had been pursued, a level of analysis even more challenging than es- tablishing the "business-as-usual" baseline. Although a full-blown comparison is obviously very difficult, greater attention to this issue is nonetheless warrant- ed. Coercive regulatory programs often tend to be criticized against a yardstick of complete compliance and economic efficiency. In contrast, there is a tenden- cy for proponents of voluntary programs to hail any accomplishments above the current baseline as a success.9 This is appropriate if the voluntary program is the only realistic alternative to the status quo. However, to the extent that voluntary and regulatory programs are being considered as substitutes, the relevant ques- tion is how they compare to each other in practice. In constructing counterfactuals (i.e., "what would have happened if" scenar- ios), a critical issue that typically arises concerns the motivation for voluntary action by business. Voluntary environmental improvements may be undertaken by a profit-oriented firm for three reasons. First, the firm may be motivated to change its behavior to save money, such as by reducing energy costs or by recovering valuable reactants, regardless of any associated environmental bene- fits. Second, a firm may be motivated by market pressures for environmental improvements from investors, lenders, insurers, customers, or workers.~° Third, a firm might opt for voluntary measures beyond those prompted by cost savings and "green" market pressures in order to forestall or avoid mandatory regulation or legal liability. A critical question in that case is whether the incentive for voluntary action lies in reducing costs by accomplishing the same environmental objective with greater flexibility than allowed by traditional regulation, or rather in reducing costs by meeting less demanding environmental objectives.
KATHRYN HARRISON 269 Which of these three motives applies in a given case has implications both for the kind of voluntary program that is likely to be effective and the kinds of issues that will arise in program evaluation. When there are direct financial savings to be realized by reducing environmental losses, businesses have strong incentives to correct their behavior even in the absence of government intervention. However, government-sponsored information dissemination and technology transfer pro- grams nonetheless may help firms recognize and take advantage of waste reduc- tion opportunities and energy efficiency improvements (Storey, 1996~. Similarly, firms facing green market pressures also have incentives to change their behavior regardless of the existence of governmental voluntary programs. Nonetheless, government verification of voluntary efforts, such as through certification and awards programs, may enhance those market forces by lending greater credibility to firms' claims with consumers. Similarly, government programs to disseminate information about all firms' behavior can help environmentally motivated con- sumers and investors identify leaders and laggards alike. With respect to voluntary actions motivated either by environmental market pressures or direct cost savings, it is important to bear in mind that firms have incentives to change their behavior regardless of the existence of a voluntary program. Indeed, voluntary programs invite self-selection and thus prompt ques- tions of whether participating firms are simply the ones that were already in- clined to comply anyway. The challenge is thus to assess the marginal impact of the program on firms' behavior. A program evaluation that simply assesses the change in behavior relative to a reference year rather than a business-as-usual baseline will tend to overstate program effectiveness. A similar issue of self-selection also arose in the context of energy conser- vation programs adopted in the 1970s. Some analysts evaluating those programs argued that failure to control for self-selection inevitably would result in over- statement of program benefits, because individuals or firms undertaking volun- tary behavioral changes tend to be those facing the lowest costs of compliance (Weinstein et al., 1989~. In contrast, Keating (1989) argued that self-selection could result in understatement of program benefits, because those who opt to participate tend to be those who perceive the greatest benefits from participation. In that case, comparison of any extra actions undertaken by participants relative to actions by nonparticipants would not provide an accurate assessment of the program's benefits because, in the absence of the program, participants actually would have done less than the "control" nonparticipants. However, as Keating noted, understatement of effectiveness seemed to be more problematic with re- spect to programs that offered subsidies for energy conservation, and indeed for those who had already invested in desired energy conservation measures would not be eligible. In contrast, overstatement was more problematic in the absence of such inducements, where those who had already assumed compliance costs not only would be allowed, but also would be most likely to participate. The latter is more consistent with contemporary voluntary challenge programs.
270 CHALLENGESIN EVALUATING VOLUNTARY ENVIRONMENTALPROG^MS The situation is quite different when firms are motivated to undertake volun- tary actions by a threat of regulation. In this case, the challenge for government is not merely to enhance existing market incentives, but to create those incen- tives in the first place in the form of a credible and sustained threat of regulation (Glachant, 1994:47; Gibson, 1999:244~. There is reason to believe that govern- ment coercion is still a powerful influence on business environmental behavior. A survey by the European Commission found that the potential to forego or postpone regulation was cited as the most important benefit of voluntary envi- ronmental agreements by roughly two thirds of industry respondents (European Commission, 1997~. Similarly, a Canadian business survey reported that 95 percent of firms cited compliance with regulations as one of the top five factors motivating their environmental improvements. One implication is that in eval- uation of voluntary agreements in particular, the applicable comparison is not necessarily the status quo or even the business-as-usual baseline, but rather what would have happened had regulations been adopted instead. The significance of coercion in motivating firms' behaviors also suggests a need to look at whether concurrent regulatory policies have contributed to any changes in behavior ob- served relative to the reference year. The problem is that those conducting an evaluation of a voluntary program typically do not know what is motivating firms targeted by the program. Differ- ent firms are likely to have different motivations, and the same firm may be motivated to varying degrees by each of the three incentives noted earlier. Firms have incentives to understate their environmental releases and impacts whether they face voluntary or regulatory programs. However, it may be particularly difficult to unpack the degree to which progress is attributable to a voluntary program as compared to regulation. Once a regulatory program is established requiring a comparable level of compliance for all facilities affected, there is little reason for a firm to overstate what it "would have done anyway." In contrast, firms participating in a voluntary program in order to forestall regula- tion have an interest in convincing regulators not only of how much they have done but also how they have sufficient market incentives to maintain that behav- ior in the absence of regulation. As a result, although the need to establish a credible baseline requires that the evaluator understand what is going on in the heads of the target population, it is by no means clear that one can get a credible answer simply by asking. CASE STUDIES Three prominent voluntary programs the U.S. 33/50 program, the Canadi- an Accelerated Reduction/Elimination of Toxics (ARET) program, and the Ca- nadian National Pollutant Release Inventory (NPRI) are reviewed in this sec- tion. No claim is offered that these three programs represent the universe of voluntary programs, though it is noteworthy that they are often advanced as
KATHRYN HARRISON 271 particular success stories. Rather, the case studies are used to illustrate some of the issues already introduced, including the issue of credibility in self-reporting, the importance of establishing a common reference year, and the importance of questioning the degree to which progress relative to that reference year can be attributed to the program. 33/5012 In 1991, the EPA challenged the business community to voluntarily reduce its releases and transfers of 17 high-priority chemicals by 33 percent by the end of 1992 and by 50 percent by the end of 1995, dubbing the program "33/50." Consistent with the previous discussion of voluntary challenges, requirements for participation were very flexible. A firm needed only to write to the EPA pledging some degree of reduction of its discharges of any of the 33/50 chemi- cals. In turn, the EPA would provide a certificate of appreciation and recognize 33/50 participants in its annual report on the Toxics Release Inventory (TRI). Although only 13 percent of firms contacted by the EPA agreed to partici- pate, the goal of a 33-percent reduction relative to the reference year was achieved a year early, by the end of 1991, and the 50-percent reduction goal also was achieved early, by the end of 1994 (EPA, 1999~. By the target year of 1995, reductions of 55 percent had been achieved. That has prompted many, including the EPA, to cite the 33/50 program favorably as an example of the potential of voluntary approaches. However, when one considers whether reductions were achieved relative to a "business-as-usual" baseline, the benefits of the 33/50 program are not as clear. As noted in Table 16-1, the first concern is that when the program was launched in 1991, the EPA chose 1988 as the reference year, because it was the most recent year for which discharge data were available. As a result, more than one quarter of the reported reductions occurred before the program' s inception. TABLE 16-1 Analysis of Reductions of Discharges and Transfers of 17 Chemicals Reported to the 33/50 Program 1988 Reference Year 1990 Reference Year Goal Total reduction by 1995 target year Reductions after start of program Reductions by participants Excess reductions by participants relative to nonparticipants 50% 55% 40% 27% 9% NA NA 47% 32% 11% Source: EPA (1999).
272 CHALLENGESIN EVALUATING VOLUNTARY ENVIRONMENTALPROGRAMS Excluding these reductions, the total reductions reported to the 33/50 program were 40 percent relative to the 1988 reference year or 47 percent relative to a more meaningful 1990 reference year. The significant downward trend in releases even before the inception of the program leads one to ask what was motivating those reductions and to what degree those trends continued during the 33/50 years. In fact, even firms that elected not to participate in the program made substantial reductions in their discharges and transfers of 33/50 chemicals over the course of the program. Indeed, more than one quarter of the reductions reported by the 33/50 program were made by nonparticipating firms. Although it is conceivable that some nonparticipating firms were motivated by the 33/50 program to reduce their releases, perhaps because they interpreted the program as a subtle threat of regu- lation rather than an opportunity for credit, the fact that firms were making deep reductions even before the program was introduced would lead one to look first for other explanations. Excluding nonparticipants, reductions relative to a 1990 reference year fell by 32 percent, as shown in Table 16-1. Circumstantial evidence does indicate that the 33/50 program encouraged firms to make reductions over and above what they would have made otherwise. Firms participating in the 33/50 program reduced their discharges of 33/50 chem- icals by 55 percent from 1990 to 1995, compared to 36 percent for nonparticipat- ing firms. This 19 percent "extra" reduction by participating firms translates to an 11 percent reduction relative to the total releases and transfers of 33/50 chem- icals in the 1990 reference year. The question of self-selection remains, however. The fact that partici- pants in the 33/50 program made greater reductions than nonparticipants does not necessarily indicate that those reductions were prompted by the 33/50 pro- gram. Firms already inclined to make substantial reductions of 33/50 chemi- cals, whether in response to negative publicity associated with mandatory re- porting of discharges to TRI, market forces, cost savings, or concurrent regulatory requirements, simply may have been the ones inclined to sign on for credit. This is supported by Arora and Cason's (1996) finding that the larger a firm's releases and transfers, the more likely it was to participate in 33/50 because these are the firms that would be expected to be subject to the greatest pressure in response to the release of TRI data, even in the absence of the 33/ 50 program. It is also problematic that none of the analyses of the 33/50 program conducted to date systematically have controlled for the effects of concurrent regulations. Yet the EPA (1999) reports that reductions of two Montreal protocol substances included in the 33/50 program alone account for 27 percent of reductions from 1990 to 1995.~3 Moreover, O'Toole et al. (1997) found that stringency of state regulations was one of the two most important factors in accounting for state-level reductions of 33/50 chemicals as reported to TRI. In summary, although it is not possible to conclude with confidence the precise benefits of the 33/50 program, it appears to have prompted dis-
KATHRYN HARRISON 273 charge reductions of the specified chemicals of less than 11 percent, well be- low the 55 percent often attributed to the program. ARET The Canadian ARET Challenge, launched in 1994, is similar in many re- spects to the 33/50 program, though more ambitious. After the details of the challenge were negotiated by the government and some industry representatives, a broader challenge was issued to all firms to reduce discharges of some 30 chemicals considered to be toxic, persistent, and bioaccumulative by 90 percent by the year 2000, and of 87 others with one or more of these characteristics by 50 percent by the same year. Characteristic of a voluntary challenge program, there was no threat of penalties for failure to achieve those goals. Indeed, as in 33/50, firms that choose to participate were not required to commit to the full 90- percent and 50-percent reductions. Preliminary assessments of the impact of ARET based on the first 4 years of participant reports are promising. By the end of 1998, action plans had been received from 316 industry and government facilities (ARET,2000~. Discharges of all ARET substances had been reduced 67 percent relative to base year levels. Reduction levels of the class of substances targeted for 50-percent reductions were surpassed 4 years ahead of schedule, with further reductions promised. As with 33/50, however, the degree to which these reductions are attribut- able to the ARET program is unclear. The base year problem is exacerbated in the ARET case, because each participating facility can pick its own base year anytime after 1987. This option allows firms to claim credit toward the ARET program for discharge reductions they made as much as 6 years before the pro- gram's inception, and to strategically choose a year with particularly high dis- charges to maximize apparent reductions (Gallon, 1998~. In fact, ARET pro- gram figures indicate that roughly half of the 67-percent reductions claimed by the end of 1998 had been achieved before the program was even launched.~4 As with 33/50, there are also questions about whether the reductions attrib- uted to ARET are in fact voluntary, and if so, whether they are attributable to the program. The ARET program notes that one quarter of base year emissions were substances regulated under the Canadian Environmental Protection Act (CEPA), and a further 11 percent were substances undergoing evaluation for potential regulation under CEPA. i5 Thus one might question whether the ARET reductions are an artifact of regulation. Harrison and Antweiler (2001) found that facilities reporting on-site releases of regulated substances to the Canadian National Pollutant Release Inventory reported greater reductions of those sub- stances over time than of other substances. There is also anecdotal evidence that significant reductions by individual sources can be attributed to regulation. Moreover, as with 33/50 and TRI, voluntary reductions may have been driven less by the positive publicity associated with the ARET challenge than the nega-
274 CHALLENGESIN EVALUATING VOLUNTARY ENVIRONMENTALPROG^MS five publicity associated with public reporting of discharges of the half of the ARET chemicals covered by the NPRI. Finally, the absence of any provisions for third-party verification of firms' own claims of discharge reductions is trou- bling, particularly for the half of ARET chemicals not covered by the regulatory NPRI program (Leiss and Associates, 1996~. NPRI The discussion of the 33/50 and ARET programs notes the potentially con- founding influence of toxic pollutant release inventories in both countries. Firms may have been motivated less by the positive incentives in the form of public recognition offered by the voluntary challenge programs as discussed earlier, and motivated more by negative incentives (shame) generated by the pollutant release inventories that underpinned them. Such inventories have increased in popularity since the EPA established its TRI in 1988. The Canadian government created its NPRI in 1993, and similar inventories have since been established in Australia and the European Union. It is noteworthy that such inventories are not exclusively voluntary programs in that all facilities leaders and laggards alike- are required by law to report their toxic releases to the state. However, the raison d'etre for those regulations compelling disclosure is to promote voluntary action. Pollutant inventories are predicated on the assumption that, armed with more complete information about firms' environmental practices, consumers, workers, and investors will be empowered to use markets and social networks to pressure firms to voluntarily reduce their releases (Herb and colleagues, this volume, Chapter 15~. The impact of such inventories, measured in terms of the least challenging baseline of change relative to the base year, has been highly encouraging. Total releases reported to the TRI declined by 46 percent from 1988 to 1995 (Natan and Miller, 1998~. Releases reported to the Canadian NPRI similarly declined by 36 percent in the first 3 years of the program, from 1993 to 1996. This success has led some commentators to conclude that discharge inventories that prompt voluntary action are more effective than regulation (Gunningham and Grabosky, 1998:64; Organization for Economic Cooperation and Development [OECD], 1996; Fung and O'Rourke, 2000~. However, a more critical look at these inventories raises a number of questions for program evaluation. First, it is important to acknowledge uncertainty about the extent to which the reductions relative to reference years are real (Hearne, 1996~. Although accurate reporting is legally mandated, individual facilities prepare their own re- ports to both TRI and NPRI with minimal oversight by regulators. Indeed, facili- ties are not required to measure their own discharges; they can estimate them using techniques of varying reliability. Thus, an apparent decline in releases from a facility from one year to the next may simply represent adoption of alternative estimation methods. There is also concern that reported reductions often reflect a
KATHRYN HARRISON 275 shell game, in which facilities merely recategorize waste streams so they no longer have to be reported (Natan and Miller, 1998~. This could be especially problemat- ic for the Canadian NPRI, which did not require reporting of recycling and reuse in the first few years of the program and thus may have invited facilities to reclassify their waste streams as "recycling" to evade reporting. There has been a tendency to assume that the reductions revealed by TRI and NPRI have been voluntary responses to market mechanisms (e.g., Tieten- berg, 1998:593; Fung and O'Rourke, 2000~. However, the picture becomes less clear when one attempts to assess the degree to which the observed changes relative to base years are attributable to information dissemination. With respect to the U.S. TRI, few studies have considered the impact of regulatory programs on TRI discharges. Shapiro (1998, 1999) found state regulatory efforts to be among the most important determinants of reductions of TRI releases over time. Similarly, Khanna and Damon (1999) found that liability under the federal Su- perfund legislation and anticipation of new hazardous air pollutant regulations under the U.S. Clean Air Act were among the most significant factors in explain- ing firms' TRI reports for 33/50 program chemicals. Finally, Santos, et al. (1996) found that regulatory compliance was one of the two reasons most fre- quently cited by facilities (the other being employee health) for reduction of their TRI releases and transfers. These studies suggest a need for greater attention to regulation and liability in accounting for the rapid progress apparent in both discharge inventories and the voluntary challenge programs predicated on them. With respect to Canada's NPRI, analyses performed by Harrison and Ant- weiler (2001) are summarized in Figure 16-1. The top line in Figure 16-1 repre- sents the sum of reported releases of all NPRI substances by all facilities that reported to NPRI in one or more of the years between 1993 and 1998. Consis- tent with experience with the TRI, there was a dramatic 36-percent decrease in the first 3 years from 1993 to 1996, though that has since been counteracted somewhat by growth in releases, resulting in an overall reduction of 30 percent from 1993 to 1998. These reductions of both on-site releases and off-site transfers may be un- derstated as a result of a growing number of facilities reporting to NPRI. A1- though some of these are new facilities contributing real increases in waste pro- duction, others are older facilities that either belatedly learned of the requirement to report to NPRI or were affected by a minor adjustment to NPRI reporting requirements in 1995. The exclusion of these older facilities from the totals in previous years thus may understate reductions over time. One way to control for this is to focus on the subset of facilities that reported in both 1993 and 1998 (though this has the disadvantage of excluding genuinely new facilities). As indicated by the middle line in Figure 16-1, these facilities are doing somewhat better than the totals for all facilities. Among this subset of continuous reporters, a relatively small number of facilities account for a substantial fraction of the total reduction in onsite releas-
276 250 225 200 175 150 125 100 75 50 25 o CHALLENGES IN EVALUATING VOLUNTARY ENVIRONMENTAL PROGRAMS AC 1993 1994 1995 1996 1997 1998 FIGURE 16-1 Trends in on-site releases (kilotons). Note: A: total on-site releases from all facilities reporting to NPRI each year. B: total on- site releases from the "continuous reporter" subset of facilities, which reported to NPRI in each year 1993 through 1998. C: total on-site releases from "continuous reporters" ex- cluding the pulp and paper industry and the Kronos facility in Varenne, Quebec. Sources: Harrison and Antwiler (2000~. Reprinted with permission of Centre for Economic and Social Policy, University of British Columbia. es (Harrison and Antweiler, 2001~. Indeed, a single Quebec facility, Kronos Canada, which adopted process changes in response to regulatory enforcement actions undertaken by both the federal and provincial governments in the early l990s (Picard, 1992; Hamilton, 1993), accounts for more than half of the total reductions. In addition to Kronos, many of the facilities that contributed the greatest reductions to NPRI were pulp and paper mills. It is noteworthy that the pulp and paper industry is the only industry that faced new discharge regulations at the national level in Canada during the past decade, and it also was subject to extensive reform of regulations and permits at the provincial level in the early l990s (Harrison, 1996~. Because there are compelling reasons to conclude that reductions by these facilities were driven by regulation (not least among them these industries' resis- tance to behavioral change before regulatory enforcement), the final series in Figure 16-1 reports trends among the continuous reporters, with Kronos and the
KATHRYN HARRISON 277 pulp and paper industry excluded. It is striking that once these most obvious impacts of national regulation are excluded, the impressive 37-percent reduction of on-site releases by continuous reporters evaporates, leaving a net increase of 7 percent. Although we cannot know whether an even greater increase in releas- es might have been observed in the absence of NPRI, this figure does suggest that the dramatic reductions of toxic discharges typically attributed to NPRI are largely the result of traditional regulation, rather than a voluntary response by firms to information disclosure. A final question concerns the impact on human health and the environment of these emissions reductions. Pollutant release inventories publicly report the volume of releases, but do not provide an estimate of the risks such releases might pose to human health or the environment. Harrison and Antweiler (2001) take one step closer to such an evaluation by adjusting reported NPRI releases for toxicity using the EPA's Chronic Human Health Indicators.~7 After adjust- ing for toxicity, the 30-percent reduction in total on-site releases between 1993 and 1998 translates to a 9-percent increase. In other words, although the total weight of releases has declined, the toxicity of those releases apparently has increased, and has done so to a degree that this substitution effect may actually outweigh the benefits of declining releases. These findings, although far from constituting a complete risk assessment, reinforce the call for program evaluations to go beyond mere "outputs" and "outcomes" to consideration of "impacts." CONCLUSIONS The one clear area of consensus among students of voluntary approaches is that there has been too little attention to evaluation of either the economic or environmental benefits of voluntary programs (Storey, 1996; Davies and Ma- zurek, 1996; European Environment Agency, 1997; Beardsley, 1996; National Research Council, 1996; OECD, 1999; Mazurek, this volume, Chapter 13~. In part, this reflects the novelty of voluntary approaches; it is simply too early to assess their effectiveness in many cases. However, it also reflects a pathology of unclear targets and inattention to the kinds of monitoring, verification, and pub- lic reporting needed to support program evaluation. At a minimum, systematic evaluation of any program, whether regulatory, voluntary, or market based, demands a consistent reference year, clear expecta- tions, monitoring and verification of environmental performance, and public re- porting. Some of these characteristics present special challenges for voluntary programs, which all too often have had unclear expectations, inconsistent base- lines, and few if any monitoring and reporting requirements. In contrast, regula- tory programs typically have quite explicit and consistent standards, self-moni- toring requirements, and provisions for independent compliance monitoring by the state. Nonetheless, these characteristics could in theory become standard practice for all government-sponsored voluntary programs as well. Nongovern-
278 CHALLENGESIN EVALUATING VOLUNTARY ENVIRONMENTALPROGRAMS mental programs present a greater challenge, because private-sector sponsors may not be as inclined to conduct rigorous and public evaluations. Moreover, many nongovernmental programs are environmental management systems, which specify objectives in the form of management "inputs" rather than environmen- tal performance "outputs." However, if participants in such programs seek ei- ther recognition or concessions from regulators, it is reasonable for the state to establish comparable expectations as for government-sponsored voluntary and regulatory programs. Of course, program evaluation also requires political will and administrative resources. The greater effort made to evaluate the voluntary energy conserva- tion programs in the 1980s may reflect the fact that many of those programs were predicated on government subsidies. With less public funding at stake (recognition is cheap after all), there simply may be less incentive to allocate resources to evaluation today, even though the opportunity costs of misdirecting scarce resources to a less effective policy instrument may be substantial. Ironi- cally, the same impulse to reform regulation to lessen the burden on regulated interests that has given rise to voluntary programs simultaneously may have rendered evaluation of the effectiveness of those new programs more difficult. For instance, in an effort to minimize the burden on the private sector, the U.S. Paperwork Reduction Act of 1995 establishes a quite elaborate process of Federal Register notices and Office of Management and Budget approval for "informa- tion collection requests" should an agency want to contact 10 or more firms or individuals. The issue of establishing a baseline for evaluation also remains a special challenge for voluntary programs. In the context of the literature on program evaluation, with voluntary programs we typically confront "quasi-experiments" involving nonequivalent groups (Cook and Campbell, 1979), in which the chal- lenge is to separate the effects of "treatment" (i.e., the voluntary program) from differences resulting from lack of comparability between the "treated" and "un- treated" groups. The tendency to assess environmental progress only relative to reference years almost certainly exaggerates program effectiveness, because some fraction of improvements typically would be attributable to market incen- tives and/or concurrent regulatory requirements in the absence of the voluntary program (to say nothing of the scenario in which the reference year is several years prior to the launch of the voluntary program). The problem is that devel- opment of a "business-as-usual" scenario in the absence of the voluntary pro- gram requires knowledge of what factors are motivating business behavior. Some recent studies have undertaken extensive interviews with program participants to address this issue.~9 Leverage also may be gained by considering trends prior to introduction of the program, looking for the impact of concurrent governmental programs, and using regression techniques to control for other factors that might explain self-selection.20 Ideally, new programs could be introduced (perhaps initially at a pilot scale) in which a randomly assigned control group is ineligi-
KATHRYN HARRISON 279 ble for participation. Although none of these approaches offers a panacea, the risk of misallocating scarce resources to ineffective programs surely justifies greater effort. NOTES 1 Although such contracts are legally binding, they can still be considered voluntary agree- ments because parties enter into the contract voluntarily. 2 For reviews of European experience, see European Environment Agency (1997), Borkey and Levesque (1998), and Rennings et al. (1997). 3 Mazurek's chapter in this volume (Chapter 13) and EPA (2001) both offer excellent over- views of U.S. challenge and information-based programs. 4 See Ehrenfeld, Response to Talking with the Donkey: Cooperative Approaches to Environ- mental Protection, and Harrison, A Response to Professor Ehrenfeld, Journal of Industrial Ecology, online letters to the editor, at http://www.yale.edu/jie. 5 Consistent with a close relationship between business and government, Dietz and Rycroft (1987) reported considerable career mobility between government and industry, but almost none between government and environmental nongovernmental organizations. 6 On the importance of the distinction between negotiation of goals versus means, see Glachant (1994). 7 For instance, Nash and Ehrenfeld (1997) evaluate environmental management systems against the criterion of whether they foster "cultural change" within a firm. 8 It is noteworthy that there is no requirement to make Responsible Care compliance audits public in the US program (Nash and Ehrenfeld, 1997). Indeed, the U.S. American Chemistry Coun- cil leaves it to individual member companies to define what constitutes "full implementation" for their own circumstances (Mazurek, 1998). 9 The EPA Partners for the Environment program's summary of benefits offers an example (see http://www.epa.gov/partners/partnerships.html). It is also noteworthy that the figures presented are based on self-reporting by individual "partners" and, in light of the discussion to follow, that no effort has been made to assess which of those benefits were prompted by the program or whether some fraction might have occurred even in the absence of the program in question. 10 Among these, second-order "green" pressures from investors, lenders, and insurers can be distinguished from first-order green pressures from customers, investors, and workers, in that sec- ond-order market pressures depend on the existence of first-order ones. However, second-order market pressure from investors, lenders, and insurers that is motivated by fear of civil liability or regulatory costs, rather than consumer or worker reactions, would fall in the third category of busi- ness motive, regulatory threat. 11 In contrast, factors such as cost savings, customer requirements, and public pressure were all cited by less than half the respondents (KPMG, 1994). 12 The analysis presented here was influenced significantly by the approach taken by Davies and Mazurek (1996). 13 The 1990 Clean Air Act Amendments also mandated regulation of discharges of volatile organic compounds (in order to achieve ground-level ozone objectives) and hazardous air pollutants, both of which could be expected to cover all 17 33/50 chemicals. 14 ARET reports total discharges of 39.4 tonnes in participants' various base years, 26.9 tonnes in 1993, and 13 tonnes in 1998. Thus, at least 12.5 of the 26.4 tonnes of reductions were achieved prior to the launch of the program in early 1994 (ARET, 2000). 15 The ARET program reports that 26 percent of base year emissions are CEPA "schedule 1" substances (ARET, 2000). The 11-percent figure was calculated using ARET base year data for the following substances on "priority substance list 2": chloroform, ethylene oxide, formaldehyde, acetaldehyde, 1,3-butadiene, phenol, and acrylonitrile.
280 CHALLENGES IN EVALUATING VOLUNTARY ENVIRONMENTAL PROGRAMS 16 Gallon (1998) notes that the 90 percent reduction in sulfur dioxide emissions claimed by INCO's Sudbury smelter under the ARET program were legally mandated. 17 Such an adjustment still does not take into account patterns of dilution or human exposure leading to chronic human health impacts, nor acute impacts on human health or environmental impacts. 18 Although such programs do not legally require participation, compliance with monitoring and reporting requirements can be made a condition for attaining program incentives (whether recogni- tion or regulatory relief). 19 See, for example, information about the National Database on Environmental Management Systems (IDEMS) project (http://www.eli.org/isopilots.htm). 20 Obstacles to causal inference and statistical techniques that attempt to address those problems in the case of quasi-experiments involving nonequivalent groups are discussed in Chapters 3 and 4 of Cook and Campbell (1979). REFERENCES Andrews, R.N.L. 1998 Environmental regulation and business 'self-regulation.' Policy Sciences 31: 177- 197. Accelerated Reduction/Elimination of Toxics 2000 Environmental Leaders 3 Update. Ottawa: ARET. Arora, S., and T.N. Cason 1996 Why do firms volunteer to exceed environmental regulations? Understanding participa- tion in EPA's 33/50 Program. Land Economics 72(4):413-432. Ayres, I., and J. Braithwaite 1992 Responsive Regulation: Transcending the Deregulation Debate. New York: Oxford Beardsley, D. University Press. 1996 Incentives for Environmental Improvement: An Assessment of Selected Innovative Pro- grams in the States and Europe. Washington, DC: Global Environmental Management Initiative. Borkey, P., and F. Levesque 1998 Voluntary Approaches for Environmental Protection in the European Union. Paris: Organization for Economic Co-operation and Development. Cook, Thomas D., and D.T. Campbell 1979 Quasi-Experimentation:DesignandAnalysisIssuesforFieldSettings. Boston: Hough- ton Mifflin. Davies, J.C., and J. Mazurek 1996 Industry Incentives for Environmental Improvement: Evaluation of U.S. Federal Initia- tives. Unpublished manuscript prepared for the Global Environmental Initiative, Wash- ington, DC. Dietz, T., and R.W. Rycroft 1987 The Risk Professionals. New York: Russell Sage Foundation European Commission 1996 Communication from the Commission to the Council and the European Parliament on Environmental Agreements. COM(96) 561. European Commission Directorate 1997 Study on Voluntary Agreements Concluded Between Industry and Public Authorities in the Field of the Environment. General III.01 - Industry. Unpublished final report, Brus- sels, Belg., January.
KATHRYN HARRISON European Environment Agency 281 1997 Environmental Agreements: Environmental Effectiveness. Copenhagen, Den.: European Environment Agency. Fung, A., and D. O'Rourke 2000 Reinventing environmental regulation from the grassroots up: Explaining and expanding the success of the tonics release inventory. Environmental Management 25:115-127. Gallon, G. 1998 Accuracy is optional in reporting voluntary success. Alternatives 24:12. Gibson, R.B. 1999 Conclusion. In Voluntary Initiatives: The New Politics of Corporate Greening, R. Gibson, ed. Peterborough, Ontario, Can.: Broadview. Glachant, M. 1994 The setting of voluntary agreements between industry and government: Bargaining and efficiency. Business Strategy and the Environment 3(2):43-49. Gormley, W.T. 2000 Environmental Performance Measures in a Federal System. Unpublished paper pre- sented at the Annual Meeting of the Association for Public Policy and Management. Gunningham, N., and P. Grabosky 1998 Smart Regulation: Designing Environmental Policy. New York: Oxford. Hamilton, G. 1993 Kronos, two executives charged with polluting St. Lawrence. The Gazette, March 9, p. A4. Harrison, K. 1995 Is cooperation the answer? Canadian environmental enforcement in comparative con- text. Journal of Policy Analysis and Management. 14:221-244. 1996 The regulator's dilemma: Regulation of pulp mill effluents in the Canadian federal state. Canadian Journal of Political Science 29: 469-496. 1998 Talking with the donkey: Cooperative approaches to environmental protection. Journal of Industrial Ecology 2(3):51-71. Harrison, K., and W. Antweiler 2001 Information Dissemination vs. Environmental Regulation: The View from Canada's National Pollutant Release Inventory. Working Paper. Vancouver: Centre for Research on Economic and Social Policy, University of British Columbia. Hearne, S.A. 1996 Tracking tonics: Chemical use end the public's 'right-to-know.' Environment 38:5-9 28-34. Keating, K.M. 1989 Self-selection: Are we beating a dead horse? Evaluation and Program Planning 12: 137- 142. Khanna, M., and L.A. Damon 1999 EPA's voluntary 33/50 program: Impact on toxic releases and economic performance of firms. Journal of Environmental Economics and Management 37: 1-25. KPMG 1994 Canadian Governmental Management Survey. Toronto, Can.: KPMG. Leiss W. and Associates Mazurek, J. 1996 Lessons Learned from ARET: A Qualitative Survey of Perceptions of Stakeholders. Final Report. Working Paper Series 96-4. Ontario, Can.: Environmental Policy Unit, School of Policy Studies, Queen's University. 1998 The Use of Unilateral Agreements in the United States: The Responsible Care Initia- tive. ENV/EPOC/GEEI(98)25/FINAL. Paris: Organization for Economic Co-operation and Development.
282 CHALLENGES IN EVALUATING VOLUNTARY ENVIRONMENTAL PROGRAMS Nash, J., and J. Ehrenfeld 1997 Codes of environmental management practice: Assessing their potential as a tool for change. Annual Review of Energy and Environment 22:487-535. Natan, T., Jr., and C.G. Miller 1998 Are toxic release inventory reductions real? Environmental Science and Technology 3(8):368A-374A. National Research Council 1996 Fostering Industry-Initiated Environmental Protection Efforts. Washington, DC: Na- tional Academy Press. Organization for Economic Co-operation and Development Storey, M. 1996 Demand Side Efficiency: Voluntary Agreements with Industry. Annex I Expert Group on the UN FCCC, supported by the Organization for Economic Co-operation and De- velopment and IEA, December. Tietenberg, T. 1998 Disclosure strategies for pollution control. Environmental and Resource Economics 1 1 :587-602. US Environmental Protection Agency 1999 33/SO Program: The Final Record. EPA-745-R-99-004. Washington, DC: U.S. Envi- ronmental Protection Agency. 2001 The United States Experience with Economic Incentives for Protecting the Environ- ment. EPA-240-R-01-001. Washington, DC: U.S. Environmental Protection Agency. Weinstein, R., R. Scott, and C. Jones 1989 Measurement of 'free-riders' in energy conservation programs. Evaluation and Pro- gram Planning 12:121-130. 1996 Pollutant Release and Transfer Registers (PRTRS): A Tool for Environmental Policy and Sustainable Development. Guidance Manual for Governments. OCDEIGD(96)32. Paris: Organization for Economic Co-operation and Development. 1999 Voluntary Approaches for Environmental Policy: An Assessment. Paris: Organization for Economic Co-operation and Development. O'Toole, L.J., Jr., C. Yu, J. Cooley, and G. Cowie 1997 Reducing toxic chemical releases and transfers: Explaining outcomes for a voluntary program. Policy Studies Journal 25(1): 11-26. Picard, A. 1992 Plant to shut down for ignoring clean-up orders. Globe and Mail, May 30, p.A2. Rennings, K., L. Brockmann, and H. Bergmann 1997 Voluntary agreements in environmental protection: Experiences in Germany and future perspectives. Business Strategy and the Environment 6:245-263. Santos, S.L., V.T. Covello, and D.B. McCallum 1996 Industry response to SARA Title III: Pollution prevention, risk reduction, and risk communication. Risk Analysis 16:57-66. Shapiro, M. 1998 The Power of Information: Community Exposure and the Efficacy of State Information Regulation Policy. Unpublished paper presented at the Annual Meeting of the Ameri- can Political Science Association, New York, October. 1999 Equity and Information: Risks from Toxic Chemicals, Environmental Justice, and Infor- mation Regulation Policy. Unpublished paper presented at the Annual Meeting of the Association for Public Policy Analysis and Management, Washington, DC, November.