Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 47
6 Measuring Success It is difficult for any current certification program to claim “success” in terms of demonstrating significant improvements in human well-being or environmental conditions. This appears to be a major deficiency of certifica- tion programs. Participants noted the dearth of peer-reviewed analyses of individual programs and of the field as a whole.1 Without more rigorous analysis of what is working on the ground, what has not worked, and why, it is difficult to envision these voluntary certification programs being widely adopted and truly transforming markets. In particular, many participants suggested the need for improvements in the baselines for measurement, the impacts measured, and the scale at which programs are analyzed. It will also be important to take a look back at experience to date, to better unpack some of these unintended consequences (positive and negative) that typically go unmeasured. DEVELOPING A BASELINE One fundamental challenge for most certification programs is that, despite the fact that they are designed to address specific problems of poorly regulated production practices, the baseline for measuring progress is rarely being defined. Existing conditions are described in general terms and are not always adapted for regional differentiation. There are exceptions, of 1 This is a rapidly expanding field, though, and Rainforest Alliance provides a com- pendium of recent analyses, available at http://www.rainforest-alliance.org/resources. cfm?id=research_analyses. 

OCR for page 47
8 certIfIaBlY sustaInaBle? course, particularly for custom-tailored, performance-based standards (ISO or LEED for buildings) but in general, these voluntary standards systems have emphasized changes in production practice, and their baselines and performance measures relate to this. The most straightforward way to measure success, then, has been to use existing market share or num- ber of compliant producers as a baseline, and focus on expanding the program’s reach. As some participants noted, most existing programs would have a dif- ficult time backing up their claims of success (in terms of on-the-ground sustainable outcomes) because little is actually being measured, especially at scales of importance to sustainability, namely ecosystems and commu- nities. Even if the particular standards are considered rigorous and based on sound science, certification programs rely heavily on their “theory of change” with the belief that, by changing certain practices through pre- scriptive guidelines, production will become more sustainable. However, the programs rarely conduct baseline assessments against which they could measure improvements in performance. Doing so would be a valuable way to demonstrate how effective, and efficient, certification programs are at enhancing positive outcomes, relative to alternative approaches. If the majority of compliant producers were top performers to begin with (as has been the case with several leading standards), this begs the question as to whether alternative approaches could have more “bang for the buck.” As some participants noted, though, a direct comparison of approaches would be difficult to carry out, akin to conducting an experiment without being able to isolate variables. Participants pointed out that success is still not well defined in any sec- tor or particular certification scheme. Defining sustainability for each and every sector would be a daunting task, and would seem counterproductive for individual certification programs to do. Most programs have a notion of the direction they must head, but as many participants emphasized, the bar will likely need to be raised and continually adjusted as programs mature and knowledge accrues. Perhaps more importantly, even when programs are directionally correct, they may need to expend more effort determining their starting point to help assess how far and how fast they will need to progress to achieve sustainable outcomes. Data availability is a critical limiting factor. As was emphasized at other points throughout the workshop, outside observers would like to see data- driven approaches. Supporters of certification schemes, however, generally want to see on-the-ground changes. With resources being limited, it is dif- ficult to apportion a large amount of up front costs to field work and data gathering before a certification program even commences. Data collection and additional bookkeeping is often cited as a significant transaction cost for producers participating in a certification program. One participant

OCR for page 47
 measurIng success emphasized, though, that this would be an area where local people and institutions could be meaningfully engaged. By recognizing that certification is but one intervention that can take place, and that it will necessarily be interacting with other forces, it may be easier to identify an appropriate baseline for performance and determine how and where certification can contribute to raising the bar. Focusing on a select set (e.g., 6-10) of impacts for a given sector may be one way to begin. These might act as guideposts for competing schemes in a sector. Partici- pants noted that there is also a need for wider agreement on metrics—costs of compliance, for example, do not sufficiently account for bottom line impacts or costs to producers. WHAT TO MEASURE Certification programs often do include some performance indicators, but as many participants pointed out, these indicators focus on manage- ment performance. In other words, they are measuring process changes but typically not their environmental and social outcomes. The primary exception is Fair Trade, which sets economic benefits for producers as its main objective. It should be noted that there is often a substantial time lag between process changes and their intended outcomes or impacts. Still, given the amount of time that some of these programs have been in effect, participants noted the lack of evidence of demonstrated benefits to the environment or to producing communities. It seems that there is a need for more standards that measure impacts, in terms of positive environmental or social outcomes. Social impacts in particular are not well monitored. One participant noted that some certifi- cation bodies are moving in this direction, and giving more thought to mea- suring their programs’ impacts. If this is done well, and the methodology is transferable, then over time fewer resources would be devoted to individual standard systems sifting through best practices on impact measurement. Some participants advocated for a “trial phase” of a year or more, to start measuring social or environmental results. Progress in impact measurement should also help create incentives for governments, banks, and insurers to adopt the standards. Many participants noted that there would be method- ological hurdles, having to do with data availability (since private regula- tors are not required to disclose their information), the difficulty of making the right comparisons (to uncertified firms, for instance), and the costs of gathering longitudinal datasets. Programs must be clear about whether they are focused on market penetration or on-the-ground results—this will help define success and determine appropriate metrics. As participants noted, most programs focus on process changes and thus encourage compliance rather than innovation.

OCR for page 47
0 certIfIaBlY sustaInaBle? On the positive side, this can provide a blueprint or step-by-step approach for producers to improve their practices. However, it does also run the risk of locking producers in to a particular paradigm. In the absence of dem- onstrated long-term positive outcomes, proscriptive programs may become increasingly difficult to justify, particularly if they are perceived as ham- pering innovation. By focusing instead on the performance of a standards system (e.g., LEED standards, Box 5), there is room for and even incentives to innovate and exceed standards. WIDENING THE LENS Given the amount of activity taking place, it is critical that we begin learning from these ongoing “field experiments” and clarify how stan- dards systems are contributing to sustainable outcomes. Some participants remarked that there is a lack of peer-reviewed, analytical literature on certification programs, specifically with regard to their impacts. This is not surprising given that, as many participants pointed out, these programs are implemented by people who care deeply about the subject and may find it difficult to face facts about what works and what does not. One participant shared the experience of commissioning a peer-reviewed analysis of a new program, so that the organization could determine how effective its investments had been in terms of promoting more sustainable outcomes in its sector. The analysis revealed that, despite spending over $100 million on interventions under the program, it was not achieving the desired results and might in fact be counterproductive. This is difficult information to accept, but it did allow the organization to retool the pro- gram and make wiser investments in the future. Existing certification programs tend to limit their scopes to a particular segment of a market or sector. This is not unreasonable, since the programs are experimenting with new approaches, requiring some amount of trial and error, and ultimately being traceable back to the intervention. The pro- grams are a response to specific problems, usually on the production end of the value chain. However, this is distinct from problem-drien approaches to sustainability, which would focus more broadly on the entire value chain, or the production-consumption system (e.g., Lebel and Lorek, 2008). As a number of participants pointed out, certification programs have not been designed to be research projects. They are practical responses to deficiencies of prevailing norms in a given industry. At best, they are providing some certainty that a particular certified product was produced and sourced responsibly and therefore did not contribute to environmental degradation or harm social well-being. In other words, these programs are not designed to “end” the big problems of unsustainable production and consumption.

OCR for page 47
 measurIng success Still, many of the stakeholders engaged in certification programs do have broader interests in promoting sustainability and addressing these large challenges. It is important to consider, for example, the complexity of manufactured or processed goods. Illegally harvested wood can be pur- chased by a factory in China, assembled into office furniture, and then exported to the United States. Manufactured goods may also contain toxic or otherwise harmful components and thus require special attention during and after their useful life. Certification may in fact be a useful tool in com- municating more desirable attributes, if there is a market for such. Lifecycle analyses (LCA) have been critical in identifying some of these important features, such as embedded energy, or downstream pollution effects follow- ing disposal of a product. Participants also discussed the social side of sustainability, which is not typically reflected by an LCA. Certification is widely used to communicate that a product was not produced with child labor, or that the producer received a legal minimum wage. Less consideration has gone into how and where certification could be used to also advance workers’ rights, or support the interests of small-scale farmers and other producers. In short, certification and standards systems are not currently being employed to address all of the sustainability challenges associated with a given sector. The complexity and systemic nature of these challenges requires a range of interventions rather than a silver bullet, but existing certification pro- grams have rarely sought to integrate themselves into more holistic efforts. There are recent instances of converging interests, such as the Sustainable Tourism Council working with the UN on a global standard. Many par- ticipants also mentioned alternative approaches, particularly supply chain and global value chain approaches, to fostering sustainability within a given sector. They suggested that focusing on the bottom percentile of producers (i.e., those who would be most “out of compliance” with a certification program) could yield more marked improvements than a comparable invest- ment of resources at the top of the market. By the same token, though, the sheer volume of production by large firms (who are more likely to be near the top of the market) provides leverage that small-scale producers do not. Though large firms often contract out aspects of their production to smaller firms, they at least have a mechanism to, on the one hand, impose require- ments on their suppliers, and on the other, certify the chain of custody of component products. Thus, certification supporters may need to evaluate which segment of the production market could yield the greatest impact on environmental and social outcomes. For any certification program, there is an apparent paradox between setting the bar high in terms of compliance, and lowering the bar to bring less compliant producers on board and increase the market share. However, the programs do not have any authority to make noncompliant producers

OCR for page 47
 certIfIaBlY sustaInaBle? BOX 5 Green Building: Leadership in Energy and Environmental Design The LEED® (Leadership in Energy and Environmental Design) green build­ ing certification system is a feature­oriented certification program that awards buildings points for satisfying specified green building criteria. The six major environmental categories of review include Sustainable Sites, Water Efficiency, Energy and Atmosphere, Materials and Resources, Indoor Environmental Quality and Innovation and Design. Certified, Silver, Gold, and Platinum levels of LEED green building certifications are awarded based on the total number of points earned within each LEED category. The first LEED Pilot Project Program, known as LEED Version 1.0 was launched at the U.S. Green Building Council (USGBC) Membership Summit in August 1998. After extensive modifications, the LEED Green Building Rating System Version 2.0 was released in March 2000. This rating system is now called LEED for New Construction and Major Renovations, or LEED for New Construction. As LEED has evolved and matured, the program has undertaken new ini­ tiatives to address the many different stages and sectors of the U.S. building m arket aside from LEED for New Construction. In total, the USGBC now over­ sees nine certification initiatives, involving: • New Construction (NC) • Existing Buildings Operation and Maintenance (O&M) • Commercial Interiors (CI) • Core and Shell (CS) • Homes • Schools • Retail (in pilot) • Healthcare (in pilot) • Neighborhood Development (in pilot) In 2007, LEED Registrations and Certifications doubled compared to the previous six years and in 2008 they doubled those of the previous seven years. LEED New Construction Registrations and Certifications continued their growth, but not at the doubling rate. The Core and Shell system grew 13­fold com­ pared with 2006, while LEED for Existing Buildings Operations & Maintenance e xpanded nearly 20­fold (Watson, 2008). LEED NC certified projects represented 5.8 percent of new U.S. commercial construction and new registrations represented approximately 30 percent of the market in 2008 (Watson, 2008). 1,082 new buildings have been certified as of July 2008. California leads the United States with 129 total buildings certified in the new construction category. Pennsylvania follows with 69 certified buildings (DOE, 2008). In 2004, LEED began to certify the sustainability of ongoing operations of existing commercial and institutional buildings. This program is open to owners and operators of existing office and retail buildings, institutional buildings, hotels and residential buildings of four or more habitable stories. Already, the floor area of new registrations has nearly caught up to that of LEED NC, though as a percent of the annual addressable market, certifications remain insignificant. Also beginning in 2004, LEED started to certify high­performance green interiors that are healthy, productive places to work; less costly to operate

OCR for page 47
 measurIng success and maintain; and have a reduced environmental footprint. This program was designed to work hand­in­hand with the LEED for Core & Shell, which became available in July 2006. This program covers base building elements such as structure, envelope and the HVAC system, and like the LEED Interior program, it establishes green building criteria that can be used by developers, building owners, and tenants. These commercial programs have a market share approaching 12 percent of new commercial starts, though they lag significantly behind LEED NC in sub­ market share and absolute terms. Certified LEED CS projects represent only about 0.5 percent of new construction starts (Watson, 2008). In December 2007, LEED developed criteria for individual homes, to promote the design and construction of high­performance green residences. Also in 2007, LEED developed a special rating for new construction and major renovations to K­12 school facilities. This new rating system addresses issues such as class­ room acoustics, mold prevention, and environmental site assessment. LEED has continued to expand into new and differentiated markets. It has developed several pilot programs, often in close collaboration with other partners. LEED Retail is currently creating criteria to recognize the unique nature of the retail environment and to addresses the different types of spaces that retailers need for their distinctive product lines. LEED’s Healthcare pilot is similarly assessing the unique needs of the healthcare market. This program represents a culmination of four years of close collaboration between the Green Guide for Healthcare (GGHC) and USGBC, addressing issues such as increased sensitivity to chemicals and pol­ lutants, traveling distances from parking facilities, and access to natural spaces. Finally, LEED is attempting to take its design criteria to the neighborhood scale, integrating the principles of smart growth, urbanism and green building into the first national system for neighborhood design. This program is a col­ laboration among USGBC, the Congress for the New Urbanism and the Natural Resources Defense Council. The post­pilot version of the rating system, which will be available to the public, was launched in 2009. Though the USGBC has been successful in partnering with other organiza­ tions, and even references existing certification programs (namely ENERGY STAR) in its own criteria, it still faces competition in a number of its markets. ENERGY STAR focuses exclusively on energy performance, but since this is such an important aspect of building performance, it has become a leading competitor to LEED for new buildings, commercial and residential structures, as well as schools and healthcare facilities. As the USGBC acknowledges, there are more than 70 highly regarded local or regional green home building programs in the U.S.—LEED attempts to distinguish itself by its ability to certify leaders in green building, and help builders (or owners) differentiate their work. Internationally, LEED is active in dozens of countries. However, its popu­ larity has also led to countries developing their own criteria, tailored to their regional conditions and development goals. LEED itself continues to evolve as well. LEED v3 debuted in April 2009 as the next major evolution of the existing LEED rating systems for commercial buildings. LEED 2009 incorporates regional credits, extra points that have been identified as priorities within a project’s given environmental zone. This will increase the importance of green building as a means of contributing immediate and measurable solutions toward energy independence, climate change mitigation, and other global priorities.

OCR for page 47
 certIfIaBlY sustaInaBle? either improve or disappear. Instead, they are relying on the market to con- vey this authority; without strong demand for certified products, the pro- grams have tended to focus on incentivizing and rewarding top performers. Standards systems could be more widely applied in value chain approaches, as one component of a broad effort to improve performance within a sector. Or as one participant put it, certification programs might be more useful as a yardstick than as a height requirement. Still, there will inevitably be a trade-off between raising the floor and, for example, preserving marginal, low-standard producers as a matter of social policy. MORE ANALySIS OF UNINTENDED CONSEqUENCES Sustainability as a general concept is still not well understood, and the impacts of certifying sustainable products can be far flung. General concepts of sustainability do not provide sufficient guidance on specific questions, e.g., what is a fair price for sustainably harvested Senegalese octopus? While more attention could be paid to measuring a program’s intended impacts, there also seems to be a need for careful consideration of the unintended consequences, positive and negative. Some of these impacts have been alluded to, such as the potential hampering of innovation, or the burdensome costs for small-scale producers. Still, these are gener- ally anecdotal rather than evidence based. When there are unanticipated impacts, we need to understand if they are caused specifically by certifica- tion as an approach, or by more systemic challenges. One of the more comprehensive studies of certification in a sector is the Committee on Sustainable Assessment (COSA) report on sustainability initiatives in the coffee industry (Giovannucci and Potts, 2008). COSA put forward what they considered to be a scientifically credible framework to assess the impacts that standards systems were having on environmental, economic, and public welfare issues. The COSA approach is still in its pilot phase, but some early findings, like the tendency of higher compliance costs to yield higher net income for farmers, appear worthy of further study. Cost distribution in general is an issue that is beginning to attract attention, given that certification programs entail costs that are neither uniformly distrib- uted nor universally recouped. Geographic coverage is another issue that had until recently not been widely studied. In general, existing certification programs have built their market share not by becoming increasingly global, but by focusing and intensifying efforts in certain markets (e.g., Ellis and Keane, 2008). From a consumer’s standpoint, this may not matter, but it has important impli- cations if these programs intend to increase their coverage. Africa in par- ticular has not been a significant party to certification programs, despite

OCR for page 47
 measurIng success its prominent role in export markets. It is unclear what the longer-term implications of such a trend might be. Finally, in thinking through the lifecycle of a certification program itself, some participants remarked that most programs are not giving due consideration to their long-term or exit strategies. The existing market share of some certification programs has been dependent upon subsidies from the donor community, but this is not often acknowledged. These subsidies are not likely to continue into perpetuity, and it remains to be seen whether any current system is ready to stand on its own, supported primarily by the market.

OCR for page 47