National Academies Press: OpenBook
« Previous: 5 The Market for Certified Products
Suggested Citation:"6 Measuring Success." National Research Council. 2010. Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12805.
×

6
Measuring Success

It is difficult for any current certification program to claim “success” in terms of demonstrating significant improvements in human well-being or environmental conditions. This appears to be a major deficiency of certification programs. Participants noted the dearth of peer-reviewed analyses of individual programs and of the field as a whole.1 Without more rigorous analysis of what is working on the ground, what has not worked, and why, it is difficult to envision these voluntary certification programs being widely adopted and truly transforming markets. In particular, many participants suggested the need for improvements in the baselines for measurement, the impacts measured, and the scale at which programs are analyzed. It will also be important to take a look back at experience to date, to better unpack some of these unintended consequences (positive and negative) that typically go unmeasured.

DEVELOPING A BASELINE

One fundamental challenge for most certification programs is that, despite the fact that they are designed to address specific problems of poorly regulated production practices, the baseline for measuring progress is rarely being defined. Existing conditions are described in general terms and are not always adapted for regional differentiation. There are exceptions, of

1

This is a rapidly expanding field, though, and Rainforest Alliance provides a compendium of recent analyses, available at http://www.rainforest-alliance.org/resources.cfm?id=research_analyses.

Suggested Citation:"6 Measuring Success." National Research Council. 2010. Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12805.
×

course, particularly for custom-tailored, performance-based standards (ISO or LEED for buildings) but in general, these voluntary standards systems have emphasized changes in production practice, and their baselines and performance measures relate to this. The most straightforward way to measure success, then, has been to use existing market share or number of compliant producers as a baseline, and focus on expanding the program’s reach.

As some participants noted, most existing programs would have a difficult time backing up their claims of success (in terms of on-the-ground sustainable outcomes) because little is actually being measured, especially at scales of importance to sustainability, namely ecosystems and communities. Even if the particular standards are considered rigorous and based on sound science, certification programs rely heavily on their “theory of change” with the belief that, by changing certain practices through prescriptive guidelines, production will become more sustainable. However, the programs rarely conduct baseline assessments against which they could measure improvements in performance. Doing so would be a valuable way to demonstrate how effective, and efficient, certification programs are at enhancing positive outcomes, relative to alternative approaches. If the majority of compliant producers were top performers to begin with (as has been the case with several leading standards), this begs the question as to whether alternative approaches could have more “bang for the buck.” As some participants noted, though, a direct comparison of approaches would be difficult to carry out, akin to conducting an experiment without being able to isolate variables.

Participants pointed out that success is still not well defined in any sector or particular certification scheme. Defining sustainability for each and every sector would be a daunting task, and would seem counterproductive for individual certification programs to do. Most programs have a notion of the direction they must head, but as many participants emphasized, the bar will likely need to be raised and continually adjusted as programs mature and knowledge accrues. Perhaps more importantly, even when programs are directionally correct, they may need to expend more effort determining their starting point to help assess how far and how fast they will need to progress to achieve sustainable outcomes.

Data availability is a critical limiting factor. As was emphasized at other points throughout the workshop, outside observers would like to see data-driven approaches. Supporters of certification schemes, however, generally want to see on-the-ground changes. With resources being limited, it is difficult to apportion a large amount of up front costs to field work and data gathering before a certification program even commences. Data collection and additional bookkeeping is often cited as a significant transaction cost for producers participating in a certification program. One participant

Suggested Citation:"6 Measuring Success." National Research Council. 2010. Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12805.
×

emphasized, though, that this would be an area where local people and institutions could be meaningfully engaged.

By recognizing that certification is but one intervention that can take place, and that it will necessarily be interacting with other forces, it may be easier to identify an appropriate baseline for performance and determine how and where certification can contribute to raising the bar. Focusing on a select set (e.g., 6-10) of impacts for a given sector may be one way to begin. These might act as guideposts for competing schemes in a sector. Participants noted that there is also a need for wider agreement on metrics—costs of compliance, for example, do not sufficiently account for bottom line impacts or costs to producers.

WHAT TO MEASURE

Certification programs often do include some performance indicators, but as many participants pointed out, these indicators focus on management performance. In other words, they are measuring process changes but typically not their environmental and social outcomes. The primary exception is Fair Trade, which sets economic benefits for producers as its main objective. It should be noted that there is often a substantial time lag between process changes and their intended outcomes or impacts. Still, given the amount of time that some of these programs have been in effect, participants noted the lack of evidence of demonstrated benefits to the environment or to producing communities.

It seems that there is a need for more standards that measure impacts, in terms of positive environmental or social outcomes. Social impacts in particular are not well monitored. One participant noted that some certification bodies are moving in this direction, and giving more thought to measuring their programs’ impacts. If this is done well, and the methodology is transferable, then over time fewer resources would be devoted to individual standard systems sifting through best practices on impact measurement. Some participants advocated for a “trial phase” of a year or more, to start measuring social or environmental results. Progress in impact measurement should also help create incentives for governments, banks, and insurers to adopt the standards. Many participants noted that there would be methodological hurdles, having to do with data availability (since private regulators are not required to disclose their information), the difficulty of making the right comparisons (to uncertified firms, for instance), and the costs of gathering longitudinal datasets.

Programs must be clear about whether they are focused on market penetration or on-the-ground results—this will help define success and determine appropriate metrics. As participants noted, most programs focus on process changes and thus encourage compliance rather than innovation.

Suggested Citation:"6 Measuring Success." National Research Council. 2010. Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12805.
×

On the positive side, this can provide a blueprint or step-by-step approach for producers to improve their practices. However, it does also run the risk of locking producers in to a particular paradigm. In the absence of demonstrated long-term positive outcomes, proscriptive programs may become increasingly difficult to justify, particularly if they are perceived as hampering innovation. By focusing instead on the performance of a standards system (e.g., LEED standards, Box 5), there is room for and even incentives to innovate and exceed standards.

WIDENING THE LENS

Given the amount of activity taking place, it is critical that we begin learning from these ongoing “field experiments” and clarify how standards systems are contributing to sustainable outcomes. Some participants remarked that there is a lack of peer-reviewed, analytical literature on certification programs, specifically with regard to their impacts. This is not surprising given that, as many participants pointed out, these programs are implemented by people who care deeply about the subject and may find it difficult to face facts about what works and what does not.

One participant shared the experience of commissioning a peer-reviewed analysis of a new program, so that the organization could determine how effective its investments had been in terms of promoting more sustainable outcomes in its sector. The analysis revealed that, despite spending over $100 million on interventions under the program, it was not achieving the desired results and might in fact be counterproductive. This is difficult information to accept, but it did allow the organization to retool the program and make wiser investments in the future.

Existing certification programs tend to limit their scopes to a particular segment of a market or sector. This is not unreasonable, since the programs are experimenting with new approaches, requiring some amount of trial and error, and ultimately being traceable back to the intervention. The programs are a response to specific problems, usually on the production end of the value chain. However, this is distinct from problem-driven approaches to sustainability, which would focus more broadly on the entire value chain, or the production-consumption system (e.g., Lebel and Lorek, 2008).

As a number of participants pointed out, certification programs have not been designed to be research projects. They are practical responses to deficiencies of prevailing norms in a given industry. At best, they are providing some certainty that a particular certified product was produced and sourced responsibly and therefore did not contribute to environmental degradation or harm social well-being. In other words, these programs are not designed to “end” the big problems of unsustainable production and consumption.

Suggested Citation:"6 Measuring Success." National Research Council. 2010. Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12805.
×

Still, many of the stakeholders engaged in certification programs do have broader interests in promoting sustainability and addressing these large challenges. It is important to consider, for example, the complexity of manufactured or processed goods. Illegally harvested wood can be purchased by a factory in China, assembled into office furniture, and then exported to the United States. Manufactured goods may also contain toxic or otherwise harmful components and thus require special attention during and after their useful life. Certification may in fact be a useful tool in communicating more desirable attributes, if there is a market for such. Lifecycle analyses (LCA) have been critical in identifying some of these important features, such as embedded energy, or downstream pollution effects following disposal of a product.

Participants also discussed the social side of sustainability, which is not typically reflected by an LCA. Certification is widely used to communicate that a product was not produced with child labor, or that the producer received a legal minimum wage. Less consideration has gone into how and where certification could be used to also advance workers’ rights, or support the interests of small-scale farmers and other producers. In short, certification and standards systems are not currently being employed to address all of the sustainability challenges associated with a given sector.

The complexity and systemic nature of these challenges requires a range of interventions rather than a silver bullet, but existing certification programs have rarely sought to integrate themselves into more holistic efforts. There are recent instances of converging interests, such as the Sustainable Tourism Council working with the UN on a global standard. Many participants also mentioned alternative approaches, particularly supply chain and global value chain approaches, to fostering sustainability within a given sector. They suggested that focusing on the bottom percentile of producers (i.e., those who would be most “out of compliance” with a certification program) could yield more marked improvements than a comparable investment of resources at the top of the market. By the same token, though, the sheer volume of production by large firms (who are more likely to be near the top of the market) provides leverage that small-scale producers do not. Though large firms often contract out aspects of their production to smaller firms, they at least have a mechanism to, on the one hand, impose requirements on their suppliers, and on the other, certify the chain of custody of component products. Thus, certification supporters may need to evaluate which segment of the production market could yield the greatest impact on environmental and social outcomes.

For any certification program, there is an apparent paradox between setting the bar high in terms of compliance, and lowering the bar to bring less compliant producers on board and increase the market share. However, the programs do not have any authority to make noncompliant producers

Suggested Citation:"6 Measuring Success." National Research Council. 2010. Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12805.
×

BOX 5

Green Building: Leadership in Energy and Environmental Design

The LEED® (Leadership in Energy and Environmental Design) green building certification system is a feature-oriented certification program that awards buildings points for satisfying specified green building criteria. The six major environmental categories of review include Sustainable Sites, Water Efficiency, Energy and Atmosphere, Materials and Resources, Indoor Environmental Quality and Innovation and Design. Certified, Silver, Gold, and Platinum levels of LEED green building certifications are awarded based on the total number of points earned within each LEED category. The first LEED Pilot Project Program, known as LEED Version 1.0 was launched at the U.S. Green Building Council (USGBC) Membership Summit in August 1998. After extensive modifications, the LEED Green Building Rating System Version 2.0 was released in March 2000. This rating system is now called LEED for New Construction and Major Renovations, or LEED for New Construction.

As LEED has evolved and matured, the program has undertaken new initiatives to address the many different stages and sectors of the U.S. building market aside from LEED for New Construction. In total, the USGBC now oversees nine certification initiatives, involving:

  • New Construction (NC)

  • Existing Buildings Operation and Maintenance (O&M)

  • Commercial Interiors (CI)

  • Core and Shell (CS)

  • Homes

  • Schools

  • Retail (in pilot)

  • Healthcare (in pilot)

  • Neighborhood Development (in pilot)

In 2007, LEED Registrations and Certifications doubled compared to the previous six years and in 2008 they doubled those of the previous seven years. LEED New Construction Registrations and Certifications continued their growth, but not at the doubling rate. The Core and Shell system grew 13-fold compared with 2006, while LEED for Existing Buildings Operations & Maintenance expanded nearly 20-fold (Watson, 2008).

LEED NC certified projects represented 5.8 percent of new U.S. commercial construction and new registrations represented approximately 30 percent of the market in 2008 (Watson, 2008). 1,082 new buildings have been certified as of July 2008. California leads the United States with 129 total buildings certified in the new construction category. Pennsylvania follows with 69 certified buildings (DOE, 2008).

In 2004, LEED began to certify the sustainability of ongoing operations of existing commercial and institutional buildings. This program is open to owners and operators of existing office and retail buildings, institutional buildings, hotels and residential buildings of four or more habitable stories. Already, the floor area of new registrations has nearly caught up to that of LEED NC, though as a percent of the annual addressable market, certifications remain insignificant.

Also beginning in 2004, LEED started to certify high-performance green interiors that are healthy, productive places to work; less costly to operate

Suggested Citation:"6 Measuring Success." National Research Council. 2010. Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12805.
×

and maintain; and have a reduced environmental footprint. This program was designed to work hand-in-hand with the LEED for Core & Shell, which became available in July 2006. This program covers base building elements such as structure, envelope and the HVAC system, and like the LEED Interior program, it establishes green building criteria that can be used by developers, building owners, and tenants.

These commercial programs have a market share approaching 12 percent of new commercial starts, though they lag significantly behind LEED NC in submarket share and absolute terms. Certified LEED CS projects represent only about 0.5 percent of new construction starts (Watson, 2008).

In December 2007, LEED developed criteria for individual homes, to promote the design and construction of high-performance green residences. Also in 2007, LEED developed a special rating for new construction and major renovations to K-12 school facilities. This new rating system addresses issues such as classroom acoustics, mold prevention, and environmental site assessment.

LEED has continued to expand into new and differentiated markets. It has developed several pilot programs, often in close collaboration with other partners. LEED Retail is currently creating criteria to recognize the unique nature of the retail environment and to addresses the different types of spaces that retailers need for their distinctive product lines. LEED’s Healthcare pilot is similarly assessing the unique needs of the healthcare market. This program represents a culmination of four years of close collaboration between the Green Guide for Healthcare (GGHC) and USGBC, addressing issues such as increased sensitivity to chemicals and pollutants, traveling distances from parking facilities, and access to natural spaces.

Finally, LEED is attempting to take its design criteria to the neighborhood scale, integrating the principles of smart growth, urbanism and green building into the first national system for neighborhood design. This program is a collaboration among USGBC, the Congress for the New Urbanism and the Natural Resources Defense Council. The post-pilot version of the rating system, which will be available to the public, was launched in 2009.

Though the USGBC has been successful in partnering with other organizations, and even references existing certification programs (namely ENERGY STAR) in its own criteria, it still faces competition in a number of its markets. ENERGY STAR focuses exclusively on energy performance, but since this is such an important aspect of building performance, it has become a leading competitor to LEED for new buildings, commercial and residential structures, as well as schools and healthcare facilities. As the USGBC acknowledges, there are more than 70 highly regarded local or regional green home building programs in the U.S.—LEED attempts to distinguish itself by its ability to certify leaders in green building, and help builders (or owners) differentiate their work.

Internationally, LEED is active in dozens of countries. However, its popularity has also led to countries developing their own criteria, tailored to their regional conditions and development goals. LEED itself continues to evolve as well. LEED v3 debuted in April 2009 as the next major evolution of the existing LEED rating systems for commercial buildings. LEED 2009 incorporates regional credits, extra points that have been identified as priorities within a project’s given environmental zone. This will increase the importance of green building as a means of contributing immediate and measurable solutions toward energy independence, climate change mitigation, and other global priorities.

Suggested Citation:"6 Measuring Success." National Research Council. 2010. Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12805.
×

either improve or disappear. Instead, they are relying on the market to convey this authority; without strong demand for certified products, the programs have tended to focus on incentivizing and rewarding top performers. Standards systems could be more widely applied in value chain approaches, as one component of a broad effort to improve performance within a sector. Or as one participant put it, certification programs might be more useful as a yardstick than as a height requirement. Still, there will inevitably be a trade-off between raising the floor and, for example, preserving marginal, low-standard producers as a matter of social policy.

MORE ANALYSIS OF UNINTENDED CONSEQUENCES

Sustainability as a general concept is still not well understood, and the impacts of certifying sustainable products can be far flung. General concepts of sustainability do not provide sufficient guidance on specific questions, e.g., what is a fair price for sustainably harvested Senegalese octopus? While more attention could be paid to measuring a program’s intended impacts, there also seems to be a need for careful consideration of the unintended consequences, positive and negative. Some of these impacts have been alluded to, such as the potential hampering of innovation, or the burdensome costs for small-scale producers. Still, these are generally anecdotal rather than evidence based. When there are unanticipated impacts, we need to understand if they are caused specifically by certification as an approach, or by more systemic challenges.

One of the more comprehensive studies of certification in a sector is the Committee on Sustainable Assessment (COSA) report on sustainability initiatives in the coffee industry (Giovannucci and Potts, 2008). COSA put forward what they considered to be a scientifically credible framework to assess the impacts that standards systems were having on environmental, economic, and public welfare issues. The COSA approach is still in its pilot phase, but some early findings, like the tendency of higher compliance costs to yield higher net income for farmers, appear worthy of further study. Cost distribution in general is an issue that is beginning to attract attention, given that certification programs entail costs that are neither uniformly distributed nor universally recouped.

Geographic coverage is another issue that had until recently not been widely studied. In general, existing certification programs have built their market share not by becoming increasingly global, but by focusing and intensifying efforts in certain markets (e.g., Ellis and Keane, 2008). From a consumer’s standpoint, this may not matter, but it has important implications if these programs intend to increase their coverage. Africa in particular has not been a significant party to certification programs, despite

Suggested Citation:"6 Measuring Success." National Research Council. 2010. Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12805.
×

its prominent role in export markets. It is unclear what the longer-term implications of such a trend might be.

Finally, in thinking through the lifecycle of a certification program itself, some participants remarked that most programs are not giving due consideration to their long-term or exit strategies. The existing market share of some certification programs has been dependent upon subsidies from the donor community, but this is not often acknowledged. These subsidies are not likely to continue into perpetuity, and it remains to be seen whether any current system is ready to stand on its own, supported primarily by the market.

Suggested Citation:"6 Measuring Success." National Research Council. 2010. Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12805.
×

This page intentionally left blank.

Suggested Citation:"6 Measuring Success." National Research Council. 2010. Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12805.
×
Page 47
Suggested Citation:"6 Measuring Success." National Research Council. 2010. Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12805.
×
Page 48
Suggested Citation:"6 Measuring Success." National Research Council. 2010. Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12805.
×
Page 49
Suggested Citation:"6 Measuring Success." National Research Council. 2010. Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12805.
×
Page 50
Suggested Citation:"6 Measuring Success." National Research Council. 2010. Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12805.
×
Page 51
Suggested Citation:"6 Measuring Success." National Research Council. 2010. Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12805.
×
Page 52
Suggested Citation:"6 Measuring Success." National Research Council. 2010. Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12805.
×
Page 53
Suggested Citation:"6 Measuring Success." National Research Council. 2010. Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12805.
×
Page 54
Suggested Citation:"6 Measuring Success." National Research Council. 2010. Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12805.
×
Page 55
Suggested Citation:"6 Measuring Success." National Research Council. 2010. Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12805.
×
Page 56
Next: 7 Optimizing Certification as a Tool »
Certifiably Sustainable?: The Role of Third-Party Certification Systems: Report of a Workshop Get This Book
×
Buy Paperback | $48.00 Buy Ebook | $38.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Consumption of goods and services represents a growing share of global economic activity. In the United States, consumption accounts for more than two-thirds of gross domestic product. This trend of increasing consumption has brought with it negative consequences for the environment and human well-being. Global demand for energy, food, and all manner of goods is on the rise, putting strains on the natural and human capital required to produce them. Extractive industries and production processes are prominent causes of species endangerment. Modern economies are underpinned by substantial energy consumption, a primary contributor to the current climate crisis. Expanding international trade has led to many economic opportunities, but has also contributed to unfair labor practices and wealth disparities.

While certain processes have improved or become more efficient, and certain practices have been outlawed or amended, the sheer scale of global consumption and its attendant impacts continue to be major challenges we face in the transition to sustainability. Third-party certification systems have emerged over the last 15 years as a tool with some promise. There has been anecdotal evidence of success, but to date the overall impact of certified goods and services has been small. Moreover, definitions of sustainable vary across sectors and markets, and rigorous assessments of these programs have been few and far between.

In order to take a step in learning from this field of practice, the National Academies' Science and Technology for Sustainability Program held a workshop to illuminate the decision making process of those who purchase and produce certified goods and services. It was also intended to help clarify the scope and limitations of the scientific knowledge that might contribute to the economic success of certified products. The workshop, summarized in this volume, involved presentations and discussions with approximately 40 invited experts from academia, business, government, and nongovernmental organizations.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!