Summary

An assessment process is a key interface between science and policy and a crucial mechanism by which science informs policy making. It can establish the importance of an issue, provide an authoritative resolution of policy-relevant scientific questions, demonstrate the benefits of policy options, identify new research directions, and provide technical solutions. As a result of an increasing number of international treaties and national mandates, the number of global change assessments, as well as the resources and the number of scientists dedicated to such assessment activities, is growing. At the same time, a wealth of experience on how to conduct assessments has accumulated. Given the continuing need for assessment activities in the future, it is an opportune time to systematically evaluate the approach and effectiveness of past assessments and learn from the available experience.

In the United States, the Global Change Research Act (GCRA) of 1990 mandates that every four years an assessment be conducted of the impacts of global change on eight areas: the natural environment, agriculture, energy production and use, land and water resources, transportation, human health and welfare, human social systems, and biological diversity. Responding to this mandate, the National Assessment of Climate Change Impacts (NACCI) was carried out during the late 1990s, and a second assessment activity, comprising 21 synthesis and assessment products within the U.S. Climate Change Science Program (CCSP), is currently under way. Given the GCRA’s mandate to provide such assessment efforts at a regular interval, assessment



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Analysis of Global Change Assessments Lessons Learned Summary An assessment process is a key interface between science and policy and a crucial mechanism by which science informs policy making. It can establish the importance of an issue, provide an authoritative resolution of policy-relevant scientific questions, demonstrate the benefits of policy options, identify new research directions, and provide technical solutions. As a result of an increasing number of international treaties and national mandates, the number of global change assessments, as well as the resources and the number of scientists dedicated to such assessment activities, is growing. At the same time, a wealth of experience on how to conduct assessments has accumulated. Given the continuing need for assessment activities in the future, it is an opportune time to systematically evaluate the approach and effectiveness of past assessments and learn from the available experience. In the United States, the Global Change Research Act (GCRA) of 1990 mandates that every four years an assessment be conducted of the impacts of global change on eight areas: the natural environment, agriculture, energy production and use, land and water resources, transportation, human health and welfare, human social systems, and biological diversity. Responding to this mandate, the National Assessment of Climate Change Impacts (NACCI) was carried out during the late 1990s, and a second assessment activity, comprising 21 synthesis and assessment products within the U.S. Climate Change Science Program (CCSP), is currently under way. Given the GCRA’s mandate to provide such assessment efforts at a regular interval, assessment

OCR for page 1
Analysis of Global Change Assessments Lessons Learned leaders and practitioners recognized the value in learning from these and other assessment processes to improve future efforts. Against this background, the CCSP, which coordinates U.S. climate and global change research conducted at 13 government agencies and is responsible for conducting global change assessments for the United States, asked the National Academies to identify lessons learned from relevant past global change assessments at both national and international levels as a guide for future assessment activities. An ad hoc committee, composed of individuals who have studied, participated in, or been users of global change assessments, was convened to prepare this report. To inform its deliberations, the committee met with scholars who have evaluated or participated in assessments, with leaders of past assessments, and with users of assessments. For the report’s conclusions, the committee draws both from existing literature and from its examination of a relatively small but varied selection of global change assessments (Table S.1), each analyzing global change processes that are at least in part driven by human activities. The committee’s recommendations provide general guidance for those who conduct assessments and also, where appropriate, identify specific issues relevant to future CCSP assessment activities. TABLE S.1 The Eight Examples of Assessment Processes Included in the Comparative Analysis Assessment Brief Description Stratospheric Ozone Assessments Prior to the 1987 Montreal Protocol, there were several national (including NRC) and international assessments analyzing ozone-depleting chemicals and the state of the stratospheric ozone layer (WMO 1982, 1986). Following the treaty, a system of expert advisory panels was established to periodically assess the atmospheric science of the ozone layer (WMO 1990a, 1990b, 1992, 1995, 1999, 2003, 2007), the impacts of ozone loss (UNEP 1991a, 1994a, 1998a, 2002a), and the technology and economics of alternatives to ozone-depleting chemicals (UNEP 1991b, 1994b, 1998b, 2002b). Intergovernmental Panel on Climate Change (IPCC) IPCC analyzes scientific and socioeconomic information on climate change and its impacts, and assesses options for mitigation and adaptation. It provides scientific, technological, and socioeconomic findings to the Conference of the Parties to the United Nations Framework Convention on Climate Change (IPCC 1990a,b,c, 1995a,b,c, 2001a,b,c). Global Biodiversity Assessment (GBA) GBA provides a synthesis and analysis of available science on biodiversity to support the work of the UN Convention on Biological Diversity (GBA 1995).

OCR for page 1
Analysis of Global Change Assessments Lessons Learned Assessment Brief Description National Assessment of Climate Change Impacts (NACCI) NACCI was undertaken in response to the Global Change Research Act (1990) to evaluate the impacts of climate change on the United States (NAST 2001). Arctic Climate Impact Assessment (ACIA) Primary objectives were to evaluate and synthesize knowledge and indicators of climate variability, climate change, and ultraviolet radiation in the region; to assess possible impacts of future changes in climate and radiation; and to provide reliable information to both governments and peoples of the region to support policy-making processes (ACIA 2004). Millennium Ecosystem Assessment (MA) MA was designed to answer questions fundamental to various UN conventions dealing with natural resource issues, in particular the consequences of diverse environmental changes on the functioning of ecosystems, including their continuing capacity to deliver services essential to human well-being (MA 2005a,b). German Enquete Kommission on “Preventive Measures to Protect the Earth’s Atmosphere” The Enquete Kommission brings scientists and policy makers together to assess, in this case, the importance and consequences of stratospheric ozone depletion and climate change for Germany among other dimensions of global environmental change (Enquete Kommission 1988, 1991). Synthesis and Assessment Products by the U.S. Climate Change Science Program (CCSP) The 21 assessment products are designed to address the mandate of the Global Change Research Act by considering science and policy issues spanning the range of topics addressed by the CCSP. The first product, on temperature trends in the lower atmosphere, was released in April 2006 (CCSP 2006). ESSENTIAL ELEMENTS OF EFFECTIVE ASSESSMENTS Certain strengths and weaknesses, common to several assessments analyzed by the committee, illuminate critical features of effective assessments. For example, a well-defined mandate and consistent support from the requesters of the assessment contributed importantly to the effectiveness of the Arctic Climate Impact Assessment (ACIA) and the stratospheric ozone assessments, while the process outcome of the Global Biodiversity Assessment (GBA) was impaired by lack of a clear mandate from the target audience. Several assessments benefited significantly from well-articulated, multifaceted, and extensive communication strategies. The ozone assessments were especially effective in providing relevant information for decision-making processes,

OCR for page 1
Analysis of Global Change Assessments Lessons Learned whereas the ACIA was outstanding in the scope of its communications outreach. Other components of effective assessments included superior leadership, extensive and well-designed stakeholder engagement, and a transparent and effective science-policy interface. Perhaps the most common weakness of past assessments has been a discrepancy between the scope of the mandate and the funding provided for the assessment effort. Drawing both on this comparative analysis and on relevant literature, the committee identified 11 essential elements of effective assessments: A clear strategic framing of the assessment process, including a well-articulated mandate, realistic goals consistent with the needs of decision makers, and a detailed implementation plan. Adequate funding that is both commensurate with the mandate and effectively managed to ensure an efficient assessment process. A balance between the benefits of a particular assessment and the opportunity costs (e.g., commitments of time and effort) to the scientific community. A timeline consistent with assessment objectives, the state of the underlying knowledge base, the resources available, and the needs of decision makers. Engagement and commitment of interested and affected parties, with a transparent science-policy interface and effective communication throughout the process. Strong leadership and an organizational structure in which responsibilities are well articulated. Careful design of interdisciplinary efforts to ensure integration, with specific reference to the assessment’s purpose, users needs, and available resources. Realistic and credible treatment of uncertainties. An independent review process monitored by a balanced panel of review editors. Maximizing the benefits of the assessment by developing tools to support use of assessment results in decision making at differing geographic scales and decision levels. Use of a nested assessment approach, when appropriate, using analysis of large-scale trends and identification of priority issues as the context for focused, smaller-scale impacts and response assessments at the regional or local level. The committee concludes that attention to these elements, many of which have been identified in the existing literature, increases the probability that an assessment will be credible, legitimate, and salient (see Box S.1), and therefore will effectively inform both decision makers and other target

OCR for page 1
Analysis of Global Change Assessments Lessons Learned BOX S.1 Definitions of Key Terms Three essential properties of a successful assessment process: Salience relates to an assessment’s ability to communicate with the users whose decisions it seeks to inform and whether the information is perceived as relevant. Credibility addresses the technical quality of information, as perceived by the relevant scientific or other expert communities. Legitimacy concerns the fairness and impartiality of an assessment process, as judged by its users and stakeholders. Global Change Assessments: Global change assessments are collective, deliberative processes by which experts review, analyze, and synthesize scientific knowledge in response to users’ information needs relevant to key questions, uncertainties or decisions. Stakeholders: Stakeholders in the assessment process are defined as all interested and affected parties. Target Audience: Target audience refers to the potential users of assessments. Often, the primary target audience consists of federal government officials who are responsible for the decisions the assessment is intended to inform. However, the target audience may also include state and municipal governments, private-sector users, the public, or intermediaries who function as science translators to decision makers (e.g., media, congressional staff, business associations, environmental organizations). SOURCES: NRC 1996, Social Learning Group 2001a, Clark and Majone 1985, and Ravetz 1971. audiences. In the full set of findings and recommendations presented in Chapter 5, the committee provides guidance for incorporating these elements into future assessments. In this summary, the committee highlights some especially challenging aspects of assessment processes and emerging approaches. These challenges include effectively framing assessments, engaging stakeholders, weighing the benefits of assessments against their opportunity costs, employing nested assessment strategies, and developing decision-support tools.

OCR for page 1
Analysis of Global Change Assessments Lessons Learned FRAMING THE ASSESSMENT A well-formulated mandate is required to ensure that the assessment process is demand-driven and effectively supports a particular set of decisions. As part of the mandate, goals and objectives need to be clearly articu-As part of the mandate, goals and objectives need to be clearly articulated, including the kinds of decisions that the assessment should inform, how the assessment will be implemented, and how progress towards goals will be measured. The assessment’s mandate and goals should be agreed upon in advance by those requesting the assessment and the assessment leaders, and should only be modified during the assessment through a transparent process. In addition, the respective roles of those requesting and those funding the assessment in the scoping process should be clarified in the original guidance document to avoid major discrepancies between the assessment’s mandate, expected results, and available funding. A detailed guidance document specifying those terms of the assessment process will also increase both legitimacy and salience. Recommendation: The leadership of and those requesting assessments should develop a guidance document that provides a clear strategic framework, including a well-articulated mandate and a detailed implementation plan realistically linked to budgetary requirements. The guidance document should specify decisions the assessment intends to inform; the assessment’s scope, timing, priorities, target audiences, leadership, communication strategy, funding, and the degree of interdisciplinary integration; and measures of success. Although CCSP has a mandate under GCRA to conduct assessments, the program lacks a long-term strategic framework for meeting this mandate. Prior to undertaking future assessments, CCSP should clearly express program goals in addition to goals for each assessment, specifying decisions the program intends to inform. A strategic plan comprising overall goals, mandate, and implementation strategy for CCSP assessment activities would enhance the salience, credibility, and legitimacy of future assessments—especially if the plan is accepted at high levels of government as well as within the science agencies and the scientific community. Such an overarching long-term strategic plan for CCSP assessment activities would foster programmatic and funding continuity that could adapt to evolving circumstances and changes in political administration. Recommendation: The CCSP should develop a broad strategic plan for its assessment activities that focuses not only on specific short-term objectives such as preparing the next report or assessment product, but also on longer-term objectives that are in the national interest.

OCR for page 1
Analysis of Global Change Assessments Lessons Learned BALANCING ASSESSMENT BENEFITS WITH OPPORTUNITY COSTS Assessments provide obvious benefits to society by applying sound science to the decision-making process. The research community also benefits from assessments in multiple ways, including establishing the state of knowledge bringing together disciplines that are often isolated, resulting in new lines of investigation and creating new interdisciplinary fields. However, assessments also involve opportunity costs. With an ever-growing number of scientists involved in an increasing number of assessments, time and resources are diverted away from producing new research results. Other unintended consequences include a decreased ability to recruit a balanced pool of high-quality assessment participants and volunteer reviewers, and the diminished impact of an individual assessment if target audiences are overwhelmed with the sheer volume of information from assessments. Given the important contributions of assessments to policy making and to society in general and the growing number of international treaties and national mandates, efficiency considerations become increasingly important to minimize the opportunity cost to the research community. New approaches to assessments might minimize such costs. For example, assessments that are scheduled at regular intervals (such as IPCC and those mandated by the GCRA) could limit their scope to examining only significant new developments and providing succinct summaries of the previous state of knowledge. Thus, each report would build on, rather than duplicate, previous efforts. In addition, assessments conducted at different scales (global, national, or regional) could be nested and run in phases to optimize the ability to build on previous assessments. Similarly, assessments that seek consensus on the underlying science could be phased such that they are completed in time to inform impacts and response assessments. Recommendation: Care is required to make sure the burden of assessments on the scientific community is proportional to the aggregate public benefits provided by the assessment. Alternative modes of participation or changes to the assessment process—such as limiting material in regularly scheduled assessments or running “nested” or phased multiscale assessments—should be considered. As appropriate, U.S. assessments should acknowledge the work of the international community and avoid redundant efforts. IDENTIFYING, ENGAGING, AND RESPONDING TO STAKEHOLDERS The assessment community has recognized the importance of broad engagement of stakeholders—including those who request and fund an

OCR for page 1
Analysis of Global Change Assessments Lessons Learned assessment, experts who participate in the assessment process, and the target audiences or users of the assessment—in order to ensure salience and legitimacy. Effective stakeholder engagement requires identifying and addressing the needs of specific target audiences, establishing appropriate boundaries at the science-policy interface, engaging stakeholders beyond the target audience, building the capacity of stakeholders to engage in assessments, and a comprehensive, multifaceted communication strategy. Meeting this objective may require significant resources and may thus need to be balanced with efficiency considerations. However, the importance of stakeholder engagement to the overall success of an assessment implies that budgetary provisions, especially for communication, should reflect this reality. Defining and responding to the needs of the target audience is a critical component of an effective assessment process, requiring a continual dialogue between scientists and the target audience. Involvement of the target audience will also promote legitimacy and ownership of the process. The target audience may also comprise intermediaries, such as media, non-governmental organizations, professional organizations, business associations, or “science translators” such as policy advisers and congressional staff members. Because engagement of target audiences in the policy arena is resource intensive and may require expert facilitation, both human and financial resources for such activities should be identified early in the process. Recommendation: The intended audiences for an assessment should be identified in advance, along with their information needs and the level of specificity required for assessment products to be most salient and useful. In most cases, the target audience should be engaged in formulating questions to be addressed throughout the process, in order to ensure that assessments are responsive to changing information needs. Both human and financial resources should be adequate for communicating assessment products to relevant audiences. Defining an appropriate interface between the assessment process and the policy makers who requested and pay for it is a critical challenge in assessment design. Although a deliberate and transparent boundary is necessary to avoid the perception of interference in scientific conclusions, a continuous dialogue is also needed to ensure that questions deemed most relevant by the decision makers are addressed. Perceptions about the degree of government influence can diminish the value of an assessment in the eyes of many stakeholders. Such perceptions may be difficult to overcome, making it especially important to establish guidelines that will stand the test of time. CCSP’s assessment activities have raised credibility and legitimacy issues with some stakeholders, particularly in the science community, due

OCR for page 1
Analysis of Global Change Assessments Lessons Learned to the way the boundary between science and policy was designed. For example, each assessment product is reviewed by the government and requires approval by high-level government officials, raising the question of whether the users of the assessments not only control the questions being asked but, at least potentially, also the scientific conclusions. This concern is addressed to some extent by posting both pre- and post-review versions of each report to allow tracking of the changes. Nonetheless, there remains skepticism about the degree to which government influence may affect scientific outcomes, not only through funding but also through review of final products. Recommendation: The leadership of and those requesting the assessment should establish a transparent and deliberate interface between participants and those who request or sponsor the assessment. Clear guidelines and boundaries should ensure both salience to those requesting the assessment and legitimacy, especially with respect to the perceived influence of those requesting the assessment might have over the scientific conclusions drawn. Despite general understanding that broad stakeholder engagement can contribute importantly to a successful assessment, how to identify and engage the appropriate stakeholders is not self-evident. Participation by broad audiences throughout the assessment process may increase legitimacy and salience, but it could also weaken the credibility of the process. In addition, involvement of too many stakeholders could make the assessment process inefficient and too costly. The appropriate balance between broad stakeholder engagement to achieve legitimacy and salience, and the need to achieve efficient and credible outcomes, will depend on the specific context of each assessment and requires careful consideration early in the assessment design process. Recommendation: A strategy for identifying and engaging appropriate stakeholders should be included in the assessment design to balance the advantages of broad participation with efficiency and credibility of the process. Capacity building to develop a common language and technical understanding among assessment participants, users, and stakeholders can greatly enhance the potential for effective assessments. When stakeholders are unfamiliar with the science or the policy context of a given assessment, their ability to engage in the process will be limited. Investments in capacity building can have payoffs in multiple areas, including: (1) expanding the informed audience for assessments, (2) contributing to future assessment

OCR for page 1
Analysis of Global Change Assessments Lessons Learned effectiveness, (3) expanding the ability of decision makers to act on scientific information, (4) equipping participants with new knowledge on assessment methodology and tools, and (5) building a scientific community that is more sensitive to needs and concerns of the broader society. Recommendation: Capacity building efforts for diverse stakeholders and assessment participants from various disciplines should be undertaken by CCSP in order to develop a common language and a mutual understanding of the science and the decision-making context. This capacity building may be required to ensure the most salient questions are being addressed and to meaningfully engage diverse stakeholders in assessment activities. DEVELOPING DECISION-SUPPORT APPLICATIONS Decision-support applications include a wide range of tools and models that link analyses, environmental and social data, and information about decisions and outcomes. They help decision makers understand the sensitivity of relevant systems, assess vulnerability, identify management alternatives, characterize uncertainties, and plan for implementation. For example, regional tools were developed during the development of the NACCI that allow web-based access to assessment data to assist in making agricultural crop decisions. Adaptation to global change in general, and climate change in particular, requires that the institutional context of decisions be recognized in the development of decision-support tools and adaptation and mitigation activities. Assessments should be designed to be policy relevant without being policy prescriptive. There are many ways to ensure that decision-support efforts are properly focused and effective, but it will not be possible to support every type of decision at every scale. When selecting the specific case studies to be nested within the broader assessment activity, CCSP needs to be strategic about the kinds of decisions to support, and the scale at which such support is most urgently needed. It is also important that sufficient resources be dedicated to supporting the development of decision-support tools, which is a relatively new area of emphasis for CCSP. The critical issue in decision support is providing useful, policy-neutral information, targeted for use in particular sectors and for specific applications. Recommendation: CCSP should foster and support the development of knowledge systems that effectively build connections between those who generate scientific information and the decision makers who are most likely to benefit from access to the knowledge that is generated. One approach is to support the development of decision-support tools and applications at various scales of decision making that can be used in the context of assess-

OCR for page 1
Analysis of Global Change Assessments Lessons Learned ments. In doing so CCSP should identify decision-making processes of high priority or broad application that address key regional or sectoral vulner-abilities, and then evaluate the decision-support needs in those applications. New analytical and predictive tools can then be devised that have direct benefits in specific assessment applications. NESTED MATRIX CONCEPT Adaptive approaches are needed to continually integrate advances in knowledge into the policy context. Although it would be ideal to address impacts and responses for each sector at local, regional, and national scales, it is unlikely that sufficient resources will ever be available to do this comprehensively on an ongoing basis. One way to address this issue is to construct a broad conceptual framework or matrix linked to smaller-scale illustrative examples. For example, an assessment could be conducted at a national level, accompanied by selected localized case studies of impacts on specific sectors or implications for specific local decision making. The work on broad themes and trends can be an ongoing effort, while individual, integrated local, or sectoral assessments can be strategically nested in the broader research agenda. This will help develop an ongoing assessments program that has more coherence over time. An example of the application of the nested matrix approach is using global climate models to identify likely future changes in temperature and precipitation at the national and regional level that may result from climate change. By connecting such outputs to hydrologic models, it is possible to identify a range of likely impacts on runoff for specific watersheds and evaluate potential vulnerabilities for regions and sectors. Based on that information, specific local or regional areas or sectors that are areas of high vulnerability can be selected for a more focused integrated assessment that includes the demographic and institutional context as well as physical parameters. At a regional scale, the vast amount of place-based information, including the additional drivers (e.g., land-use change), can be incorporated into the analysis to provide a more comprehensive treatment of potential changes in water quality and quantity. Recommendation: CCSP should consider implementing this nested matrix concept in developing subsequent assessments.

OCR for page 1
Analysis of Global Change Assessments Lessons Learned This page intentionally left blank.