Evaluating Progress of the U.S. Climate Change Science Program: Methods and Preliminary Results (2007)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Summary T he U.S. Climate Change Science Program (CCSP) was created in February 2002 under a new cabinet-level management structure designed to improve government-wide management of climate and related environmental science. The CCSP integrated the then-existing U.S. Global Change Research Program (USGCRP) with the administration’s Climate Change Research Initiative. The CCSP was formed with an ambi- tious, but practical, guiding vision: a nation and the global community empowered with the science based knowledge to manage the risks and op- portunities of change in the climate and related environmental systems. Although the U.S. government has sponsored research on climate and related environmental change through the CCSP or USGCRP for more than 15 years, the progress of either program has never been evaluated. Such evaluations are important for identifying strengths and weaknesses and determining what adjustments should be made to achieve program goals. At the request of Dr. James Mahoney, then director of the CCSP, the National Research Council (NRC) established the Committee on Strategic Advice on the U.S. Climate Science Program to carry out three tasks over a three-year period. The first task—an evaluation of program progress—is the subject of this report: Task 1. The committee will assist the CCSP in evaluating progress toward program goals. The CCSP Strategic Plan and the guidelines given in the 2005 NRC report Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program will provide a starting point for this examination. The report will address two subtasks:  

 EVALUATING PROGRESS OF THE U.S. CCSP 1a. Findings and recommendations on the process for evaluating progress toward the five goals in the CCSP strategic plan. The recommen- dations should be practical and consider the trade-offs between strategic utility and program costs associated with implementing metrics. 1b. A preliminary assessment of progress made toward the pro- gram’s goals. The results will serve as an interim report for a more compre- hensive analysis of the program’s progress to be completed in subsequent years. The focus of this report is on progress made over the past four years— the lifetime of the CCSP. How the program should evolve to address gaps and weaknesses or to respond to new needs is the subject of the committee’s second task and report. The CCSP’s structure, activities, and time line for delivering products are laid out in a 2003 strategic plan. Thirteen federal agencies participate in the CCSP, which has an annual budget of about $1.7 billion. The budget is provided and managed by the participating agencies, which also help set the direction of the program through interagency committees at various levels. The overall program is guided by a director (currently an acting director) and carried out by the agencies and a small program office. The CCSP is divided into three main components: (1) overarching goals, which represent what the overall program is trying to achieve (e.g., scientific understanding, reduction of uncertainties, risk management); (2) research elements (e.g., atmospheric composition, carbon cycle, human con- tributions and responses), which lay out the research agenda in the form of 33 questions to be answered; and (3) cross-cutting issues (e.g., observations, decision support resources, communications), which are common to all of the research elements. A method for evaluating the progress of all three components and conclusions from the committee’s preliminary evaluation are described below. METHOD FOR EVALUATING PROGRESS Recommendation. CCSP progress should be evaluated in two stages: (1) a broad overview of the entire program based on the knowledge of the reviewers, and (2) a more in-depth analysis of areas in which progress has been inadequate, using the process and input metrics from NRC (2005). A 2005 NRC report proposed a framework of 24 metrics that could be used to evaluate the CCSP from end to end—from program processes (e.g., strategic planning, peer review) to inputs (e.g., resources), to short- term outputs (e.g., publications), to long-term outcomes (e.g., improved 

SUMMARY  understanding, use of science to support decision making) and impacts (e.g., improved public policy). The committee found that this framework yields a wealth of information on CCSP progress, but the detailed budget and management information necessary to score the process and input metrics is not readily available, even to CCSP agencies. Consequently, the commit- tee developed an alternative two-stage evaluation approach that balances practicality and strategic utility. The first stage would be a high-level as- sessment of strengths and weaknesses of the entire program, based mainly on the reviewers’ knowledge of program results. The entire program can be evaluated using a matrix of the 33 research questions in the research ele- ments (rows of the matrix) versus five categories of outputs and outcomes (columns of the matrix): A: Improve data sets in space and time, and improve estimates of physical quantities B: Improve understanding and representation of processes C: Improve predictability, predictive capabilities, or assessment of uncertainty D: Improve synthesis and assessment to inform E: Improve assessment and management of risk, and improve decision support for management and policy making The rows of the matrix (research questions) are connected to the CCSP overarching goals, and the columns of the matrix overlap with the cross- cutting issues. In particular, category A includes observations and monitor- ing, category C includes modeling, category D includes communication, and category E includes decision support resources. By combining the scores of the cells of the matrix in different ways, it is possible to assess progress in the CCSP research elements, overarching goals, and cross-cutting issues. The second stage of evaluation would be a careful analysis of areas identified as not meeting expectations. These areas would be evaluated with the process and input metrics from NRC (2005), which provide tools for diagnosing the reasons for program weaknesses and making strategic deci- sions about where adjustments should be made to improve outcomes. PRELIMINARY ASSESSMENT OF PROGRESS The committee used its matrix to carry out the first stage of the evalua- tion of CCSP progress. Findings based on that evaluation are given below. The separation of leadership and budget authority presents a serious obstacle to progress in the CCSP. 

 EVALUATING PROGRESS OF THE U.S. CCSP Leadership to guide the program is generally required if a program is to succeed (NRC, 2005). The strength of the current CCSP leadership structure lies in its potential to engage the expertise found across U.S. gov- ernment agencies and international partners to address climate science and applications. CCSP leaders can advocate for the program at higher levels in the government or with participating agencies when the decisions of a single agency adversely affect the entire program (e.g., cancellation of critical climate sensors) or when changing CCSP priorities would require changes in agency programs (e.g., a greater emphasis on supporting decision mak- ing). However, the CCSP director and agency principals lack authority to allocate or prioritize funding across the agencies, and the interagency work- ing group members often have little budgetary authority to implement the research directions that they define. Such authority usually resides at higher levels in the participating agencies. As a result, progress is most likely when CCSP and agency interests coincide. Discovery science and understanding of the climate system are pro- ceeding well, but use of that knowledge to support decision making and to manage risks and opportunities of climate change is proceeding slowly. Good progress has been made in documenting the climate changes of the past few decades and in unraveling the anthropogenic influences on the observed climate changes. The period has witnessed improved understand- ing of many aspects of the climate and related environmental systems, including aerosol direct forcing, land use change, sea ice retreat, glacier melting, and atmospheric warming. Predictive capabilities have also im- proved, especially of coupled ocean-atmosphere-land climate models used to evaluate the human impact on observed trends, although models that enable exploration of feedbacks, predictions at regional to local scales, or trade-offs of different resource management and mitigation options are still relatively immature. In contrast, progress in synthesizing research results or supporting decision making and risk management has been inadequate. Although the temperature trends assessment (CCSP, 2006b) was influential in the 2007 report of the Intergovernmental Panel on Climate Change, 19 other CCSP synthesis and assessment products scheduled to be released by now are still in the production stage. Also, only a few small programs (e.g., Regional Integrated Sciences and Assessments program, Decision Making Under Uncertainty centers) have been initiated to identify and engage deci- sion makers. 

SUMMARY  Progress in understanding and predicting climate change has improved more at global, continental, and ocean basin scales than at regional and local scales. Information at regional and local scales is most relevant for state and local resource managers and policy makers, as well as for the general popu- lation, but progress on these smaller spatial scales has been inadequate. Improving understanding of regional-scale climate processes and their im- pacts in North America, for example, would require improved integrated modeling, regional-scale observations, and the development of scenarios of climate change and impacts. Our understanding of the impact of climate changes on human well-be- ing and vulnerabilities is much less developed than our understanding of the natural climate system. Progress in human dimensions research has lagged progress in natural climate science, and the two fields have not yet been integrated in a way that would allow the potential societal impacts of climate change and management responses to be addressed. This disparity in progress likely re- flects the inability of the CCSP to support a consistent and cogent research agenda as recommended in previous studies. The level of investment ($25 million to $30 million) remains substantially lower than the level of invest- ment in the other research elements, and funding is atomized across many agency programs. Few social scientists are in leadership positions in the federal agencies, which makes it difficult for the CCSP to increase program emphasis in this area or to establish links with the academic social science community. Finally, the research community is small and thus may be un- able to advocate effectively for changing program priorities. Science quality observation systems have fueled advances in climate change science and applications, but many existing and planned observ- ing systems have been cancelled, delayed, or degraded, which threatens future progress. Knowledge of climate variability and change rests on consistent long- term observations that are broadly disseminated and archived for future generations of scientists. The contribution of remote sensing and in situ observations and their associated information systems to Earth system science and applications has been a major achievement of the CCSP-US- GCRP agencies. However, a number of planned satellite sensors critical to the long-term (multidecadal) data record have been cancelled or seriously 

 EVALUATING PROGRESS OF THE U.S. CCSP delayed (e.g., National Polar-orbiting Environmental Satellite System cli- mate instruments, Hydros, Landsat, Global Precipitation Measurement mis- sion), and long-standing (decades to a century or longer) in situ networks are deteriorating (e.g., stream gauge network, Snowpack Telemetry snow observation system) because of funding shortfalls. The loss of existing and planned satellite sensors is perhaps the single greatest threat to the future success of the CCSP. Without a wide array of continuous satellite and in situ observations, the U.S. capability to monitor trends, document the impacts of future climate change, and further improve prediction and assimilation models through comparison with observations will decline even as the ur- gency of addressing climate change increases. Progress in communicating CCSP results and engaging stakeholders is inadequate. The program has had some successes interacting with scientists, federal government agencies, and water resource managers. However, efforts to identify and engage in a two-way dialogue with state and local officials, nongovernmental organizations, and the climate change technology com- munity have generally been limited and ad hoc. As a result, the program is not gaining the input it needs on what scientifically based CCSP products to create, and opportunities to inform decision making are being missed. The committee notes that differences in the rates of program progress between the natural and social sciences and between science and commu- nications and decision support are not surprising, given the long history of support of fundamental research through the USGCRP and the allocation of CCSP funding. Only a small fraction of the CCSP budget is devoted to decision support resources and communication (CCSP, 2006a). However, if the program is to achieve its vision of producing information that can be used to formulate strategies for preventing, mitigating, and adapting to effects of climate change, adjustments will have to be made in the balance between science and applications.