The following HTML text is provided to enhance online
readability. Many aspects of typography translate only awkwardly to HTML.
Please use the page image
as the authoritative form to ensure accuracy.
Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program
creation of new products or techniques). However, most will be qualitative, especially those that focus on the research and development process, the outcome of research, and its impact on society. These generally require peer review (e.g., to evaluate scientific quality) or stakeholder assessments.
CCSP objectives range from the general (e.g., overarching goals) to the specific (e.g., milestones, products, and payoffs). The more general the objective, the greater is the number of qualitative contributing factors and the less quantitative are the metrics. For example, improvements to databases of water cycle variables can generally be measured quantitatively, but the resulting improvement in drought prediction models and resource decisions that use those predictions will require increasingly subjective analysis and a greater emphasis on expert assessment.
2. Identify three to five areas of climate change and global change research that can and should be evaluated through quantitative performance measures.
Both the Office of Management and Budget (OMB) and the agencies participating in the CCSP are seeking a manageable number of quantitative performance measures to monitor the progress of the program. The metric cited most often is the reduction of uncertainty (Chapter 4). However, by itself, reduction of uncertainty is a poor metric because (1) uncertainty about future climate states may increase, decrease, or remain the same as more is understood about the governing elements, and (2) the data needed to calculate errors in the probability estimates are limited or nonexistent. The danger of using this metric is that increasing uncertainty might be interpreted as a failure of the program, when the reverse may well be true.
The committee agrees that a limited set of metrics should be chosen. It would be expensive to implement all possible measures, and the results may be difficult for individual agencies to use to manage their programs and demonstrate success to Congress, OMB, and the public. However, the CCSP strategic plan provides neither a sense of priorities nor a definition of success.1 Indeed, a National Research Council review of the CCSP strategic plan noted that “many of the objectives in the plan are too vaguely worded
The CCSP strategic plan does not list measures of success, but examples can be found elsewhere. For instance, measures of success proposed for the Department of Energy’s program on environmental quality research include the degree to which the program has led to improved performance, reduced risks to human health or the environment, decreased costs, and advanced schedules. See National Research Council, 2001, A Strategic Vision for Department of Energy Environmental Quality Research and Development, National Academy Press, Washington, D.C., 170 pp.