example, through an external advisory committee including social and natural scientists).
The committee was also asked to consider, as part of its review, the metrics3 that would be employed to evaluate progress toward the goals outlined in the seven Themes. The thematic sections in the Strategic Plan do not provide an explicit description of the metrics for evaluation, thus the committee found this component of its task particularly challenging.
Since issuance of the Government Performance and Results Act (GPRA) of 1993 and related Office of Management and Budget (OMB) policies, agencies have increased efforts to establish metrics and track progress (NRC, 2005). However, it is often difficult to evaluate research using strictly quantitative measures because the discovery process is complex and outcomes that matter to society are not always traceable to specific projects; these achievements often result from a combination of research findings and their use in formulating policy. Most agencies, therefore, rely on expert peer review to assess progress in research (NRC, 2005). Performance measures are also agency-specific and may include qualitative outputs or outcomes (see Box 2.3 for additional details). For example, NOAA’s National Marine Fisheries Service might use the reduction in numbers of overfished major stocks while NSF might need more general input or output metrics. Therefore, a broad range of metrics need to be used and they need to be tailored to the specific research goals and agency missions.
The FOARAM Act requires the National Ocean Acidification Program to provide biennial progress reports and a revised 10-year plan every 5 years. This requirement dictates that progress of the various elements of the program be evaluated. To this end, it is important that metrics be established to measure progress toward the Strategic Plan’s goals, and that these evaluations of the program be used to set priorities in an iterative fashion. The committee recognizes the difficulty in establishing a set of specific metrics for scientific research. It is also recognized that the Strategic Plan is not an implementation plan, which would be a more appropriate document to provide a detailed description of metrics for evaluation. However, there are many reports (NRC, 2005) that provide guidance on developing such performance measures. For example, the committee finds that many of the suggested metrics for the former CCSP, now referred to as the USGCRP, could be applied to the Ocean Acidification Program (NRC, 2005).
Lastly, the issues of “metrics” and “prioritization” are closely linked. As stated in the 2005 NRC Report, Thinking Strategically: The Appropriate
3 Government agencies use “metrics” (synonymous with “performance measures”) to assess progress toward pre-established goals (NRC, 2005).