10. Metrics must evolve to keep pace with scientific progress and program objectives.
The development of metrics is a learning process. No one gets it right the first time, but practice and adjustments based on previous trials will eventually yield useful measures and show what information must be collected to evaluate them. Metrics must also evolve to keep pace with changes in program goals and objectives. Scientific enterprises experience considerable evolution as they move through various phases of exploration and understanding. Metrics for newly created science programs, which focus on data collection, analysis, and model development to increase understanding, will tend to focus on process and inputs. As the science matures and the resulting knowledge is applied to serve society, metrics will focus more on outputs and, finally, on outcomes and impacts. As science transitions from the discovery phase to the operational or mission-oriented phase, the types of metrics should also be expected to evolve.
11. The development and application of meaningful metrics will require significant human, financial, and computational resources.
The development and application of metrics, especially those that focus on quality, is far from a bookkeeping exercise. Efforts to assess programmatic plans, scientific progress, and outcomes require substantial resources, including the use of experts to carry out the reviews. Funding to support the logistics of the reviews is also required. The CCSP strategic plan includes a substantial number of assessments and a growing emphasis on measurable outcomes. As these are implemented, the choice of meaningful measures of progress must be deliberate. If the IPCC process is a representative example, the growing emphasis on assessments has the potential to increasingly divert resources from research and discovery to assessment.