. "4. Measure for Measure." The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science to Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press, 2003.
The following HTML text is provided to enhance online
readability. Many aspects of typography translate only awkwardly to HTML.
Please use the page image
as the authoritative form to ensure accuracy.
WHAT ARE METRICS?
Geisler (2000) defines metrics for the evaluation of a research program as “a system of measurement that includes, (1) the item or object that is being measured; (2) units to be measured, also referred to as ‘standard units’; and (3) value of a unit as compared to other units of reference.” Geisler (2000) goes on to clarify this definition as follows:
A refinement of the definition of metric extends it to include: (a) the item measured (what we are measuring), (b) units of measurement (how we measure), and (c) the inherent value associated with the metric (why we measure, or what we intend to achieve by this measurement). So, for instance, the metric peer review includes the item measured (scientific outcomes), the unit of measurement (subjective assessment), and inherent value (performance and productivity of scientists, engineers, and S&T units).
Types of Metrics
Metrics may be classified as quantitative, semiquantitative, and qualitative. For the purpose of this report, the committee characterizes metrics as quantitative or qualitative, grouping the semiquantitative measures with the qualitative.
Quantitative measures, such as the number of peer-reviewed publications resulting from a grant, have the desirable attributes of public availability and reproducibility. A drawback to quantitative metrics is that they tend to be reductive or one-dimensional, measuring a single quantity. As a result, quantitative metrics, although outwardly simpler to use, are not necessarily more informative than qualitative metrics. Quantitative metrics tend to be more useful at lower levels of evaluation, when information tends to be more discrete, such as a review of a specific grant or center, but become less useful as one evaluates higher levels of integration, such as a review of the entire STAR program.
Qualitative metrics have the advantage of being multidimensional, that is, of comprising an intricate and complex set of measures. Therefore, qualitative metrics are more useful for evaluating higher levels of integration, such as an entire research program. Qualitative metrics can have numerical components; for instance, in reviewing grants, a scoring system of 1 to 5 is commonly used, in which the numbers represent such labels as “excellent”