defined plan for regular, external reviews of its research quality, and has this plan been effectively carried out? Has the program made significant contributions to advancing the state of the science in particular research topics? Does the program award grants expeditiously? Does the program have a schedule for the products it intends to produce and how well is it adhering to the schedule?
The committee’s judgment is that quantitative metrics, although outwardly simpler to use, are not necessarily more informative than qualitative metrics. In some cases, quantitative metrics can be misleading, and emphasizing inappropriate metrics can distort the research outputs of a program. Qualitative metrics are less likely to have such effects, but they need to be interpreted carefully.
The committee judges that expert review by a group of people with appropriate expertise is the best method of evaluating broad research programs, such as the STAR program. Expert review is appropriate for evaluating both the processes and the products of the STAR program. The types of experts needed depend on the level of review being conducted—individual projects or programmatic levels. Both qualitative and quantitative metrics can provide valuable support for such expert reviews.
In planning for future reviews, the committee recommends that STAR and ORD consider an evaluation structure for the STAR program that has four levels: level 1 should examine the individual research projects, level 2 should focus on topics or groups of research projects on the same subject, level 3 should address the STAR program as a whole, and level 4 should tackle the question of how the STAR program relates to the broader institutions of ORD and EPA. The primary mechanism of review at levels 2-4 should be the panel of independent experts with the appropriate scientific, management, and policy backgrounds; the panels’ evaluations can use metrics appropriate to the specific level of review. Such a structured review strategy could replace the number of ad hoc, unplanned, and uncoordinated reviews.
Recommendation. STAR and ORD should establish a structured program of reviews by panels of independent experts and should collect the appropriate information to support these reviews.