The committee was not able to evaluate some of the metrics completely, particularly those related to products, primarily because STAR has not been in operation long enough to produce a sufficient number of products to allow a complete evaluation. Other reasons for incomplete evaluations are that EPA has not collected sufficient information relevant to a metric and that some of the metrics are not intended to be addressed solely by the STAR program, but cover issues that are broader and that must be addressed by ORD and EPA. The set of metrics used by the committee in evaluating the STAR grants program is presented in Table 5-1. The goals and objectives of the fellowship program differ from the grants program, and it is evaluated more briefly at the end of this chapter. For each metric, this chapter summarizes the pertinent information provided to (or otherwise obtained by) the committee and then presents the committee’s conclusions about how adequately the STAR program appears to be addressing it with respect to the available information—except that, as stated above, for some product metrics the STAR program is too young to have produced sufficient products to permit a complete evaluation.
Evaluating research quality is extremely difficult. It cannot be measured with a simple metric, such as a thermometer or a yardstick, or by the number of reports or number of pages produced. As indicated in Chapter 4, even more sophisticated measures, such as the number of citations in the technical literature, need to be carefully interpreted.
The STAR program has tended to focus more on the quality of its process than on the quality of its products. That is understandable and appropriate. It is understandable because, being a relatively young program, STAR has had to focus on trying to get the process right and is only now beginning to accumulate a sufficient number of products to support a quality evaluation and because most of the external reviews of the program have tended to focus on process issues.
It is appropriate because, as indicated in Chapter 4, a high-quality pro-cess is generally a necessary condition for producing high-quality products. Frequent evaluations of the quality of the process will also provide an early warning of possible problems in the quality of the products. The causes of inadequate products usually lie in inadequacies of procedures that occurred many years previously.