Einstein stated, is that “not everything that counts can be counted, and not everything that can be counted counts”—a phrase that was quoted several times during the workshop.
The Multiple Uses of Performance Measures
Performance measures have many uses, Feller continued. First, they are used to do retrospective assessments of realized, observed, and measured impacts. In this case, basic questions are: How has that program worked? Has it produced the results for which it was funded? How could these research advances contribute to societal objectives?
Second, performance measures can be used to assess the best direction in which to head. Is this where scientific advances will occur? Will these scientific advances lead to the achievement of societal objectives?
Finally, performance measures can benchmark accomplishments against historical or international measures and advocate for particular actions.
In each of these cases, performance measures have little relevance in the abstract, Feller said. They need to be related to the decisions at hand, and their promise and limitations depend on the decision being made. “They are quite necessary and productive for certain types of decisions, problematic for others, and harmful for others.”
The context of performance measures determines much of their promise and limitations, according to Feller. A critical question is who is asking the questions. In a university setting, a promotion and tenure committee might ask about publications and citations while a dean or president might ask which areas of the university to support. In the federal government, a member of Congress might ask whether appropriations for a particular laboratory will produce jobs in his or her district, the director of OSTP might ask questions about recommendations to make to the President, and the director of the Office of Management and Budget (OMB) might ask about U.S. research expenditures relative to all other demands on the budget. Similarly, different federal agencies might ask different questions. NSF might want to know how to use research to advance the frontiers of knowledge, while the EPA might want to use science to support regulatory decisions.
Performance measures have been the focus of longstanding and diverse research traditions, Feller said. Over the course of four decades, he has studied patent data, bibliometrics, and many other measures