including the investments that the government is making in the science and technology areas. This has brought to the forefront issues such as assessing the results of scientific research and applying metrics to fundamental science. This is particularly true for the second phase of the National Performance Review; that is, all agencies are being examined critically by the Office of Management and Budget (OMB). The President and Vice President will be making final decisions in the next few months. We can expect more cuts or arguments for greater “savings.”
To establish performance metrics for basic research and education is a difficult task, as has been pointed out by several sources, including a recent report of the National Research Council's (NRC 's) Commission on Physical Sciences, Mathematics, and Applications, Quantitative Assessments of the Physical and Mathematical Sciences: A Summary of Lessons Learned (National Academy Press, Washington, D.C., 1994). As noted above, however, we do not have an option. It is the law. In reality, this can be turned into an opportunity for NSF if we approach it sensibly and responsibly.
We must have a strategic plan and we must have a clear way to measure the impact and value of the public investment. Thus, MPS staff in the winter of 1993 and currently the MPS advisory committee, led by Dean Eastman of IBM and Ed Hayes of Ohio State, are involved in the MPS effort to develop a useful and rational approach to this subject.
It also is important to note that the National Science and Technology Council began to look at this issue in late 1994 and will sponsor a government-wide colloquium on May 17, 1995. This colloquium, entitled “Assessing the Contribution of Fundamental Science,” will be much larger than today's workshop in terms of the number of participants, and it will try to cover a much broader spectrum of issues. The colloquium will consider questions such as the following:
What quantitative and qualitative measures are meaningful and appropriate for science? How do these requirements differ among the different sciences? What is the impact of metrics on the conduct of science?
How can the freedom to openly explore frontiers of science be balanced against the need to respond to near-term demands?
How can near-term assessment methods be harmonized with the long-term nature of fundamental research? What types of metrics, assessment methods, or evaluation techniques are used in different scientific disciplines and institutions?
These basic questions will be addressed by leaders and scientists from colleges and universities, from industry, and from federal agencies and laboratories. The findings of today's workshop on industry metrics will be presented there as well.
Within NSF, there is a strong sense that we will be better equipped to address these issues by learning from the experience of leaders from Bell Labs, IBM, Ford, and Xerox—industries whose research contributions have been so valuable to the mathematical and physical sciences. Thus, the purpose of today's workshop is to provide the Mathematical and Physical Sciences Directorate, and the National Science Foundation, with workable performance models and methods—or with examples of what not to do. It is important to measure the right things. Or if there are no formal metrics, what are the questions or issues that drive the decision-making process with respect to investing in R &D? At the same time, we believe it is important that the metrics that are put in place also include the input of the party to whom the measures are to be applied—the research community that we support.
Thus, in this workshop we have brought together individuals from corporations who have addressed in some manner the issue of performance evaluation of their research organizations, and who also have been involved in restructuring their research portfolios in response to reductions in the amount of available funding; individuals from the academic world of the physical and mathematical sciences who are now addressing a restructuring of their research portfolios in response to reduced government and private funding; and NSF colleagues and other government officials.
Although the perspective of the measures used by corporations is only one of many that the National Science Foundation will have to consider, their experience will help the institution to focus on workable models and methods. At the same time, we hope to learn what approaches are not workable.