Experiments in International Benchmarking of US Research Fields (2000)

In 1992, the Committee on Science, Engineering, and Public Policy (COSEPUP) of the National Academies examined the federal government's rationale for funding research in science and technology. There is little doubt that science and technology are powerful tools in moving the nation toward social and economic goals,² and yet it has proved difficult to devise a consistently fair and effective mechanism for allocating federal funds for research.

National policy-makers desire better mechanisms for several reasons. From the funder's point of view, it is natural to ask what the outcomes of a particular program or project are likely to be and when they might be expected. But it is seldom possible to predict the outcomes of basic research or to know which fields of research will ultimately contribute to important new ideas or technologies.

At the same time, there is little debate about the necessity for sustained agendas of both basic and applied research.³ Retrospective studies consistently describe the power of research to produce break-throughs years and even decades after the work has been performed.⁴ Therefore, the committee has no doubt about the need for sustained federal funding of basic research in every major field.

When the committee issued its "goals report" (Science, Technology, and the Federal Government: National Goals for a New Era),⁵ it recommended that the federal government continue vigorous funding of basic research and that it seek to support this research across the entire spectrum of scientific and technological investigation. That advice was justified by the fact that leadership in science had become one of the defining characteristics of the United States and other great nations.

Specifically, COSEPUP suggested two goals. First:

"Major areas" refers to broad disciplines of science (such as biology, physics, mathematics, chemistry, earth science, and astronomy) and to their major subdisciplines (such as the neurosciences, condensed-matter physics, and seismology). "Among the world leaders" means that the United States should have capabilities and infrastructures of support that are not substantially exceeded elsewhere. The primary rationale for the recommendation is that working at a world standard of excellence in all fields allows this nation to apply and extend scientific advances quickly no matter when or where in the world they occur. The value of being among the leaders was dramatized when, for example, the phenomenon of high-temperature superconductivity was demonstrated in an international laboratory in Switzerland. US researchers, although they did not participate in that breakthrough, were able to replicate the results in a matter of weeks because they were working at the frontiers of solid-state physics (the general field in which the breakthrough occurred) and they had the ability to move quickly. In addition, because of the degree of interconnection between fields, there is a concern that if one were neglected (placing the United States behind the world leaders), others might be slowed as a result. For example, much of the progress in life sciences research is made possible by the availability of instruments designed by scientists and engineers in physics and chemistry.

Second:

Such areas would include those which are required to meet national objectives, which capture the imagination of society, or which have multiplicative effects in other important fields. For example, the United

States desires to maintain clear leadership in molecular genetics because of its central importance to human health and to the biotechnology industry.

Both those goals were reiterated in another Academies report, Allocating Federal Funds for Science and Technology,⁶ developed by a committee chaired by former National Academy of Sciences President Frank Press which stated that "to continue as a world leader, the United States should strive for clear leadership in the most promising areas of science and technology and those deemed most important to our national goals. In other major fields, the United States should perform on a par with other nations so that it is 'poised to pounce' if future discoveries increase the importance of one of these fields."

In the goals report, COSEPUP considered the question of how to measure leadership. How can the federal government gauge the overall health of scientific research—as a whole and in its parts—and determine whether national funding adequately supports national research objectives? The committee wrote that it is feasible to monitor US performance with field-by-field peer assessments and that this might be done through

This technique of comparative international assessments, or "international benchmarking", had been discussed in theory and applied in specific cases. But COSEPUP made the decision to undertake a set of realistic experiments that would test the utility of the technology in evaluating entire research fields.

The committee acknowledged that quantitative indicators commonly used to assess research programs—for example, dollars spent, papers cited, and numbers of scientists supported—are useful information but noted that by themselves they are inadequate indicators of leadership, both because quantitative information is often difficult to obtain or compare across national borders and because it often illuminates only a portion of the research process. For example, a paper that describes truly innovative research may receive few citations if no one else is doing comparable work. Similarly, the pervasive value of mathematics to the other sciences is not accurately measured by the small number of dollars flowing into mathematical research.

The committee decided that the expert judgment of panel members afforded the most effective means for assessing research. The basis for this decision is that those immersed in a particular field are best qualified to appraise the quality of its researchers, identify the most promising advances, and project the status of the field into the future. Such panelists are also well positioned to pinpoint locations where the most promising ideas are emerging, describe where the best new scientific talent chooses to work, and judge the comparative quality of research facilities and human resources. The participation of representatives of other countries, people in other fields of research, and groups that use the results of research helps to provide perspective and objectivity to the exercise and to keep the panels' judgments from being self-serving.

In addition, passage of the Government Performance and Results Act (GPRA) in 1993 emphasized the need for a method to assess the results of scientific and technological research investments. Although COSEPUP began its deliberations about benchmarking before GPRA became law, it recognized while developing its report that benchmarking might provide input that is useful to some federal agencies as they evaluate their own research programs. The committee's GPRA report⁷ concluded that the most effective means of evaluating federal research is expert review. COSEPUP defined expert review in terms of three elements—quality, relevance, and leadership—and proposed that leadership be evaluated via international benchmarking. A benchmarking panel would be asked, "Is the research being performed at the forefront of scientific and technological knowledge?" The panel would then consider the mission objectives of the particular agency in the context of an assessment of the nation's scientific leadership status.

In the language of COSEPUP's GPRA report:

In this report, COSEPUP evaluates the feasibility and utility of the benchmarking technique. As explained in greater depth in chapter 2, it

does so by internationally benchmarking three fields: mathematics, immunology, and materials science and engineering. The results of these experiments are summarized in chapter 3. The committee held a workshop to obtain input from agencies, policy-makers, disciplinary societies, and others on the feasibility and utility of benchmarking; the workshop provided material for chapter 4, in which COSEPUP develops its findings, and chapter 5, where those findings are discussed. Conclusions are listed in chapter 6.