understand current developments and trends. The committee also found that the geographic diversity and professional diversity of panel membership are essential to ensure a fair and comprehensive assessment. Over the course of the studies, panels came to agree that no more than half their members should be US academic researchers. The immunology panel, for example, found in its initial response a clear bias related to the laboratory location of the pollees: US-based investigators routinely named a higher percentage of Americans than did non-US-based investigators. The nationality of the poller also appeared to have an influence: the three non-US pollers often obtained a virtual-congress list with a higher proportion of non-US speakers. The panel decided in its second iteration that it needed to increase foreign representation to ensure objectivity; on doing so, it obtained results that agree more closely with those of citation analysis and journal-publication analysis and with the judgment of the panelists.

The committee concluded that at least one-third of panel members should be non-US researchers. An additional one-third should be a combination of researchers in industry and in related fields who use the results of research. In the experience of the panels, that mix of perspectives, including especially the representatives of research-intensive industries (such as biotechnology, telecommunications, and aerospace), was essential for understanding not only the scholarly and technical achievements of researchers, but also the broader importance of those achievements to social and economic objectives.

4.5 Findings About the Utility of Benchmarking

On the basis of presentations made at the workshop by congressional and agency staff and feedback from disciplinary-society members who were briefed by panelists, COSEPUP found that benchmarking is potentially useful to the research communities in selected fields and in fields related to them, and to the government sponsors of research in selected and related fields. The panel reports were able to identify weaknesses in particular subfields and sub-subfields and to point out issues that need to be addressed in making policy.

The committee also suggests that benchmarking might be useful in efforts to comply with GPRA. Representatives of federal agencies that support research were asked during the benchmarking workshop about the utility of the technique. Although the terminology and, to some extent, the concept were new to some, there were some indications that benchmarking was likely to be useful in evaluating agency research programs and in providing information that would help them to comply with GPRA.

The National Academies of Sciences, Engineering, and Medicine
500 Fifth St. N.W. | Washington, D.C. 20001

Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement