National Academies Press: OpenBook
« Previous: Front Matter
Suggested Citation:"Executive Summary." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2000. Experiments in International Benchmarking of US Research Fields. Washington, DC: The National Academies Press. doi: 10.17226/9784.
×

EXECUTIVE SUMMARY

In its report Science, Technology, and the Federal Government: National Goals for a New Era,1 the Committee on Science, Engineering, and Public Policy (COSEPUP) of The National Academies recommended that the status of research fields might be evaluated by the use of international benchmarking. COSEPUP was referring to a technique that is designed to compare one country's or region's leadership status in research with world standards of excellence.

To assess the feasibility and utility of international benchmarking, COSEPUP proposed a set of experiments in three fields: mathematics, immunology, and materials science and engineering. The results of the experiments suggest that research leadership status by field can be assessed in a timely fashion at reasonable cost. International benchmarking has potential utility for policy-makers and federal agencies in determining the leadership status of the United States in particular fields, subfields, and possibly sub-subfields. It might also help federal agencies to comply with the Government Performance and Results Act by evaluating the quality of their own performance.

Committee members note especially the following points:

  • Panel leaders began the process as skeptics but came to believe that benchmarking can be reasonably objective and timely.

  • Two separate benchmarking experiments in mathematics—one at the National Academies, the other at the National Science Founda-

1  

Known as the ''Goals" report.

Suggested Citation:"Executive Summary." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2000. Experiments in International Benchmarking of US Research Fields. Washington, DC: The National Academies Press. doi: 10.17226/9784.
×

tion—despite dissimilar panels and mandates produced similar results, lending credence to the technique.

During the experiments, committee members were able to identify several particular strengths of international benchmarking:

  • Panels were able to identify institutional and human-resource factors crucial to maintaining leadership status in a field that is unlikely to have been identified by other methods.

  • Benchmarking allows a panel to determine the best measures for a particular field while providing corroboration through the use of different methods, as opposed to the "one-size-fits-all" approach of some common evaluation methods.

  • Benchmarking can produce a timely but broadly accurate "snapshot" of a field.

The experiments were sufficiently thorough to provide guidelines for future experiments, including the following:

  • Because of the panels' use of expert judgment, the choice of panelists is a key to the credibility of the results. A tendency toward national biases can be mitigated by ensuring diverse geographic membership of panels; the same is true of the groups that select the panel members. In particular, it is critical to include non-US participants in the selection of panelists and as panel members because they provide perspective and objectivity.

  • Because major fields of research change slowly, benchmarking can probably detect important changes in quality, relevance, and leadership in fields when conducted at intervals of 3-5 years. It is unlikely that changes can be detected by annual benchmarking.

  • The choice of research fields to be evaluated is both challenging and critical. A "field" might best be considered the array of related domains between which investigators can move without leaving their primary area of expertise.

  • Benchmarking produces information that administrators, policy-makers, and funding agencies find useful as they make decisions as to what activities a federal research program should undertake and respond to demands for accountability, such as the Government Performance and Results Act.

  • If federal agencies use benchmarking, the wide variation in agency missions dictates that each agency tailor the technique to its own needs.

  • Use of indicators that provide information on degree of uncertainty and reliability of benchmarking results might enhance the presentation of panel assessments of leadership status.

Suggested Citation:"Executive Summary." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2000. Experiments in International Benchmarking of US Research Fields. Washington, DC: The National Academies Press. doi: 10.17226/9784.
×

The present experiments should be regarded as an encouraging first step toward the design of an efficient and reliable evaluation tool. An accumulation of benchmarking exercises will lead to more-effective methods, better understanding of the factors that promote research excellence, and better decision-making by funders of science and technology.

Suggested Citation:"Executive Summary." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2000. Experiments in International Benchmarking of US Research Fields. Washington, DC: The National Academies Press. doi: 10.17226/9784.
×
Page 1
Suggested Citation:"Executive Summary." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2000. Experiments in International Benchmarking of US Research Fields. Washington, DC: The National Academies Press. doi: 10.17226/9784.
×
Page 2
Suggested Citation:"Executive Summary." National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. 2000. Experiments in International Benchmarking of US Research Fields. Washington, DC: The National Academies Press. doi: 10.17226/9784.
×
Page 3
Next: 1 Introduction »
Experiments in International Benchmarking of US Research Fields Get This Book
×
Buy Paperback | $87.00 Buy Ebook | $69.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

How can the federal government gauge the overall health of scientific research--as a whole and in its parts--and determine whether national funding adequately supports national research objectives? It is feasible to monitor US performance with field-by-field peer assessments. This might be done through the establishment of independent panels consisting of researchers who work in a field, individuals who work in closely related fields, and research "users" who follow the field closely. Some of these individuals should be outstanding foreign scientists in the field being examined. This technique of comparative international assessments is also known as international benchmarking.

Experiments in International Benchmarking of U.S. Research Fields evaluates the feasibility and utility of the benchmarking technique. In order to do this, the report internationally benchmarks three fields: mathematics, immunology, and materials science and engineering, then summarizes the results of these experiments.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!