NATIONAL ACADEMY PRESS
2101 Constitution Ave., NW Washington, DC 20418
NOTICE: This volume was produced as part of a project approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. It is a result of work done by the Committee on Science, Engineering, and Public Policy (COSEPUP) as augmented, which has authorized its release to the public. This report has been reviewed by a group other than the authors according to procedures approved by COSEPUP and the Report Review Committee.
The Committee on Science, Engineering, and Public Policy (COSEPUP) is a joint committee of NAS, NAE, and IOM. It includes members of the councils of all three bodies.
Financial Support: The development of this report was supported by the Sloan Foundation and the National Research Council.
International Standard Book Number 0-309-06898-3
Internet Access: This report is available on COSEPUP's World Wide Web site at http://www4.nationalacademies.org/pd/cosepup.nsf
Order from:
National Academy Press,
2101 Constitution Ave., NW, Washington, DC 20418. All orders must be prepaid with delivery to a single address. No additional discounts apply. Prices are subject to change without notice. To order by credit card, call 1-800-624-6242 or 202-334-3313 (in Washington metropolitan area).
Copyright 2000 by the National Academy of Sciences. All rights reserved. This document may be reproduced solely for educational purposes without the written permission of the National Academy of Sciences.
Printed in the United States of America.
Committee on Science, Engineering, and Public Policy
MAXINE F. SINGER (Chair), President,
Carnegie Institution of Washington
BRUCE M. ALBERTS,* President,
National Academy of Sciences
ENRIQUETA C. BOND, President,
The Burroughs Wellcome Fund
LEWIS BRANSCOMB, Professor Emeritus,
Center for Science and International Affairs, John F. Kennedy School of Government, Harvard University
PETER DIAMOND, Institute Professor and Professor of Economics,
Massachusetts Institute of Technology
GERALD DINNEEN,* Retired Vice President,
Science and Technology, Honeywell, Inc.
MILDRED S. DRESSELHAUS, Institute Professor of Electrical Engineering and Physics,
Massachusetts Institute of Technology
JAMES J. DUDERSTADT, President Emeritus and University Professor of Science and Engineering,
Millennium Project, University of Michigan
MARYE ANNE FOX, Chancellor,
North Carolina State University
RALPH E. GOMORY, President,
Alfred P. Sloan Foundation
RUBY P. HEARN, Vice President,
The Robert Wood Johnson Foundation
BRIGID L. M. HOGAN, Investigator,
Howard Hughes Medical Institute, and
Hortense B. Ingram Professor,
Department of Cell Biology, Vanderbilt University School of Medicine
SAMUEL PRESTON, Dean,
University of Pennsylvania School of Arts and Sciences
KENNETH I. SHINE,* President,
Institute of Medicine
MORRIS TANENBAUM, Retired Vice Chairman and Chief Financial Officer,
AT&T
IRVING L. WEISSMAN, Karel and Avice Beekhuis Professor of Cancer Biology,
Stanford University School of Medicine
SHEILA E. WIDNALL, Abbey Rockefeller Mauze Professor of Aeronautics,
Massachusetts Institute of Technology
WILLIAM JULIUS WILSON, Lewis P. and Linda L. Geyer University Professor,
Harvard University
WILLIAM A. WULF,* President,
National Academy of Engineering
_______________
RICHARD E. BISSELL, Director
DEBORAH D. STINE, Associate Director
MARION RAMSEY, Administrative Associate
COSEPUP Benchmarking Guidance Group
MARYE ANNE FOX (Chair), Chancellor,
North Carolina State University
DAVID CHALLONER, Director,
Institute for Science and Health Policy, University of Florida
ELLIS COWLING, University Distinguished Professor At-Large,
North Carolina State University
GERALD DINNEEN, Retired Vice President,
Science and Technology, Honeywell, Inc.
MILDRED S. DRESSELHAUS, Institute Professor of Electrical Engineering and Physics,
Massachusetts Institute of Technology
ALEXANDER FLAX, Consultant
RALPH E. GOMORY, President,
Alfred P. Sloan Foundation
_______________
DEBORAH D. STINE, Study Director
ALAN ANDERSON, Consultant Writer
REBECCA BURKA, Administrative Associate
AUBREY SABALA, Project Assistant
NORMAN GROSSBLATT, Editor
PREFACE
The American people, through their elected representatives, support the nation's research enterprise in the expectation of substantial returns on their investment: a higher standard of living, a healthier society, an environmentally sustainable economy, and a strong national security. Knowing the power of research in addressing national objectives, the nation has committed itself to a broad set of investments to uphold its research capability.
The National Research Council has already prepared studies that describe the effectiveness of research investments in addressing national concerns. Research investments affect the quality of research done. The present study asks how to evaluate research-leadership status. COSEPUP proposed contributing a set of experiments in international benchmarking. International benchmarking compares the quality and impact of research in one country (or region) with world standards of excellence. The use of international benchmarking was also advocated in 1995 by the "Press report," Allocating Federal Funds for Science and Technology (see appendix B), for the purpose of providing objective information for the executive branch and Congress. The need for objective evaluations has intensified since the passage in 1993 of the Government Performance and Results Act (GPRA), which requires annual performance reports by all federal agencies, including those which support research. GPRA is discussed in the 1999 COSEPUP report Evaluating Federal Research Programs: Research and the Government Performance and Results Act .
Although the use of international benchmarking was not new, it had not been attempted on a scale large enough to contribute to national
policy. Accordingly, in 1997, COSEPUP decided to undertake a set of experiments to test the efficacy of international benchmarking. The committee chose three areas of research—mathematics, immunology, and materials science and engineering—that are quite different from one another in size, funding, numbers of subdisciplines, and other qualities. COSEPUP appointed a panel for each field and sought to experiment with providing information for relatively modest commitments of time and money. Once the panels had completed their reports, the committee held a workshop with agency representatives, congressional staff, and oversight bodies to discuss the findings. (See appendix C for a summary of the workshop.)
This report describes the background, methodology, experimental results, and findings of the international benchmarking experiments and concludes that international benchmarking by a panel of experts can be efficient and reasonably objective. International benchmarking might also be a valuable assessment tool for those seeking to implement GPRA.
Maxine Singer
Chair
Committee on Science, Engineering, and Public Policy
ACKNOWLEDGMENTS
This report is the product of many individuals. First, COSEPUP acknowledges those who made presentations at the Workshop on International Benchmarking: Arden Bement, chair, Panel on Materials Science and Engineering; Irving Weissman, chair, Panel on Immunology; Peter Lax, chair, Panel on Mathematics; and the discussants: Richard Russell, House Committee on Science; Don Lewis, director, Division of Mathematical Sciences, National Science Foundation; Patricia Dehmer, associate director, Office of Science, Office of Basic Energy Sciences, Department of Energy; Helen Quill, chief, Basic Immunology Branch, Division of Allergy, Immunology and Transplantation, National Institute of Allergy and Infectious Diseases, National Institutes of Health; and Irwin Feller, director, Institute for Policy Research and Evaluation, and professor of economics, Pennsylvania State University.
The report was reviewed by persons chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Research Council's Report Review Committee. The purposes of this independent review are to provide candid and critical comments that will assist COSEPUP in making its report as sound as possible and to ensure that the report meets institutional standards of objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following for their participation in the review of this report: Arden L. Bement, Jr., Robert M. White, Margaret H. Wright, Paul A. Fleury, Susan Cozzens, Arthur Bienenstock, and the report review coordinator, Anita
Jones, and the report review monitor, John Ahearne. Although those persons provided many constructive comments and suggestions, responsibility for the content of the report rests solely with COSEPUP.
The production of this report was the result of hard work by the committee as a whole and by the extra effort of the Guidance Group, consisting of current and former COSEPUP members Marye Anne Fox (Chair), David Challoner, Ellis Cowling, Gerald Dinneen, Mildred S. Dresselhaus, Alexander Flax, and Ralph E. Gomory.
The project was aided by the invaluable help of COSEPUP professional staff: Deborah D. Stine, associate director of COSEPUP and study director; Rebecca Burka, administrative associate; Aubrey Sabala, project assistant; Alan Anderson, consultant writer; and Norman Grossblatt, editor. The committee also thanks Richard Bissell, executive director of the Policy Division and director of COSEPUP, who oversaw the committee's activities.