EXPERIMENTS IN INTERNATIONAL BENCHMARKING OF US RESEARCH FIELDS

Committee on Science, Engineering, and Public Policy

National Academy of Sciences

National Academy of Engineering

Institute of Medicine

NATIONAL ACADEMY PRESS
Washington, D.C.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
Experiments in International Benchmarking of US Research Fields EXPERIMENTS IN INTERNATIONAL BENCHMARKING OF US RESEARCH FIELDS Committee on Science, Engineering, and Public Policy National Academy of Sciences National Academy of Engineering Institute of Medicine NATIONAL ACADEMY PRESS Washington, D.C.

OCR for page R1
Experiments in International Benchmarking of US Research Fields NATIONAL ACADEMY PRESS 2101 Constitution Ave., NW Washington, DC 20418 NOTICE: This volume was produced as part of a project approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. It is a result of work done by the Committee on Science, Engineering, and Public Policy (COSEPUP) as augmented, which has authorized its release to the public. This report has been reviewed by a group other than the authors according to procedures approved by COSEPUP and the Report Review Committee. The Committee on Science, Engineering, and Public Policy (COSEPUP) is a joint committee of NAS, NAE, and IOM. It includes members of the councils of all three bodies. Financial Support: The development of this report was supported by the Sloan Foundation and the National Research Council. International Standard Book Number 0-309-06898-3 Internet Access: This report is available on COSEPUP's World Wide Web site at http://www4.nationalacademies.org/pd/cosepup.nsf Order from: National Academy Press, 2101 Constitution Ave., NW, Washington, DC 20418. All orders must be prepaid with delivery to a single address. No additional discounts apply. Prices are subject to change without notice. To order by credit card, call 1-800-624-6242 or 202-334-3313 (in Washington metropolitan area). Copyright 2000 by the National Academy of Sciences. All rights reserved. This document may be reproduced solely for educational purposes without the written permission of the National Academy of Sciences. Printed in the United States of America.

OCR for page R1
Experiments in International Benchmarking of US Research Fields Committee on Science, Engineering, and Public Policy MAXINE F. SINGER (Chair), President, Carnegie Institution of Washington BRUCE M. ALBERTS,* President, National Academy of Sciences ENRIQUETA C. BOND, President, The Burroughs Wellcome Fund LEWIS BRANSCOMB, Professor Emeritus, Center for Science and International Affairs, John F. Kennedy School of Government, Harvard University PETER DIAMOND, Institute Professor and Professor of Economics, Massachusetts Institute of Technology GERALD DINNEEN,* Retired Vice President, Science and Technology, Honeywell, Inc. MILDRED S. DRESSELHAUS, Institute Professor of Electrical Engineering and Physics, Massachusetts Institute of Technology JAMES J. DUDERSTADT, President Emeritus and University Professor of Science and Engineering, Millennium Project, University of Michigan MARYE ANNE FOX, Chancellor, North Carolina State University RALPH E. GOMORY, President, Alfred P. Sloan Foundation RUBY P. HEARN, Vice President, The Robert Wood Johnson Foundation BRIGID L. M. HOGAN, Investigator, Howard Hughes Medical Institute, and Hortense B. Ingram Professor, Department of Cell Biology, Vanderbilt University School of Medicine SAMUEL PRESTON, Dean, University of Pennsylvania School of Arts and Sciences KENNETH I. SHINE,* President, Institute of Medicine MORRIS TANENBAUM, Retired Vice Chairman and Chief Financial Officer, AT&T IRVING L. WEISSMAN, Karel and Avice Beekhuis Professor of Cancer Biology, Stanford University School of Medicine SHEILA E. WIDNALL, Abbey Rockefeller Mauze Professor of Aeronautics, Massachusetts Institute of Technology WILLIAM JULIUS WILSON, Lewis P. and Linda L. Geyer University Professor, Harvard University WILLIAM A. WULF,* President, National Academy of Engineering _______________ RICHARD E. BISSELL, Director DEBORAH D. STINE, Associate Director MARION RAMSEY, Administrative Associate *    Ex officio member

OCR for page R1
Experiments in International Benchmarking of US Research Fields COSEPUP Benchmarking Guidance Group MARYE ANNE FOX (Chair), Chancellor, North Carolina State University DAVID CHALLONER, Director, Institute for Science and Health Policy, University of Florida ELLIS COWLING, University Distinguished Professor At-Large, North Carolina State University GERALD DINNEEN, Retired Vice President, Science and Technology, Honeywell, Inc. MILDRED S. DRESSELHAUS, Institute Professor of Electrical Engineering and Physics, Massachusetts Institute of Technology ALEXANDER FLAX, Consultant RALPH E. GOMORY, President, Alfred P. Sloan Foundation _______________ DEBORAH D. STINE, Study Director ALAN ANDERSON, Consultant Writer REBECCA BURKA, Administrative Associate AUBREY SABALA, Project Assistant NORMAN GROSSBLATT, Editor

OCR for page R1
Experiments in International Benchmarking of US Research Fields PREFACE The American people, through their elected representatives, support the nation's research enterprise in the expectation of substantial returns on their investment: a higher standard of living, a healthier society, an environmentally sustainable economy, and a strong national security. Knowing the power of research in addressing national objectives, the nation has committed itself to a broad set of investments to uphold its research capability. The National Research Council has already prepared studies that describe the effectiveness of research investments in addressing national concerns. Research investments affect the quality of research done. The present study asks how to evaluate research-leadership status. COSEPUP proposed contributing a set of experiments in international benchmarking. International benchmarking compares the quality and impact of research in one country (or region) with world standards of excellence. The use of international benchmarking was also advocated in 1995 by the "Press report," Allocating Federal Funds for Science and Technology (see appendix B), for the purpose of providing objective information for the executive branch and Congress. The need for objective evaluations has intensified since the passage in 1993 of the Government Performance and Results Act (GPRA), which requires annual performance reports by all federal agencies, including those which support research. GPRA is discussed in the 1999 COSEPUP report Evaluating Federal Research Programs: Research and the Government Performance and Results Act . Although the use of international benchmarking was not new, it had not been attempted on a scale large enough to contribute to national

OCR for page R1
Experiments in International Benchmarking of US Research Fields policy. Accordingly, in 1997, COSEPUP decided to undertake a set of experiments to test the efficacy of international benchmarking. The committee chose three areas of research—mathematics, immunology, and materials science and engineering—that are quite different from one another in size, funding, numbers of subdisciplines, and other qualities. COSEPUP appointed a panel for each field and sought to experiment with providing information for relatively modest commitments of time and money. Once the panels had completed their reports, the committee held a workshop with agency representatives, congressional staff, and oversight bodies to discuss the findings. (See appendix C for a summary of the workshop.) This report describes the background, methodology, experimental results, and findings of the international benchmarking experiments and concludes that international benchmarking by a panel of experts can be efficient and reasonably objective. International benchmarking might also be a valuable assessment tool for those seeking to implement GPRA. Maxine Singer Chair Committee on Science, Engineering, and Public Policy

OCR for page R1
Experiments in International Benchmarking of US Research Fields ACKNOWLEDGMENTS This report is the product of many individuals. First, COSEPUP acknowledges those who made presentations at the Workshop on International Benchmarking: Arden Bement, chair, Panel on Materials Science and Engineering; Irving Weissman, chair, Panel on Immunology; Peter Lax, chair, Panel on Mathematics; and the discussants: Richard Russell, House Committee on Science; Don Lewis, director, Division of Mathematical Sciences, National Science Foundation; Patricia Dehmer, associate director, Office of Science, Office of Basic Energy Sciences, Department of Energy; Helen Quill, chief, Basic Immunology Branch, Division of Allergy, Immunology and Transplantation, National Institute of Allergy and Infectious Diseases, National Institutes of Health; and Irwin Feller, director, Institute for Policy Research and Evaluation, and professor of economics, Pennsylvania State University. The report was reviewed by persons chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Research Council's Report Review Committee. The purposes of this independent review are to provide candid and critical comments that will assist COSEPUP in making its report as sound as possible and to ensure that the report meets institutional standards of objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following for their participation in the review of this report: Arden L. Bement, Jr., Robert M. White, Margaret H. Wright, Paul A. Fleury, Susan Cozzens, Arthur Bienenstock, and the report review coordinator, Anita

OCR for page R1
Experiments in International Benchmarking of US Research Fields Jones, and the report review monitor, John Ahearne. Although those persons provided many constructive comments and suggestions, responsibility for the content of the report rests solely with COSEPUP. The production of this report was the result of hard work by the committee as a whole and by the extra effort of the Guidance Group, consisting of current and former COSEPUP members Marye Anne Fox (Chair), David Challoner, Ellis Cowling, Gerald Dinneen, Mildred S. Dresselhaus, Alexander Flax, and Ralph E. Gomory. The project was aided by the invaluable help of COSEPUP professional staff: Deborah D. Stine, associate director of COSEPUP and study director; Rebecca Burka, administrative associate; Aubrey Sabala, project assistant; Alan Anderson, consultant writer; and Norman Grossblatt, editor. The committee also thanks Richard Bissell, executive director of the Policy Division and director of COSEPUP, who oversaw the committee's activities.

OCR for page R1
Experiments in International Benchmarking of US Research Fields CONTENTS     EXECUTIVE SUMMARY   1 1   INTRODUCTION   4 2   METHODOLOGY   9     2.1 General Features of Methodology   9     2.2 Specific Charge to the Panels   10     2.3 Selection of Panel Members   10     2.4 Selection of Research Fields   11     2.5 Evaluation of Panel Results   12 3   RESULTS OF THE BENCHMARKING EXPERIMENTS   13     3.1 How Panels Were Selected   13     3.2 How Panels Assessed Their Fields   14     3.2.1 The Virtual Congress   15     3.2.2 Citation Analysis   16     3.2.3 Journal Publication Analysis   16     3.2.4 Quantitative Data Analysis   17     3.2.5 Prize Analysis   17     3.2.6 International Congress Speakers   17     3.3 How Panels Characterized US Research   18     3.4 Factors Influencing US Performance   18     3.5 Future Relative Position of US Research   19

OCR for page R1
Experiments in International Benchmarking of US Research Fields 4   FINDINGS   20     4.1 Findings About Objectives   21     4.2 Findings About Results   21     4.3 Findings About the Methodology of Benchmarking   21     4.4 Findings About the Membership of Panels   22     4.5 Findings About the Utility of Benchmarking   23 5   DISCUSSION   24     5.1 Some Strengths of Benchmarking   24     5.2 Some Weaknesses of Benchmarking   24     5.3 Other Observations About Benchmarking   25 6   CONCLUSION   27     APPENDIXES     A   COMMITTEE ON SCIENCE, ENGINEERING, AND PUBLIC POLICY: MEMBER AND STAFF BIOGRAPHICAL INFORMATION   29 B   EXCERPTS FROM NATIONAL ACADEMIES REPORTS:   37     B-1 Excerpt from: Science, Technology, and the Federal Government: National Goals for a New Era   39     B-2 Excerpt from: Allocating Federal Funds for Science and Technology   46 C   WORKSHOP ON INTERNATIONAL BENCHMARKING OF US RESEARCH   49     ATTACHMENTS     I   INTERNATIONAL BENCHMARKING OF US MATHEMATICS RESEARCH     II   INTERNATIONAL BENCHMARKING OF US MATERIALS SCIENCE AND ENGINEERING RESEARCH     III   INTERNATIONAL BENCHMARKING OF US IMMUNOLOGY RESEARCH