Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page R1
Steve Olson and Stephen Merrill, Rapporteurs
Committee on Measuring Economic and Other Returns
on
Federal Research Investments
Board on Science, Technology, and Economic Policy
Committee on Science, Engineering, and Public Policy
Policy and Global Affairs
OCR for page R2
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the
Governing Board of the National Research Council, whose members are drawn
from the councils of the National Academy of Sciences, the National Academy
of Engineering, and the Institute of Medicine. The members of the committee
responsible for the report were chosen for their special competences and with
regard for appropriate balance.
This study was supported by Contract/Grant No. SMA-1019816 between the
National Academy of Sciences and the National Science Foundation;
Contract/Grant No. N01-OD-4-2139, TO #231, between the National Academy
of Sciences and the National Institutes of Health; Contract/Grant No.
G104P00159 between the National Academy of Sciences and the U.S.
Geological Survey; Contract/Grant No. 59-9000-0-0093 between the National
Academy of Sciences and the U.S. Department of Agriculture; Contract/Grant
No. EP-11-H-001414 between the National Academy of Sciences and the
Environmental Protection Agency; Contract/Grant No. DE-SC000614 between
the National Academy of Sciences and the Department of Energy;
Contract/Grant No. NNH10CC488,TO #5, between the National Academy of
Sciences and NASA. Any opinions, findings, conclusions, or recommendations
expressed in this publication are those of the author(s) and do not necessarily
reflect the views of the organizations or agencies that provided support for the
project
International Standard Book Number -13:978-0-309-21748-4
International Standard Book Number -10:0-309-21748-2
Additional copies of this report are available from the National Academies
Press, 500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-
6242 or (202) 334-3313 (in the Washington metropolitan area); Internet,
http://www.nap.edu.
Cover: The cover design incorporates a feature of the 1924 National Academy
of Sciences building in Washington. Sculpted by Lee Lawrie, the bronze
cheneau, running the length of the roof, features alternating figures of owls and
lynxes, symbolizing wisdom and alert observation, respectively.
Copyright 2011 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
OCR for page R3
The National Academy of Sciences is a private, nonprofit, self-perpetuating
society of distinguished scholars engaged in scientific and engineering research,
dedicated to the furtherance of science and technology and to their use for the
general welfare. Upon the authority of the charter granted to it by the Congress
in 1863, the Academy has a mandate that requires it to advise the federal
government on scientific and technical matters. Dr. Ralph J. Cicerone is
president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the
charter of the National Academy of Sciences, as a parallel organization of
outstanding engineers. It is autonomous in its administration and in the selection
of its members, sharing with the National Academy of Sciences the
responsibility for advising the federal government. The National Academy of
Engineering also sponsors engineering programs aimed at meeting national
needs, encourages education and research, and recognizes the superior
achievements of engineers. Dr. Charles M. Vest is president of the National
Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of
Sciences to secure the services of eminent members of appropriate professions
in the examination of policy matters pertaining to the health of the public. The
Institute acts under the responsibility given to the National Academy of Sciences
by its congressional charter to be an adviser to the federal government and, upon
its own initiative, to identify issues of medical care, research, and education. Dr.
Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of
Sciences in 1916 to associate the broad community of science and technology
with the Academy’s purposes of furthering knowledge and advising the federal
government. Functioning in accordance with general policies determined by the
Academy, the Council has become the principal operating agency of both the
National Academy of Sciences and the National Academy of Engineering in
providing services to the government, the public, and the scientific and
engineering communities. The Council is administered jointly by both
Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles
M. Vest are chair and vice chair, respectively, of the National Research Council.
www.national-academies.org
OCR for page R4
OCR for page R5
COMMITTEE ON MEASURING ECONOMIC AND OTHER
RETURNS ON FEDERAL RESEARCH INVESTMENTS
NEAL LANE (Co-Chair), Malcolm Gillis University Professor, Rice
University
BRONWYN HALL (Co-Chair), Professor of Economics, University of
California at Berkeley and University of Maastricht
ALAN GARBER, Henry J. Kaiser, Jr. Professor and Professor of
Medicine; Director, Center for Health Policy, Stanford University
PAULA STEPHAN, Professor of Economics, Georgia State University
PRABHU PINGALI, Deputy Director, Agricultural Development,
Global Development Program, The Bill and Melinda Gates
Foundation
WALTER POWELL, Professor of Education, Stanford University and
External Professor, The Santa Fe Institute
DAVID GOLDSTON, Director, Government Affairs, Natural
Resources Defense Council
ALEXANDER FRIEDMAN, Chief Investment Officer, UBS Wealth
Management
JOHN STASKO, Professor and Associate Chair, School of Interactive
Computing, Georgia Institute of Technology
ALFRED SPECTOR, Vice President of Research and Special
Initiatives, Google, Inc.
ERIC WARD, President, The Two Blades Foundation
NEELA PATEL, Director of External Research, Global Pharmaceutical
R and D, Abbott Laboratories
MICHAEL TURNER, Bruce V. and Diana M. Rauner Distinguished
Service Professor, Kavli Institute for Cosmological Physics, The
University of Chicago
Staff
STEPHEN A. MERRILL, Project Director
GURUPRASAD MADHAVAN, Program Officer and Project Co-
director
KEVIN FINNERAN, Director, Committee on Science, Engineering,
and Public Policy
v
OCR for page R6
STEVE OLSON, Consultant Writer
DANIEL MULLINS, Program Associate
CYNTHIA GETNER, Financial Associate
vi
OCR for page R7
BOARD ON SCIENCE, TECHNOLOGY, AND ECONOMIC
POLICY
National Research Council
PAUL JOSKOW (Chair), President, Alfred P. Sloan Foundation
LEWIS COLEMAN, President, DreamWorks Animation
JOHN DONOVAN, Chief Technology Officer, AT and T
ALAN GARBER, Henry J. Kaiser, Jr. Professor and Professor of
Medicine; Director, Center for Health Policy, Stanford University
RALPH GOMORY, President Emeritus, Alfred P. Sloan Foundation
MARY GOOD, Donaghey University Professor and Dean Emeritus,
Donaghey College of Information Science and Systems Engineering,
University of Arkansas at Little Rock
RICHARD LESTER, Professor and Department Head, Department of
Nuclear Science and Engineering, Massachusetts Institute of
Technology
AMORY HOUGHTON, Jr., Former Member of Congress
DAVID MORGENTHALER, Founding Partner, Morgenthaler
Ventures
WILLIAM MEEHAN, Lecturer in Strategic Management and Raccoon
Partners Lecturer in Management, Stanford Graduate School of
Business; and Director Emeritus, McKinsey and Company
JOSEPH NEWHOUSE, John D. MacArthur Professor of Health Policy
and Management and Director, Division of Health Policy Research
and Education, Harvard University
EDWARD PENHOET, Director, Alta Partners
ARATI PRABHAKAR, General Partner, U.S. Venture Partners
WILLIAM RADUCHEL, Strategic Advisor and Independent Director
KATHYRN SHAW, Earnest C. Arbuckle Professor of Economics,
Graduate School of Business, Stanford University
LAURA D'ANDREA TYSON, S.K. and Angela Chan Professor of
Global Management, Haas School of Business, University of
California, Berkeley
HAL VARIAN, Chief Economist, Google, Inc.
ALAN WM. WOLFF, Of Counsel, Dewey and LeBoeuf LLP
vii
OCR for page R8
Ex-Officio Members
RALPH CICERONE, President, National Academy of Sciences
CHARLES VEST, President, National Academy of Engineering
HARVEY FINEBERG, President, Institute of Medicine
Staff
STEPHEN A. MERRILL, Executive Director
CHARLES WESSNER, Program Director
SUJAI SHIVAKUMAR, Senior Program Officer
DAVID DIERKSHEIDE, Program Officer
MCALISTER CLABAUGH, Program Officer
PAUL BEATON, Program Officer
CYNTHIA GETNER, Financial Associate
DANIEL MULLINS, Program Associate
DAVID DAWSON, Program Associate
viii
OCR for page R9
COMMITTEE ON SCIENCE, ENGINEERING, AND
PUBLIC POLICY
National Academy of Sciences
National Academy of Engineering
Institute of Medicine
GEORGE WHITESIDES (Chair), Woodford L. and Ann A. Flowers
University Professor, Harvard University
LINDA ABRIOLA, Dean of Engineering, Tufts University
CLAUDE CANIZARES, Vice President for Research, Associate
Provost and Bruno Rossi Professor of Experimental Physics,
Massachusetts Institute of Technology
MOSES CHAN, Evan Pugh Professor of Physics, Pennsylvania State
University
RALPH CICERONE (Ex-Officio), President, National Academy of
Sciences
PAUL CITRON, Retired Vice President, Technology Policy and
Academic Relations, Medtronic, Inc.
RUTH DAVID, President and Chief Executive Officer, ANSER
(Analytic Services), Inc.
HARVEY FINEBERG (Ex-Officio), President, Institute of Medicine
JUDITH KIMBLE, Investigator, Howard Hughes Medical Institute;
Professor of Biochemistry and Molecular Biology and Medical
Genetics, University of Wisconsin
DAN MOTE, Jr. (Ex-Officio), President and Glenn Martin Institute
Professor of Engineering, University of Maryland
PERCY PIERRE, Vice President and Professor Emeritus, Michigan
State University
ALBERT REECE, Vice President for Medical Affairs, Bowers
Distinguished Professor and Dean, School of Medicine, University of
Maryland, Baltimore
SUSAN SCRIMSHAW, President, The Sage Colleges
WILLIAM SPENCER, Chairman Emeritus, SEMATECH
MICHAEL TURNER, Bruce V. and Diana M. Rauner Distinguished
Service Professor, Kavli Institute for Cosmological Physics, The
University of Chicago
ix
OCR for page R10
CHARLES VEST (Ex-Officio), President, National Academy of
Engineering
NANCY WEXLER, Higgins Professor of Neuropsychology, Columbia
University
Staff
KEVIN FINNERAN, Director
THOMAS ARRISON, Senior Program Officer
GURUPRASAD MADHAVAN, Program Officer
PETER HUNSBERGER, Financial Associate
MARION RAMSEY, Administrative Associate
NEERAJ GORKHALY, Research Associate
x
OCR for page R11
ACKNOWLEDGMENT OF REVIEWERS
This report has been reviewed in draft form by individuals
chosen for their diverse perspectives and technical expertise, in
accordance with procedures approved by the National Academies’
Report Review Committee. The purpose of this independent review is to
provide candid and critical comments that will assist the institution in
making its published report as sound as possible and to ensure that the
report meets institutional standards for quality and objectivity. The
review comments and draft manuscript remain confidential to protect the
integrity of the process.
We wish to thank the following individuals for their review of
this report: George Bo-Linn, Gordon and Betty Moore Foundation;
Susan Cozzens, Georgia Institute of Technology; Kenneth Gertz,
University of Maryland; Diana Hicks, Georgia Institute of Technology;
and Peter Hussey, RAND Corporation.
Although the reviewers listed above have provided many
constructive comments and suggestions, they were not asked to endorse
the content of the report, nor did they see the final draft before its release.
Responsibility for the final content of this report rests entirely with the
rapporteurs and the institution.
xi
OCR for page R12
OCR for page R13
CONTENTS
1 INTRODUCTION AND OVERVIEW ............................................... 1
2 THE USES AND MISUSES OF PERFORMANCE MEASURES..... 7
The Promise and the Limits of Measuring the Impact of Federally
Supported Research ........................................................................ 7
Innovation as an Ecosystem............................................................... 11
Overcoming the Challenges of Research Measures .......................... 14
Discussion .......................................................................................... 16
3 IMPACTS ON THE U.S. ECONOMY AND QUALITY OF LIFE.. 19
Federal Research and Productivity .................................................... 19
Indirect Economic Benefits of Research ........................................... 21
Beyond Citations and Patent Reference Counts ................................ 22
Discussion .......................................................................................... 23
4 IMPACTS ON BIOMEDICAL AND HEALTH RESEARCH ......... 25
Reviewing the Literature on Health Impacts ..................................... 25
The Volatility of Federal R and D Support ....................................... 29
Medical Device Innovation................................................................ 30
Making Decisions in the Pharmaceutical Industry ............................ 31
Research and Outcomes Case Study: Pediatric HIV ......................... 33
Discussion .......................................................................................... 35
5 MIXED MARKET AND NON-MARKET IMPACTS OF
RESEARCH ...................................................................................... 37
Measuring Progress toward Goals in Agricultural Productivity ........ 37
Investment Decisions at DuPont........................................................ 39
Challenges in Quantifying Research Value in Agriculture ............... 40
Measuring Success in Conservation .................................................. 42
National Security Benefits ................................................................. 44
Public Problem Solving ..................................................................... 46
Discussion .......................................................................................... 47
6 IMPACTS OF RESEARCH ON THE LABOR MARKET AND
CAREER DEVELOPMENT ............................................................. 49
R and D Spending and the R and D Workforce ................................. 49
Surveys of Graduate Students and Postdoctoral Fellows .................. 51
xiii
OCR for page R14
The Complex Network of Skills and Investments ............................. 55
Discussion .......................................................................................... 59
7 INTERNATIONAL PERSPECTIVES ON MEASURING
RESEARCH IMPACTS .................................................................... 61
Medical Research Council Evaluation System .................................. 61
Measuring Impacts of Research Funding In the European Union ..... 63
Measuring Impacts of Science, Technology, and Innovation
Investments in Brazil .................................................................... 66
Discussion .......................................................................................... 67
8 EMERGING METRICS AND MODELS ......................................... 69
Assessing Research at NSF ............................................................... 69
The STAR Metrics Project ................................................................ 72
Reconstructing Networks of Discovery ............................................. 75
Creating Knowledge from Data ......................................................... 76
Measuring the Impact of Star Scientists ............................................ 78
Visual Analytics ................................................................................ 79
Considerations in Building Comprehensive Databases ..................... 81
Discussion .......................................................................................... 82
9 PITFALLS, PROGRESS, AND OPPORTUNITIES ......................... 85
Pitfalls on the Road to Understanding ............................................... 85
Progress in Understanding the Issues ................................................ 86
Opportunities Posed by Greater Understanding ................................ 87
Concluding Remarks ......................................................................... 88
REFERENCES ....................................................................................... 89
A WORKSHOP AGENDA .................................................................. 91
B BIOGRAPHICAL INFORMATION ................................................ 99
C THE PROMISES AND LIMITATIONS OF PERFORMANCE
MEASURES, Irwin Feller ............................................................... 119
D THE IMPACT OF PUBLICLY FUNDED BIOMEDICAL AND
HEALTH RESEARCH: A REVIEW, Bhaven Sampat ................... 153
xiv
