Evaluating Federal Research Programs: Research and the Government Performance and Results Act (1999)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

EVALUATING FEDERAL RESEARCH PROGRAMS Research and the Government Performance and Results Act Committee on Science, Engineering, and Public Policy National Academy of Sciences National Academy of Engineering Institute of Medicine NATIONAL ACADEMY PRESS Washington, D.C. 1999 

NATIONAL ACADEMY PRESS • 2101 Constitution Ave., NW • Washington, DC 20418 NOTICE: This volume was produced as part of a project approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. It is a result of work done by the Committee on Science, Engineering, and Public Policy (COSEPUP) as augmented, which has authorized its release to the public. This report has been reviewed by a group other than the authors according to procedures approved by COSEPUP and the Report Review Committee. The National Academy of Sciences (NAS) is a private, nonprofit, self- perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Under the authority of the charter granted to it by Congress in 1863, the Academy has a working mandate that calls on it to advise the federal gov- ernment on scientific and technical matters. Dr. Bruce M. Alberts is president of NAS. The National Academy of Engineering (NAE) was established in 1964, under the charter of NAS, as a parallel organization of distinguished engineers. It is autonomous in its administration and in the selection of members, sharing with NAS its responsibilities for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. William A. Wulf is president of NAE. The Institute of Medicine (IOM) was established in 1970 by NAS to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to NAS in its congressional charter to be an adviser to the federal government and, on its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of IOM. The Committee on Science, Engineering, and Public Policy (COSEPUP) is a joint committee of NAS, NAE, and IOM. It includes members of the councils of all three bodies. Financial Support: The development of this report was supported by the National Research Council, the National Science Foundation, the National Institutes of Health, the National Aeronautics and Space Administration, the Department of Agriculture, the Department of Transportation, and the Department of Defense. International Standard Book Number 0-309-06430-9 Limited copies are available from: Committee on Science, Engineering, and Public Policy, 2101 Constitution Ave., NW, Washington, DC 20418. Additional copies are available for sale from: National Academy Press, Box 285, 2101 Constitution Ave., NW, Washington, DC 20055. You may order by phone at 1-800- 624-6242 (202-334-3313 in the Washington metropolitan area) or by visiting the National Academy Press web site, www.nap.edu. Copyright 1999 by the National Academy of Sciences. All rights reserved. Printed in the United States of America ii 

COMMITTEE ON SCIENCE, ENGINEERING, AND PUBLIC POLICY PHILLIP A. GRIFFITHS, (Chair), Director, Institute for Advanced Study, Princeton, New Jersey BRUCE M. ALBERTS,* President, National Academy of Sciences, Washington, D.C. PETER DIAMOND, Professor of Economics, Massachusetts Institute of Technology, Cambridge GERALD DINNEEN,* Retired Vice President, Science and Technology, Honeywell, Inc., Edina, Minnesota MILDRED S. DRESSELHAUS, Institute Professor of Electrical Engineering and Physics, Massachusetts Institute of Technology, Cambridge JAMES J. DUDERSTADT, President Emeritus and University Professor of Science and Engineering, Millennium Project, University of Michigan, Ann Arbor MARYE ANNE FOX, Chancellor, North Carolina State University, Raleigh RALPH E. GOMORY, President, Alfred P. Sloan Foundation, New York, New York RUBY P. HEARN, Vice President, The Robert Wood Johnson Foundation, Princeton, New Jersey PHILIP W. MAJERUS, Professor of Medicine, Biochemistry, and Molecular Biophysics and Director, Division of Hematology-Oncology, Washington University School of Medicine, St. Louis, Missouri JUNE E. OSBORN, President, Josiah Macy, Jr. Foundation, New York, New York KENNETH I. SHINE,* President, Institute of Medicine, Washington, D.C. MORRIS TANENBAUM, Retired Vice Chairman and Chief Financial Officer, AT&T, Short Hills, New Jersey WILLIAM JULIUS WILSON, Malcolm Wiener Professor, Center for Social Policy, John F. Kennedy School of Government, Harvard University, Cambridge, Massachusetts WILLIAM A. WULF,* President, National Academy of Engineering, Washington, D.C. *Ex officio member RICHARD BISSELL, Executive Director DEBORAH D. STINE, Associate Director and Study Director ANNE-MARIE MAZZA, Senior Program Officer BRETT WILLETTE, Research Associate CAROLYN RYAN, Research Associate MARION RAMSEY, Administrative Associate SUSAN COZZENS, Consultant ALAN ANDERSON, Consultant Science Writer NORMAN GROSSBLATT, Editor iii 



Preface he 1993 Government Performance and Results Act (“the Results T Act,” or GPRA) requires federal agencies to set strategic goals and to use performance measures for management and budgeting. The objective of GPRA is to encourage greater efficiency, effectiveness, and accountability in federal programs and spending. Development of plans to implement the act has been particularly difficult for agencies responsible for research activities supported by the federal government because of the difficulty of linking results with annual investments in research. The Committee on Science, Engineering, and Public Policy (COSEPUP) of the National Academy of Sci- ences, the National Academy of Engineering, and the Institute of Medicine is conducting a three-part study to: 1. Identify and analyze the most effective ways to assess the results of research, on the basis of consultation with federal agencies, oversight entities, the research community, industry, states, and agencies of other nations. 2. Help the federal government determine how its agen- cies can better incorporate research activities in strategic and performance plans and improve the management and effectiveness of research programs, including a determination of what can be reliably measured and the best mechanism for doing so, a determi- nation of what cannot be measured, an evaluation of the extent to which common analytic paradigms could be used across agencies for assessing the results of extramural and intramural research programs and of the training and education of the scientific and engineering v 

✯ EVALUATING FEDERAL RESEARCH PROGRAMS workforce, development of such paradigms if feasible, and develop- ment of implementation principles and guidelines that could assist Congress and the Office of Management and Budget (OMB) in reviewing agency performance plans. 3. Develop mechanisms to evaluate the effects of imple- menting GPRA on agency program decisions and on the practices of research, which requires COSEPUP to identify lessons learned and best practices that could be used by other agencies or programs and to determine effective ways for Congress and OMB to use the results of these plans. The committee plans to begin its work on the third charge in the Spring of 1999. The current study, which addresses parts 1 and 2 described above, began in January 1998. During Part 1 of the study, COSEPUP held three workshops: the first focused on industry methods to evaluate the performance of research, the second on agencies’ strategic and performance plans, and the third on methods to evaluate the outcomes of research. Each workshop provided valuable information and a chance for input from all interested parties, including the agencies, the Office of Science and Technol- ogy Policy, OMB, congressional committees, the General Account- ing Office, disciplinary societies, and researchers. The committee thanks all those who attended for their valuable participation. The first workshop indicated that applied research is pursued widely in technology-based industries, is focused on achieving well-defined practical outcomes, and is planned and measured in ways similar to ways that other high-risk business undertakings are planned and measured—with business cases, risk- adjusted rate-of-return requirements, and so on. Basic research, where it is still supported by industry, is focused on well-defined areas when such work is judged to be able to lead to technological breakthroughs. Basic research programs also serve as a human resource pool for a firm and as a working contact with similar basic research performed worldwide. Basic research is supported and vi 

Preface evaluated on the basis of how well it serves those purposes, that is, is the work relevant to the firm’s future? Is it of high quality? Are the best people recruited to do it? The second workshop demonstrated the wide range of procedures and methods that agencies have used in responding to GPRA’s requirements for planning programs and measuring outcomes. Because applied research programs by definition have desired outcomes directly related to agency missions, evaluating such programs can be relatively straightforward and agencies can use methods similar to those used by industry. We found variations of these methods in place in some agencies and recommend that they be fully and consistently used. It became clear, however, that substantial problems existed for agencies trying to evaluate basic research programs. Urgent concern was expressed that basic research could not be effectively evaluated in the context of GPRA and that misguided attempts to do so could cause great damage. We strongly agree that misguided attempts must be avoided, and we recommend how evaluation can be meaningfully performed. Since measuring the outcomes from basic research is the most troublesome for the agencies and, in COSEPUP’s opinion, the most crucial issue for the long-term economic health of the nation, it has received the most attention in this report. We also deal with measuring the outcomes of applied research and the question of the coordination of planning for all federal research programs. COSEPUP received encouragement in conducting this study from Representatives F. James Sensenbrenner, Jr., (chair, House Science Committee) and George Brown (member and former chair of the House Science Committee). In a letter (Appen- dix B) from them, they described the broad range of issues that they see for the planning and evaluation of research in the context of GPRA. This study addresses a portion of those issues, and future work will address other portions. We have limited this discussion to general principles. Each of our recommendations requires specific guidelines for implemen- vii 

EVALUATING FEDERAL RESEARCH PROGRAMS tation, and we have not attempted to provide such guidelines in this report. Because GPRA is in operation today with annual reporting and planning requirements, it seemed important to state general principles as soon as possible and, to the degree that they are acceptable, to direct future work to provide more specific recom- mendations on implementation. Following the release of this report, COSEPUP will arrange meetings and workshops with agency representatives, congressional staff, and oversight bodies to identify the recommendations they would like COSEPUP to develop further. PHILLIP A. GRIFFITHS Chair Committee on Science, Engineering, and Public Policy viii 

ACKNOWLEDGMENTS his report has benefited from input from various individuals. T COSEPUP acknowledges those who made presentations at the workshops and panel discussions: John McTague, Ford Motor Company; Pradip Das, Analytical Sciences Center, Monsanto; Melvin Cohen, Lucent Technologies; David Thomas, Amoco Corporation; Rajen Dalal, Chiron Corporation; Charles Larson, Industrial Research Institute, Inc.; David Rust, George Cooper, Tom Sexton, Bob MacDonald, and Sally Rockey, U.S. Department of Agriculture; Ernest Moniz, Patricia Dehmer, and Robert Astheimer, U.S. Department of Energy; Harold Varmus, Lana Skirboll, John Uzzell, and Wendy Baldwin, National Institutes of Health; Alan Ladwig and Gary Steinberg, National Aeronautics and Space Administration; Neal Lane, Judy Sunley, and Dave Schindel, National Science Foundation; Bob Trew, Joanna Spriggs, and Ed Brown, U.S. Department of Defense; Dani Brzezinska, Fenton Carey, and Paula Ewen, U.S. Department of Transportation; Kevin Teichman, Environmental Protection Agency; Curt Marshall, National Oceanic and Atmospheric Administration; William Ott and Paul Doremus, National Institute of Standards and Technol- ogy; T.J. Glauthier and Sarah Horrigan, Office of Management and Budget; Beverly Hartline, Office of Science and Technology Policy; Phillip Kiko, Beth Sokul, and Scott Rayder, House Science Com- mittee; Robert Simon, Senator Jeff Bingaman’s office; Genevieve Knezo, Congressional Research Service; Susan Kladiva, General Accounting Office; Henry Small, Institute for Scientific Informa- tion; Al Link, University of North Carolina; Irwin Feller, Pennsyl- ix 

EVALUATING FEDERAL RESEARCH PROGRAMS vania State University; Paul David, Stanford University and Oxford; Barry Bozeman, Georgia Institute of Technology; and Deborah Boehm-Davis, George Mason University. In addition, COSEPUP would like to thank all those who attended the workshops for their valuable participation This report has been reviewed by persons chosen for their diverse perspectives and technical expertise in accordance with procedures approved by the National Research Council’s Report Review Committee. The purposes of this independent review are to provide candid and critical comments that will assist COSEPUP in making its report as sound as possible and to ensure that the report meets institutional standards of objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following for their participation in the review of this report: Harvey Bernstein, Barry Bozeman, Rad Byerly, Linda Capuano, Irwin Feller, William Gordon, Paul Gray, Marcelle LaFollette, Stephen Lukasik, Bill Morrill, Ann Peterson, Robert Shannon, Christine Sloane, Bob White, and the report review coordinator, Anita Jones. Although those persons have provided many constructive comments and suggestions, responsibility for the final content of this report rests solely with COSEPUP. The production of the report was the result of hard work by the committee as a whole and by the extra effort of the Guidance Group consisting of Peter Diamond, Gerald Dinneen, Mildred Dresselhaus, Ralph Gomory, Philip Majerus, Kenneth Shine, Morris Tanenbaum, and me, co chaired by Ralph Gomory and Morris Tanenbaum, which convened between regular COSEPUP meetings. The project was aided by the invaluable help of COSEPUP professional staff—Deborah D. Stine, associate director of COSEPUP; Anne-Marie Mazza, senior program officer; Brett Willette, research associate; Carolyn Ryan, research associate; x 

Acknowledgments Preface Michael Hostetler, intern, and its consultants, Susan Cozzens, who provided guidance to the committee on GPRA, and Alan Anderson, science writer. They all helped to draft and revise the report throughout the project. The committee also thanks Richard Bissell, executive director of COSEPUP, who oversaw the committee’s activities, and the report’s diligent editor, Norman Grossblatt. PHILLIP A. GRIFFITHS Chair Committee on Science, Engineering, and Public Policy xi 



Contents EXECUTIVE SUMMARY 1 1 STATEMENT OF THE PROBLEM GPRA and Research 13 13 Findings from Workshops 15 2 RESEARCH AND THE FEDERAL GOVERNMENT 25 Results of Federal Investment in Research 25 Goals of Federal Investments in Research 26 3 MEASURING AND EVALUATING FEDERALLY FUNDED RESEARCH 29 Measuring Research 29 Using Expert Review to Evaluate Research Programs 32 Enhancing the Expert Review Process 34 Summary 35 4 RECOMMENDATIONS 37 APPENDIXES A MEMBERS’ BIOGRAPHICAL INFORMATION 43 B HOUSE SCIENCE COMMITTEE LETTER 53 C PROJECT SUMMARY 55 D GOVERNMENT PERFORMANCE AND RESULTS ACT 57 E REFERENCES 80 xiii