National Academies Press: OpenBook
Suggested Citation:"Front Matter." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page R1
Suggested Citation:"Front Matter." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page R2
Suggested Citation:"Front Matter." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page R3
Suggested Citation:"Front Matter." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page R4
Suggested Citation:"Front Matter." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page R5
Suggested Citation:"Front Matter." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page R6
Suggested Citation:"Front Matter." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page R7
Page viii Cite
Suggested Citation:"Front Matter." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page R8
Suggested Citation:"Front Matter." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page R9
Suggested Citation:"Front Matter." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page R10
Suggested Citation:"Front Matter." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page R11
Suggested Citation:"Front Matter." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page R12
Page xiii Cite
Suggested Citation:"Front Matter." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page R13
Suggested Citation:"Front Matter." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page R14
Suggested Citation:"Front Matter." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page R15
Suggested Citation:"Front Matter." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page R16
Page xvii Cite
Suggested Citation:"Front Matter." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page R17
Page xviii Cite
Suggested Citation:"Front Matter." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page R18
Suggested Citation:"Front Matter." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page R19
Suggested Citation:"Front Matter." National Research Council. 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press. doi: 10.17226/12150.
×
Page R20

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Committee on Evaluating the Efficiency of Research and Development Programs at the U.S. Environmental Protection Agency Committee on Science, Engineering, and Public Policy Policy and Global Affairs Board on Environmental Studies and Toxicology Division on Earth and Life Studies

THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Insti- tute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This project was supported by Contract 68-C-03-081 between the National Academy of Sciences and the U.S. Environmental Protection Agency. Any opinions, findings, conclu- sions, or recommendations expressed in this publication are those of the authors and do not necessarily reflect the view of the organizations or agencies that provided support for this project. International Standard Book Number-13 978-0-309-11684-8 International Standard Book Number-10 0-309-11684-8 Additional copies of this report are available from The National Academies Press 500 Fifth Street, NW Box 285 Washington, DC 20055 800-624-6242 202-334-3313 (in the Washington metropolitan area) http://www.nap.edu Copyright 2008 by the National Academy of Sciences. All rights reserved. Printed in the United States of America.

The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the re- sponsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the Na- tional Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council. www.national-academies.org

COMMITTEE ON EVALUATING THE EFFICIENCY OF RESEARCH AND DEVELOPMENT PROGRAMS AT THE U.S. ENVIRONMENTAL PROTECTION AGENCY Members GILBERT S. OMENN (Chair), University of Michigan, Ann Arbor GEORGE V. ALEXEEFF, California Environmental Protection Agency, Oakland RADFORD BYERLY, JR., University of Colorado, Boulder EDWIN H. CLARK II, Earth Policy Institute, Washington, DC SUSAN E. COZZENS, Georgia Institute of Technology, Atlanta LINDA J. FISHER, E. I. du Pont de Nemours and Company, Wilmington, DE J. PAUL GILMAN, Oak Ridge Center for Advanced Studies, Oak Ridge, TN T.J. GLAUTHIER, TJG Energy Associates, LLC, Moss Beach, CA CAROL J. HENRY, Independent Consultant, Bethesda, MD ROBERT J. HUGGETT, College of William and Mary, Seaford, VA SALLY KATZEN, George Mason University School of Law, Fairfax, VA TERRY F. YOUNG, Environmental Defense, Oakland, CA Staff RICHARD BISSELL, Executive Director, Committee on Science, Engineering, and Public Policy DEBORAH STINE, Associate Director, Committee on Science, Engineering, and Public Policy (up to August 2007) EILEEN ABT, Senior Program Officer ALAN ANDERSON, Consultant Writer NORMAN GROSSBLATT, Senior Editor JENNIFER SAUNDERS, Associate Program Officer RAE BENEDICT, Mirzayan Science & Technology Policy Fellow MIRSADA KARALIC-LONCAREVIC, Manager, Toxicology Information Center NEERAJ P. GORKHALY, Senior Program Assistant MORGAN R. MOTTO, Senior Program Assistant Sponsor U.S. ENVIRONMENTAL PROTECTION AGENCY v

COMMITTEE ON SCIENCE, ENGINEERING, AND PUBLIC POLICY Members GEORGE WHITESIDES (Chair), Woodford L. and Ann A. Flowers University Professor, Harvard University, Boston, MA CLAUDE R. CANIZARES, Vice President for Research, Massachusetts Institute of Technology, Cambridge RALPH J. CICERONE (Ex officio), President, National Academy of Sciences, Washington, DC EDWARD F. CRAWLEY, Executive Director, CMI, and Professor, Massachusetts Institute of Technology, Cambridge RUTH A. DAVID, President and Chief Executive Officer, Analytic Services, Inc., Arlington, VA HAILE T. DEBAS, Executive Director, UCSF Global Health Sciences, Maurice Galante Distinguished Professor of Surgery, San Francisco, CA HARVEY FINEBERG (Ex officio), President, Institute of Medicine, Washington, DC JACQUES S. GANSLER, Vice President for Research, University of Maryland, College Park ELSA M. GARMIRE, Professor, Dartmouth College, Hanover, NH M. R. C. GREENWOOD (Ex officio), Professor of Nutrition and Internal Medicine, University of California, Davis W. CARL LINEBERGER, Professor of Chemistry, University of Colorado, Boulder C. DAN MOTE, JR. (Ex officio), President and Glenn Martin Institute Professor of Engineering, University of Maryland, College Park ROBERT M. NEREM, Parker H. Petit Professor and Director, Institute for Bioengineering and Bioscience, Georgia Institute of Technology, Atlanta LAWRENCE T. PAPAY, Retired, Sector Vice President for Integrated Solutions, Science Applications International Corporation, La Jolla, CA ANNE C. PETERSEN, Professor of Psychology, Stanford University, Stanford, CA SUSAN C. SCRIMSHAW, President, Simmons College, Boston, MA WILLIAM J. SPENCER, Chairman Emeritus, SEMATECH, Austin, TX LYDIA THOMAS (Ex officio), Retired, Mitretek Systems, Inc., Falls Church, VA CHARLES M. VEST (Ex officio), President, National Academy of Engineering, Washington, DC NANCY S. WEXLER, Higgins Professor of Neuropsychology, Columbia University, New York, NY MARY LOU ZOBACK, Vice President Earthquake Risk Applications, Risk Management Solutions, Inc., Newark, CA vi

Staff RICHARD BISSELL, Executive Director DEBORAH STINE, Associate Director (up to August 2007) MARION RAMSEY, Administrative Coordinator NEERAJ P. GORKHALY, Senior Program Assistant vii

BOARD ON ENVIRONMENTAL STUDIES AND TOXICOLOGY Members JONATHAN M. SAMET (Chair), Johns Hopkins University, Baltimore, MD RAMON ALVAREZ, Environmental Defense Fund, Austin, TX JOHN M. BALBUS, Environmental Defense Fund, Washington, DC DALLAS BURTRAW, Resources for the Future, Washington, DC JAMES S. BUS, Dow Chemical Company, Midland, MI RUTH DEFRIES, University of Maryland, College Park COSTEL D. DENSON, University of Delaware, Newark E. DONALD ELLIOTT, Willkie Farr & Gallagher LLP, Washington, DC MARY R. ENGLISH, University of Tennessee, Knoxville J. PAUL GILMAN, Oak Ridge Center for Advanced Studies, Oak Ridge, TN SHERRI W. GOODMAN, Center for Naval Analyses, Alexandria, VA JUDITH A. GRAHAM (Retired), Pittsboro, NC WILLIAM P. HORN, Birch, Horton, Bittner and Cherot, Washington, DC WILLIAM M. LEWIS, JR., University of Colorado, Boulder JUDITH L. MEYER, University of Georgia, Athens DENNIS D. MURPHY, University of Nevada, Reno PATRICK Y. O’BRIEN, ChevronTexaco Energy Technology Company, Richmond, CA DOROTHY E. PATTON, (Retired) U.S. Environmental Protection Agency, Chicago, IL DANNY D. REIBLE, University of Texas, Austin JOSEPH V. RODRICKS, ENVIRON International Corporation, Arlington, VA ARMISTEAD G. RUSSELL, Georgia Institute of Technology, Atlanta ROBERT F. SAWYER, University of California, Berkeley KIMBERLY M. THOMPSON, Massachusetts Institute of Technology, Cambridge MONICA G. TURNER, University of Wisconsin, Madison MARK J. UTELL, University of Rochester Medical Center, Rochester, NY CHRIS G. WHIPPLE, ENVIRON International Corporation, Emeryville, CA LAUREN ZEISE, California Environmental Protection Agency, Oakland Senior Staff JAMES J. REISA, Director DAVID J. POLICANSKY, Scholar RAYMOND A. WASSEL, Senior Program Officer for Environmental Studies EILEEN N. ABT, Senior Program Officer for Risk Analysis SUSAN N.J. MARTEL, Senior Program Officer for Toxicology KULBIR BAKSHI, Senior Program Officer ELLEN K. MANTUS, Senior Program Officer RUTH E. CROSSGROVE, Senior Editor viii

OTHER REPORTS OF THE COMMITTEE ON SCIENCE, ENGINEERING, AND PUBLIC POLICY Advanced Research Instrumentation and Facilities (2006) Beyond Bias and Barriers: Fulfilling the Potential of Women in Academic Science and Engineering (2006) Rising Above the Gathering Storm: Energizing and Employing America for a Brighter Economic Future (2005) Policy Implications of International Graduate Students and Postdoctoral Scholars in the United States (2005) Setting Priorities for NSF-sponsored Large Research Facility Projects (2004) Facilitating Interdisciplinary Research (2004) Science and Technology in the National Interest: Ensuring the Best Presidential and Federal Advisory Committee and Technology Appointments (2004) Electronic Scientific, Technical and Medical Journal Publishing and ITS Implications (2004) Observations on the President’s Fiscal Year 2003 Federal Science and Technology Budget (2003) Implementing the Government Performance and Results Act for Research: A Status Report (2003) Scientific and Medical Aspects of Human Reproductive Cloning (2002) Observations on the President’s Fiscal Year 2002 Federal Science and Technology Budget (2002) Experiments in International Benchmarking of U.S. Research Fields (2000) Observations on the President’s Fiscal Year 2001 Federal Science and Technology Budget (2000) Enhancing the Postdoctoral Experience for Scientists and Engineers: A Guide for Postdoctoral Scholars, Advisors, Institutions, Funding Organizations, and Disciplinary Societies (2000) Science and Technology in the National Interest: The Presidential Appointment Process (2000) Evaluating Federal Research Programs: Research and the Government Performance and Results Act (1999) Capitalizing on Investments in Science and Technology (1999) Observations on the President’s Fiscal Year 2000 Federal Science and Technology Budget (1999) Observations on the President’s Fiscal Year 1999 Federal Science and Technology Budget (1998) Adviser, Teacher, Role Model, Friend: On Being a Mentor to Students in Science and Engineering (1997) Proceedings of the National Convocation on Science and Engineering Doctoral Education (1996) Careers in Science and Engineering: A Student Planning Guide to Grad School and Beyond (1996) ix

An Assessment of the National Science Foundation’s Science and Technology Centers Program (1996) Allocating Federal Funds for Science and Technology (1995) Reshaping the Graduate Education of Scientists and Engineers (1995) On Being a Scientist: Responsible Conduct in Research (1995) Major Award Decisionmaking at the National Science Foundation (1994) Science, Technology, and the Federal Government: National Goals for a New Era (1993) Responsible Science Volume 2: Background Papers and Resource Documents (1993) Responsible Science Volume 1: Ensuring the Integrity of the Research Process (1992) Policy Implications of Greenhouse Warming: Mitigation, Adaptation, and the Science Base (1991) Copies of these reports may be ordered from the National Academies Press (800) 624-6242 or (202) 334-3313 www.nap.edu x

OTHER REPORTS OF THE BOARD ON ENVIRONMENTAL STUDIES AND TOXICOLOGY Respiratory Diseases Research at NIOSH (2008) Hydrology, Ecology, and Fishes of the Klamath River Basin (2008) Estimating Mortality Risk Reduction and Economic Benefits from Controlling Ozone Air Pollution (2008) Applications of Toxicogenomic Technologies to Predictive Toxicology and Risk Assessment (2007) Models in Environmental Regulatory Decision Making (2007) Toxicity Testing in the Twenty-first Century: A Vision and a Strategy (2007) Sediment Dredging at Superfund Megasites: Assessing the Effectiveness (2007) Environmental Impacts of Wind-Energy Projects (2007) Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget (2007) Assessing the Human Health Risks of Trichloroethylene: Key Scientific Issues (2006) New Source Review for Stationary Sources of Air Pollution (2006) Human Biomonitoring for Environmental Chemicals (2006) Health Risks from Dioxin and Related Compounds: Evaluation of the EPA Re- assessment (2006) Fluoride in Drinking Water: A Scientific Review of EPA’s Standards (2006) State and Federal Standards for Mobile-Source Emissions (2006) Superfund and Mining Megasites—Lessons from the Coeur d’Alene River Ba- sin (2005) Health Implications of Perchlorate Ingestion (2005) Air Quality Management in the United States (2004) Endangered and Threatened Species of the Platte River (2004) Atlantic Salmon in Maine (2004) Endangered and Threatened Fishes in the Klamath River Basin (2004) Cumulative Environmental Effects of Alaska North Slope Oil and Gas Devel- opment (2003) Estimating the Public Health Benefits of Proposed Air Pollution Regulations (2002) Biosolids Applied to Land: Advancing Standards and Practices (2002) The Airliner Cabin Environment and Health of Passengers and Crew (2002) Arsenic in Drinking Water: 2001 Update (2001) Evaluating Vehicle Emissions Inspection and Maintenance Programs (2001) Compensating for Wetland Losses Under the Clean Water Act (2001) A Risk-Management Strategy for PCB-Contaminated Sediments (2001) Acute Exposure Guideline Levels for Selected Airborne Chemicals (six volumes, 2000-2008) Toxicological Effects of Methylmercury (2000) Strengthening Science at the U.S. Environmental Protection Agency (2000) Scientific Frontiers in Developmental Toxicology and Risk Assessment (2000) xi

Ecological Indicators for the Nation (2000) Waste Incineration and Public Health (2000) Hormonally Active Agents in the Environment (1999) Research Priorities for Airborne Particulate Matter (four volumes, 1998-2004) The National Research Council’s Committee on Toxicology: The First 50 Years (1997) Carcinogens and Anticarcinogens in the Human Diet (1996) Upstream: Salmon and Society in the Pacific Northwest (1996) Science and the Endangered Species Act (1995) Wetlands: Characteristics and Boundaries (1995) Biologic Markers (five volumes, 1989-1995) Science and Judgment in Risk Assessment (1994) Pesticides in the Diets of Infants and Children (1993) Dolphins and the Tuna Industry (1992) Science and the National Parks (1992) Human Exposure Assessment for Airborne Pollutants (1991) Rethinking the Ozone Problem in Urban and Regional Air Pollution (1991) Decline of the Sea Turtles (1990) Copies of these reports may be ordered from the National Academies Press (800) 624-6242 or (202) 334-3313 www.nap.edu xii

Preface In an effort to ensure the wise use of taxpayers’ money, the federal gov- ernment has undertaken major initiatives to evaluate the performance and results of federally funded programs, including research and development (R&D) pro- grams. The initiatives include the Government Performance and Results Act of 1993 and the Office of Management and Budget (OMB) Program Assessment Rating Tool (PART), developed in 2002. The latter was designed in the context of “performance budgeting” and “performance measurement” and focused on evaluating the efficiency of programs. Evaluation of R&D programs has proved challenging for federal agencies, including the Environmental Protection Agency (EPA), a mission-oriented agency with a substantial research component. All agencies have experienced difficulties in complying with the PART requirements to measure the efficiency of their research, to use outcome-based metrics in doing so, and to achieve and quantitate annual efficiency improvements. In 2006, EPA asked the National Academies for assistance in developing better assessment tools to comply with PART, with emphasis on efficiency. The Academies’ Committee on Science, Engineering, and Public Policy (COSEPUP) and the National Research Council Board on Environmental Studies and Toxi- cology (BEST) convened the Committee on Evaluating the Efficiency of Re- search and Development Programs at the U.S. Environmental Protection Agency. In this report, the committee addresses the efficiency measures now used for federally funded R&D programs and evaluates whether they are sufficient and are based on outcomes, not only inputs and outputs. The committee provides principles that it hopes will guide the development of efficiency measures for federally funded R&D programs and makes recommendations about efficiency measures for EPA’s basic and applied R&D programs and about OMB’s review process. The committee gratefully acknowledges the following for making presen- tations to the committee: Diana Espinosa, Daren Wong, Brian Kleinman, and Kevin Neyland, of OMB; Marcus Peacock, Hugh Tilson, Phillip Juengst, Lori Kowalski, of EPA; Bill Valdez, Darrell Beschen, and Brian Card, of the De- partment of Energy; Pat Tsuchitani, of the National Science Foundation; Deb- xiii

xiv Preface orah Duran, of the National Institutes of Health; Julie Pollitt, of the National Aeronautics and Space Administration; Raymond Sinclair, of the National Insti- tute for Occupational Safety and Health; George Daston, of the Procter & Gam- ble Company; Joseph Kenney and Bernice Rogowitz, of IBM Global Business Services; James Bus, of Dow Chemical Company; and Patrick Atkins, retired from Alcoa. The committee is grateful for the assistance of the National Research Council staff in preparing this report. Staff members who contributed to the ef- fort are Richard Bissell, project director from August 2007 and director of COSEPUP; Deborah Stine, project director (up to August 2007); Eileen Abt, senior program officer; James Reisa, director of BEST; Alan Anderson, consult- ant science writer; Jennifer Saunders, associate program officer; Rae Benedict, Mirzayan Science & Technology Policy Fellow; Norman Grossblatt, senior edi- tor; Mirsada Karalic-Loncarevic, manager of the Toxicology Information Cen- ter; and Neeraj Gorkhaly and Morgan Motto, senior project assistants. I especially thank my colleagues on the committee for their engagements in the contentious issues underlying our charge and for what we believe are con- structive principles and feasible recommendations that have emerged from our deliberations and the iterative development of this report. Gilbert S. Omenn, Chair Committee on Evaluating the Efficiency of Research and Development Programs at the U.S. Environmental Protection Agency

Acknowledgment of Review Participants This report has been reviewed in draft form by persons chosen for their diverse perspectives and technical expertise in accordance with procedures ap- proved by the National Research Council’s Report Review Committee. The pur- pose of the independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards of objectivity, evidence, and responsiveness to the study charge. The review comments and draft manu- script remain confidential to protect the integrity of the deliberative process. We thank the following for their review of the report: Irwin Feller, Pennsylvania State University; Haren S. Gandhi, Ford Motor Company; Bernard D. Goldstein, University of Pittsburgh; Victoria F. Haynes, Research Triangle Institute; Mar- tha A. Krebs, California Energy Commission; James H. Johnson, Jr., Howard University; Genevieve M. Matanoski, Johns Hopkins University; Shelley H. Metzenbaum, University of Maryland School of Public Policy; and David Trin- kle, University of California, Berkeley. Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of the report was overseen by John F. Ahearne, Sigma Xi, The Sci- entific Research Society, and Alan Schriesheim, Argonne National Laboratory. Appointed by the National Research Council, they were responsible for making certain that an independent examination of the report was carried out in accor- dance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the author committee and the institution. xv

Contents SUMMARY ........................................................................................................ 3 1 INTRODUCTION: THE GOVERNMENT PERFORMANCE AND RESULTS ACT, THE PROGRAM ASSESSMENT RATING TOOL, AND THE ENVIRONMENTAL PROTECTION AGENCY ............... 11 Inherent Difficulties in Evaluating Research, 11 Research Terms at the Environmental Protection Agency, 12 Evaluating Research under the Government Performance and Results Act, 13 The Rationale and Function of the Program Assessment Rating Tool, 14 The Application of the Program Assessment Rating Tool to Research, 14 The Organization and Performance of Research and Development at the Environmental Protection Agency, 16 Uses of Results of Environmental Protection Agency Research, 18 Summary, 18 References, 19 2 EFFICIENCY METRICS USED BY THE ENVIRONMENTAL PROTECTION AGENCY AND OTHER FEDERAL RESEARCH AND DEVELOPMENT PROGRAMS ................................................... 21 Evaluating Research and Development, 22 The Program Assessment Rating Tool and Efficiency, 22 Applying Efficiency to Inputs, Outputs, and Outcomes, 22 The Program Assessment Rating Tool Grading System, 23 The Use of “Expert Review” at the Environmental Protection Agency, 24 Emerging Issues, 27 Metrics Proposed by the Environmental Protection Agency, 27 Metrics That Did Not Pass the Program Assessment Rating Tool Process, 29 The Consequences of a “No” Answer to a Program Assessment Rating Tool Question, 30 Evaluation Mechanisms Used by Other Agencies, 31 Methods Used by Industry, 35 Summary, 35 References, 36 xvii

xviii Contents 3 ARE THE EFFICIENCY METRICS USED BY FEDERAL RESEARCH AND DEVELOPMENT PROGRAMS SUFFICIENT AND OUTCOME-BASED? ............................................ 38 Attempting to Evaluate Efficiency in Terms of Ultimate Outcomes, 38 Placing “Research Efficiency” in Perspective, 40 Process Efficiency and Investment Efficiency, 41 What Are “Sufficient” Metrics of Process Efficiency? 42 A Critique of the Efficiency Metrics Used by Federal Research Programs, 44 Factors that Reduce the Efficiency of Research, 47 Evaluating Research Efficiency in Industry, 48 The Shortcomings of Retrospective Review, 49 Summary and Recommendations, 49 References, 50 4 A MODEL FOR EVALUATING RESEARCH AND DEVELOPMENT PROGRAMS ............................................................. 52 Summary, 56 References, 56 5 FINDINGS, PRINCIPLES, AND RECOMMENDATIONS................. 58 Findings, 59 Principles, 60 Recommendations, 65 Additional Recommendation for the Office of Management and Budget, 67 References, 68 APPENDIX A: BIOGRAPHIC INFORMATION ON THE COMMITTEE ON EVALUATING THE EFFICIENCY OF RESEARCH AND DEVELOPMENT PROGRAMS AT THE U.S. ENVIRONMENTAL PROTECTION AGENCY ................................. 69 APPENDIX B: EVALUATING THE EFFICIENCY OF RESEARCH AND DEVELOPMENT PROGRAMS AT THE U.S. ENVIRONMENTAL PROTECTION AGENCY: WORKSHOP SUMMARY ............................................................................ 75 APPENDIX C: PROGRAM ASSESSMENT RATING TOOL (PART) QUESTIONS ..................................................................................... 86 APPENDIX D: THE ENVIRONEMENTAL PROTECTION AGENCY’S STRATEGIC AND MULTI-YEAR PLANNING PROCESS................................................................................... 90

Contents xix APPENDIX E: AGENCY AND INDUSTRY EFFICIENCY MEASURES............................................................................ 95 APPENDIX F: DRAFT BOARD OF SCIENTIFIC COUNSELORS HANDBOOK FOR SUBCOMMITTEE CHAIRS: DRAFT PROPOSED CHARGE QUESTIONS FOR BOSC REVIEWS ................ 109 APPENDIX G: OMB’S RESEARCH AND DEVELOPMENT PROGRAM INVESTMENT CRITERIA .................................................... 113 APPENDIX H: CHARGE TO THE BOSC SUBCOMMITTEE ON SAFE PESTICIDES/SAFE PRODUCTS RESEARCH...................... 119 APPENDIX I: PART GUIDANCE ON EFFICIENCY MEASURES ...... 128 BOXES, FIGURES, AND TABLES BOXES 1-1 Distribution of PART Scores, 15 1-2 How PART Questions Are Scored, 16 2-1 BOSC: An Example of Expert Review, 28 I-1 Measures and PARTWeb, 129 I-2 Question Linkages, 131 FIGURES 4-1 EPA research presented as a logical model, 54 TABLES E-1 Agency and Industry Efficiency Measures, 96

Next: Evaluating Research Efficiency in the U.S. Environmental Protection Agency »
Evaluating Research Efficiency in the U.S. Environmental Protection Agency Get This Book
×
Buy Paperback | $48.00 Buy Ebook | $38.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

A new book from the National Research Council recommends changes in how the federal government evaluates the efficiency of research at EPA and other agencies. Assessing efficiency should be considered only one part of gauging a program's quality, relevance, and effectiveness. The efficiency of research processes and that of investments should be evaluated using different approaches. Investment efficiency should examine whether an agency's R&D portfolio, including the budget, is relevant, of high quality, matches the agency's strategic plan. These evaluations require panels of experts. In contrast, process efficiency should focus on "inputs" (the people, funds, and facilities dedicated to research) and "outputs" (the services, grants, publications, monitoring, and new techniques produced by research), as well as their timelines and should be evaluated using quantitative measures. The committee recommends that the efficiency of EPA's research programs be evaluated according to the same standards used at other agencies. To ensure this, OMB should train and oversee its budget examiners so that the PART questionnaire is implemented consistently and equitably across agencies.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!