Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page R1
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
Committee on Evaluating the Efficiency of Research and Development Programs at the U.S. Environmental Protection Agency
Committee on Science, Engineering, and Public Policy
Policy and Global Affairs
Board on Environmental Studies and Toxicology
Division on Earth and Life Studies
NATIONAL RESEARCH COUNCIL OF THE NATIONAL ACADEMIES
THE NATIONAL ACADEMIES PRESS
Washington, D.C.
www.nap.edu
OCR for page R2
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
THE NATIONAL ACADEMIES PRESS
500 Fifth Street, NW Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This project was supported by Contract 68-C-03-081 between the National Academy of Sciences and the U.S. Environmental Protection Agency. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the authors and do not necessarily reflect the view of the organizations or agencies that provided support for this project.
International Standard Book Number-13 978-0-309-11684-8
International Standard Book Number-10 0-309-11684-8
Additional copies of this report are available from
The National Academies Press
500 Fifth Street, NW Box 285 Washington, DC 20055 800-624-6242 202-334-3313 (in the Washington metropolitan area) http://www.nap.edu
Copyright 2008 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America.
OCR for page R3
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council.
www.national-academies.org
OCR for page R4
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
This page intentionally left blank.
OCR for page R5
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
COMMITTEE ON EVALUATING THE EFFICIENCY OF RESEARCH AND DEVELOPMENT PROGRAMS AT THE U.S. ENVIRONMENTAL PROTECTION AGENCY
Members
GILBERT S. OMENN (Chair),
University of Michigan, Ann Arbor
GEORGE V. ALEXEEFF,
California Environmental Protection Agency, Oakland
RADFORD BYERLY, JR.,
University of Colorado, Boulder
EDWIN H. CLARK II,
Earth Policy Institute, Washington, DC
SUSAN E. COZZENS,
Georgia Institute of Technology, Atlanta
LINDA J. FISHER,
E. I. du Pont de Nemours and Company, Wilmington, DE
J. PAUL GILMAN,
Oak Ridge Center for Advanced Studies, Oak Ridge, TN
T.J. GLAUTHIER,
TJG Energy Associates, LLC, Moss Beach, CA
CAROL J. HENRY, Independent Consultant,
Bethesda, MD
ROBERT J. HUGGETT,
College of William and Mary, Seaford, VA
SALLY KATZEN,
George Mason University School of Law, Fairfax, VA
TERRY F. YOUNG,
Environmental Defense, Oakland, CA
Staff
RICHARD BISSELL, Executive Director,
Committee on Science, Engineering, and Public Policy
DEBORAH STINE, Associate Director,
Committee on Science, Engineering, and Public Policy (up to August 2007)
EILEEN ABT, Senior Program Officer
ALAN ANDERSON, Consultant Writer
NORMAN GROSSBLATT, Senior Editor
JENNIFER SAUNDERS, Associate Program Officer
RAE BENEDICT, Mirzayan Science & Technology Policy Fellow
MIRSADA KARALIC-LONCAREVIC, Manager,
Toxicology Information Center
NEERAJ P. GORKHALY, Senior Program Assistant
MORGAN R. MOTTO, Senior Program Assistant
Sponsor
U.S. ENVIRONMENTAL PROTECTION AGENCY
OCR for page R6
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
COMMITTEE ON SCIENCE, ENGINEERING, AND PUBLIC POLICY
Members
GEORGE WHITESIDES (Chair), Woodford L. and Ann A. Flowers University Professor,
Harvard University, Boston, MA
CLAUDE R. CANIZARES, Vice President for Research,
Massachusetts Institute of Technology, Cambridge
RALPH J. CICERONE (Ex officio), President,
National Academy of Sciences, Washington, DC
EDWARD F. CRAWLEY, Executive Director,
CMI, and
Professor,
Massachusetts Institute of Technology, Cambridge
RUTH A. DAVID, President and Chief Executive Officer,
Analytic Services, Inc., Arlington, VA
HAILE T. DEBAS, Executive Director,
UCSF Global Health Sciences,
Maurice Galante Distinguished Professor of Surgery,
San Francisco, CA
HARVEY FINEBERG (Ex officio), President,
Institute of Medicine, Washington, DC
JACQUES S. GANSLER, Vice President for Research,
University of Maryland, College Park
ELSA M. GARMIRE, Professor,
Dartmouth College, Hanover, NH
M. R. C. GREENWOOD (Ex officio), Professor of Nutrition and Internal Medicine,
University of California, Davis
W. CARL LINEBERGER, Professor of Chemistry,
University of Colorado, Boulder
C. DAN MOTE, JR. (Ex officio), President and Glenn Martin Institute Professor of Engineering,
University of Maryland, College Park
ROBERT M. NEREM, Parker H. Petit Professor and Director,
Institute for Bioengineering and Bioscience, Georgia Institute of Technology, Atlanta
LAWRENCE T. PAPAY, Retired, Sector Vice President for Integrated Solutions,
Science Applications International Corporation, La Jolla, CA
ANNE C. PETERSEN, Professor of Psychology,
Stanford University, Stanford, CA
SUSAN C. SCRIMSHAW, President,
Simmons College, Boston, MA
WILLIAM J. SPENCER, Chairman Emeritus,
SEMATECH, Austin, TX
LYDIA THOMAS (Ex officio),
Retired, Mitretek Systems, Inc., Falls Church, VA
CHARLES M. VEST (Ex officio), President,
National Academy of Engineering, Washington, DC
NANCY S. WEXLER, Higgins Professor of Neuropsychology,
Columbia University, New York, NY
MARY LOU ZOBACK, Vice President Earthquake Risk Applications,
Risk Management Solutions, Inc., Newark, CA
OCR for page R7
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
Staff
RICHARD BISSELL, Executive Director
DEBORAH STINE, Associate Director (up to August 2007)
MARION RAMSEY, Administrative Coordinator
NEERAJ P. GORKHALY, Senior Program Assistant
OCR for page R8
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
BOARD ON ENVIRONMENTAL STUDIES AND TOXICOLOGY
Members
JONATHAN M. SAMET (Chair),
Johns Hopkins University, Baltimore, MD
RAMÓN ALVAREZ,
Environmental Defense Fund, Austin, TX
JOHN M. BALBUS,
Environmental Defense Fund, Washington, DC
DALLAS BURTRAW,
Resources for the Future, Washington, DC
JAMES S. BUS,
Dow Chemical Company, Midland, MI
RUTH DEFRIES,
University of Maryland, College Park
COSTEL D. DENSON,
University of Delaware, Newark
E. DONALD ELLIOTT,
Willkie Farr & Gallagher LLP, Washington, DC
MARY R. ENGLISH,
University of Tennessee, Knoxville
J. PAUL GILMAN,
Oak Ridge Center for Advanced Studies, Oak Ridge, TN
SHERRI W. GOODMAN,
Center for Naval Analyses, Alexandria, VA
JUDITH A. GRAHAM
(Retired), Pittsboro, NC
WILLIAM P. HORN,
Birch, Horton, Bittner and Cherot, Washington, DC
WILLIAM M. LEWIS, JR.,
University of Colorado, Boulder
JUDITH L. MEYER,
University of Georgia, Athens
DENNIS D. MURPHY,
University of Nevada, Reno
PATRICK Y. O’BRIEN,
ChevronTexaco Energy Technology Company, Richmond, CA
DOROTHY E. PATTON,
(Retired) U.S. Environmental Protection Agency, Chicago, IL
DANNY D. REIBLE,
University of Texas, Austin
JOSEPH V. RODRICKS,
ENVIRON International Corporation, Arlington, VA
ARMISTEAD G. RUSSELL,
Georgia Institute of Technology, Atlanta
ROBERT F. SAWYER,
University of California, Berkeley
KIMBERLY M. THOMPSON,
Massachusetts Institute of Technology, Cambridge
MONICA G. TURNER,
University of Wisconsin, Madison
MARK J. UTELL,
University of Rochester Medical Center, Rochester, NY
CHRIS G. WHIPPLE,
ENVIRON International Corporation, Emeryville, CA
LAUREN ZEISE,
California Environmental Protection Agency, Oakland
Senior Staff
JAMES J. REISA, Director
DAVID J. POLICANSKY, Scholar
RAYMOND A. WASSEL, Senior Program Officer for Environmental Studies
EILEEN N. ABT, Senior Program Officer for Risk Analysis
SUSAN N.J. MARTEL, Senior Program Officer for Toxicology
KULBIR BAKSHI, Senior Program Officer
ELLEN K. MANTUS, Senior Program Officer
RUTH E. CROSSGROVE, Senior Editor
OCR for page R9
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
OTHER REPORTS OF THE COMMITTEE ON SCIENCE, ENGINEERING, AND PUBLIC POLICY
Advanced Research Instrumentation and Facilities (2006)
Beyond Bias and Barriers: Fulfilling the Potential of Women in Academic Science and Engineering (2006)
Rising Above the Gathering Storm: Energizing and Employing America for a Brighter Economic Future (2005)
Policy Implications of International Graduate Students and Postdoctoral Scholars in the United States (2005)
Setting Priorities for NSF-sponsored Large Research Facility Projects (2004)
Facilitating Interdisciplinary Research (2004)
Science and Technology in the National Interest: Ensuring the Best Presidential and Federal Advisory Committee and Technology Appointments (2004)
Electronic Scientific, Technical and Medical Journal Publishing and ITS Implications (2004)
Observations on the President’s Fiscal Year 2003 Federal Science and Technology Budget (2003)
Implementing the Government Performance and Results Act for Research: A Status Report (2003)
Scientific and Medical Aspects of Human Reproductive Cloning (2002)
Observations on the President’s Fiscal Year 2002 Federal Science and Technology Budget (2002)
Experiments in International Benchmarking of U.S. Research Fields (2000)
Observations on the President’s Fiscal Year 2001 Federal Science and Technology Budget (2000)
Enhancing the Postdoctoral Experience for Scientists and Engineers: A Guide for Postdoctoral Scholars, Advisors, Institutions, Funding Organizations, and Disciplinary Societies (2000)
Science and Technology in the National Interest: The Presidential Appointment Process (2000)
Evaluating Federal Research Programs: Research and the Government Performance and Results Act (1999)
Capitalizing on Investments in Science and Technology (1999)
Observations on the President’s Fiscal Year 2000 Federal Science and Technology Budget (1999)
Observations on the President’s Fiscal Year 1999 Federal Science and Technology Budget (1998)
Adviser, Teacher, Role Model, Friend: On Being a Mentor to Students in Science and Engineering (1997)
Proceedings of the National Convocation on Science and Engineering Doctoral Education (1996)
Careers in Science and Engineering: A Student Planning Guide to Grad School and Beyond (1996)
OCR for page R10
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
An Assessment of the National Science Foundation’s Science and Technology Centers Program (1996)
Allocating Federal Funds for Science and Technology (1995)
Reshaping the Graduate Education of Scientists and Engineers (1995)
On Being a Scientist: Responsible Conduct in Research (1995)
Major Award Decisionmaking at the National Science Foundation (1994)
Science, Technology, and the Federal Government: National Goals for a New Era (1993)
Responsible Science Volume 2: Background Papers and Resource Documents (1993)
Responsible Science Volume 1: Ensuring the Integrity of the Research Process (1992)
Policy Implications of Greenhouse Warming: Mitigation, Adaptation, and the Science Base (1991)
Copies of these reports may be ordered from the National Academies Press
(800) 624-6242 or (202) 334-3313
www.nap.edu
OCR for page R11
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
OTHER REPORTS OF THE BOARD ON ENVIRONMENTAL STUDIES AND TOXICOLOGY
Respiratory Diseases Research at NIOSH (2008)
Hydrology, Ecology, and Fishes of the Klamath River Basin (2008)
Estimating Mortality Risk Reduction and Economic Benefits from Controlling Ozone Air Pollution (2008)
Applications of Toxicogenomic Technologies to Predictive Toxicology and Risk Assessment (2007)
Models in Environmental Regulatory Decision Making (2007)
Toxicity Testing in the Twenty-first Century: A Vision and a Strategy (2007)
Sediment Dredging at Superfund Megasites: Assessing the Effectiveness (2007)
Environmental Impacts of Wind-Energy Projects (2007)
Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget (2007)
Assessing the Human Health Risks of Trichloroethylene: Key Scientific Issues (2006)
New Source Review for Stationary Sources of Air Pollution (2006)
Human Biomonitoring for Environmental Chemicals (2006)
Health Risks from Dioxin and Related Compounds: Evaluation of the EPA Reassessment (2006)
Fluoride in Drinking Water: A Scientific Review of EPA’s Standards (2006)
State and Federal Standards for Mobile-Source Emissions (2006)
Superfund and Mining Megasites—Lessons from the Coeur d’Alene River Basin (2005)
Health Implications of Perchlorate Ingestion (2005)
Air Quality Management in the United States (2004)
Endangered and Threatened Species of the Platte River (2004)
Atlantic Salmon in Maine (2004)
Endangered and Threatened Fishes in the Klamath River Basin (2004)
Cumulative Environmental Effects of Alaska North Slope Oil and Gas Development (2003)
Estimating the Public Health Benefits of Proposed Air Pollution Regulations (2002)
Biosolids Applied to Land: Advancing Standards and Practices (2002)
The Airliner Cabin Environment and Health of Passengers and Crew (2002)
Arsenic in Drinking Water: 2001 Update (2001)
Evaluating Vehicle Emissions Inspection and Maintenance Programs (2001)
Compensating for Wetland Losses Under the Clean Water Act (2001)
A Risk-Management Strategy for PCB-Contaminated Sediments (2001)
Acute Exposure Guideline Levels for Selected Airborne Chemicals (six volumes, 2000-2008)
Toxicological Effects of Methylmercury (2000)
Strengthening Science at the U.S. Environmental Protection Agency (2000)
Scientific Frontiers in Developmental Toxicology and Risk Assessment (2000)
OCR for page R12
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
Ecological Indicators for the Nation (2000)
Waste Incineration and Public Health (2000)
Hormonally Active Agents in the Environment (1999)
Research Priorities for Airborne Particulate Matter (four volumes, 1998-2004)
The National Research Council’s Committee on Toxicology: The First 50 Years (1997)
Carcinogens and Anticarcinogens in the Human Diet (1996)
Upstream: Salmon and Society in the Pacific Northwest (1996)
Science and the Endangered Species Act (1995)
Wetlands: Characteristics and Boundaries (1995)
Biologic Markers (five volumes, 1989-1995)
Science and Judgment in Risk Assessment (1994)
Pesticides in the Diets of Infants and Children (1993)
Dolphins and the Tuna Industry (1992)
Science and the National Parks (1992)
Human Exposure Assessment for Airborne Pollutants (1991)
Rethinking the Ozone Problem in Urban and Regional Air Pollution (1991)
Decline of the Sea Turtles (1990)
Copies of these reports may be ordered from the National Academies Press
(800) 624-6242 or (202) 334-3313
www.nap.edu
OCR for page R13
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
Preface
In an effort to ensure the wise use of taxpayers’ money, the federal government has undertaken major initiatives to evaluate the performance and results of federally funded programs, including research and development (R&D) programs. The initiatives include the Government Performance and Results Act of 1993 and the Office of Management and Budget (OMB) Program Assessment Rating Tool (PART), developed in 2002. The latter was designed in the context of “performance budgeting” and “performance measurement” and focused on evaluating the efficiency of programs.
Evaluation of R&D programs has proved challenging for federal agencies, including the Environmental Protection Agency (EPA), a mission-oriented agency with a substantial research component. All agencies have experienced difficulties in complying with the PART requirements to measure the efficiency of their research, to use outcome-based metrics in doing so, and to achieve and quantitate annual efficiency improvements.
In 2006, EPA asked the National Academies for assistance in developing better assessment tools to comply with PART, with emphasis on efficiency. The Academies’ Committee on Science, Engineering, and Public Policy (COSEPUP) and the National Research Council Board on Environmental Studies and Toxicology (BEST) convened the Committee on Evaluating the Efficiency of Research and Development Programs at the U.S. Environmental Protection Agency.
In this report, the committee addresses the efficiency measures now used for federally funded R&D programs and evaluates whether they are sufficient and are based on outcomes, not only inputs and outputs. The committee provides principles that it hopes will guide the development of efficiency measures for federally funded R&D programs and makes recommendations about efficiency measures for EPA’s basic and applied R&D programs and about OMB’s review process.
The committee gratefully acknowledges the following for making presentations to the committee: Diana Espinosa, Daren Wong, Brian Kleinman, and Kevin Neyland, of OMB; Marcus Peacock, Hugh Tilson, Phillip Juengst, Lori Kowalski, of EPA; Bill Valdez, Darrell Beschen, and Brian Card, of the Department of Energy; Pat Tsuchitani, of the National Science Foundation; Deb-
OCR for page R14
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
orah Duran, of the National Institutes of Health; Julie Pollitt, of the National Aeronautics and Space Administration; Raymond Sinclair, of the National Institute for Occupational Safety and Health; George Daston, of the Procter & Gamble Company; Joseph Kenney and Bernice Rogowitz, of IBM Global Business Services; James Bus, of Dow Chemical Company; and Patrick Atkins, retired from Alcoa.
The committee is grateful for the assistance of the National Research Council staff in preparing this report. Staff members who contributed to the effort are Richard Bissell, project director from August 2007 and director of COSEPUP; Deborah Stine, project director (up to August 2007); Eileen Abt, senior program officer; James Reisa, director of BEST; Alan Anderson, consultant science writer; Jennifer Saunders, associate program officer; Rae Benedict, Mirzayan Science & Technology Policy Fellow; Norman Grossblatt, senior editor; Mirsada Karalic-Loncarevic, manager of the Toxicology Information Center; and Neeraj Gorkhaly and Morgan Motto, senior project assistants.
I especially thank my colleagues on the committee for their engagements in the contentious issues underlying our charge and for what we believe are constructive principles and feasible recommendations that have emerged from our deliberations and the iterative development of this report.
Gilbert S. Omenn, Chair
Committee on Evaluating the Efficiency of Research and Development Programs at the U.S. Environmental Protection Agency
OCR for page R15
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
Acknowledgment of Review Participants
This report has been reviewed in draft form by persons chosen for their diverse perspectives and technical expertise in accordance with procedures approved by the National Research Council’s Report Review Committee. The purpose of the independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards of objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We thank the following for their review of the report: Irwin Feller, Pennsylvania State University; Haren S. Gandhi, Ford Motor Company; Bernard D. Goldstein, University of Pittsburgh; Victoria F. Haynes, Research Triangle Institute; Martha A. Krebs, California Energy Commission; James H. Johnson, Jr., Howard University; Genevieve M. Matanoski, Johns Hopkins University; Shelley H. Metzenbaum, University of Maryland School of Public Policy; and David Trinkle, University of California, Berkeley.
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of the report was overseen by John F. Ahearne, Sigma Xi, The Scientific Research Society, and Alan Schriesheim, Argonne National Laboratory. Appointed by the National Research Council, they were responsible for making certain that an independent examination of the report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the author committee and the institution.
OCR for page R16
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
This page intentionally left blank.
OCR for page R17
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
Contents
SUMMARY
3
1
INTRODUCTION: THE GOVERNMENT PERFORMANCE AND RESULTS ACT, THE PROGRAM ASSESSMENT RATING TOOL, AND THE ENVIRONMENTAL PROTECTION AGENCY
11
Inherent Difficulties in Evaluating Research,
11
Research Terms at the Environmental Protection Agency,
12
Evaluating Research under the Government Performance and Results Act,
13
The Rationale and Function of the Program Assessment Rating Tool,
14
The Application of the Program Assessment Rating Tool to Research,
14
The Organization and Performance of Research and Development at the Environmental Protection Agency,
16
Uses of Results of Environmental Protection Agency Research,
18
Summary,
18
References,
19
2
EFFICIENCY METRICS USED BY THE ENVIRONMENTAL PROTECTION AGENCY AND OTHER FEDERAL RESEARCH AND DEVELOPMENT PROGRAMS
21
Evaluating Research and Development,
22
The Program Assessment Rating Tool and Efficiency,
22
Applying Efficiency to Inputs, Outputs, and Outcomes,
22
The Program Assessment Rating Tool Grading System,
23
The Use of “Expert Review” at the Environmental Protection Agency,
24
Emerging Issues,
27
Metrics Proposed by the Environmental Protection Agency,
27
Metrics That Did Not Pass the Program Assessment Rating Tool Process,
29
The Consequences of a “No” Answer to a Program Assessment Rating Tool Question,
30
Evaluation Mechanisms Used by Other Agencies,
31
Methods Used by Industry,
35
Summary,
35
References,
36
OCR for page R18
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
3
ARE THE EFFICIENCY METRICS USED BY FEDERAL RESEARCH AND DEVELOPMENT PROGRAMS SUFFICIENT AND OUTCOME-BASED?
38
Attempting to Evaluate Efficiency in Terms of Ultimate Outcomes,
38
Placing “Research Efficiency” in Perspective,
40
Process Efficiency and Investment Efficiency,
41
What Are “Sufficient” Metrics of Process Efficiency?
42
A Critique of the Efficiency Metrics Used by Federal Research Programs,
44
Factors that Reduce the Efficiency of Research,
47
Evaluating Research Efficiency in Industry,
48
The Shortcomings of Retrospective Review,
49
Summary and Recommendations,
49
References,
50
4
A MODEL FOR EVALUATING RESEARCH AND DEVELOPMENT PROGRAMS
52
Summary,
56
References,
56
5
FINDINGS, PRINCIPLES, AND RECOMMENDATIONS
58
Findings,
59
Principles,
60
Recommendations,
65
Additional Recommendation for the Office of Management and Budget,
67
References,
68
APPENDIX A:
BIOGRAPHIC INFORMATION ON THE COMMITTEE ON EVALUATING THE EFFICIENCY OF RESEARCH AND DEVELOPMENT PROGRAMS AT THE U.S. ENVIRONMENTAL PROTECTION AGENCY
69
APPENDIX B:
EVALUATING THE EFFICIENCY OF RESEARCH AND DEVELOPMENT PROGRAMS AT THE U.S. ENVIRONMENTAL PROTECTION AGENCY: WORKSHOP SUMMARY
75
APPENDIX C:
PROGRAM ASSESSMENT RATING TOOL (PART) QUESTIONS
86
APPENDIX D:
THE ENVIRONEMENTAL PROTECTION AGENCY’S STRATEGIC AND MULTI-YEAR PLANNING PROCESS
90
OCR for page R19
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
APPENDIX E:
AGENCY AND INDUSTRY EFFICIENCY MEASURES
95
APPENDIX F:
DRAFT BOARD OF SCIENTIFIC COUNSELORS HANDBOOK FOR SUBCOMMITTEE CHAIRS: DRAFT PROPOSED CHARGE QUESTIONS FOR BOSC REVIEWS
109
APPENDIX G:
OMB’S RESEARCH AND DEVELOPMENT PROGRAM INVESTMENT CRITERIA
113
APPENDIX H:
CHARGE TO THE BOSC SUBCOMMITTEE ON SAFE PESTICIDES/SAFE PRODUCTS RESEARCH
119
APPENDIX I:
PART GUIDANCE ON EFFICIENCY MEASURES
128
BOXES, FIGURES, AND TABLES
BOXES
1-1
Distribution of PART Scores,
15
1-2
How PART Questions Are Scored,
16
2-1
BOSC: An Example of Expert Review,
28
I-1
Measures and PARTWeb,
129
I-2
Question Linkages,
131
FIGURES
4-1
EPA research presented as a logical model,
54
TABLES
E-1
Agency and Industry Efficiency Measures,
96
OCR for page R20
Evaluating Research Efficiency in the U.S. Environmental Protection Agency
This page intentionally left blank.
