Evaluating Research Efficiency in the U.S. Environmental Protection Agency (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

1 Introduction: The Government Performance and Results Act, the Program Assessment Rating Tool, and the Environmental Protection Agency Federal administrations have long attempted to improve alignment of the spending decisions of the U.S. federal government with the expected results of the decisions (OMB 2004). In the 1990s, Congress and the executive branch devised a statutory and management framework to strengthen the performance and accountability of all federal agencies; the Government Performance and Results Act (GPRA) of 1993 was its centerpiece. GPRA focused agency and oversight attention on the performance and re- sults of government activities by requiring federal agencies to measure and re- port annually on the results of their activities. For the first time, each of about 1,000 federal programs was required to explicitly identify metrics and goals for judging its performance and to collect information each year to determine whether it was meeting the goals (OMB 2004). GPRA required agencies to de- velop a strategic plan that set goals and objectives for at least a 5-year period, an annual performance plan that translated the goals into annual targets, and an annual performance report that demonstrated whether the targets were met (NRC 1999). A key objective of GPRA was to create closer and clearer links between the process of allocating limited resources and the expected results to be achieved with them (Posner 2004). INHERENT DIFFICULTIES IN EVALUATING RESEARCH As agencies developed strategies to comply with GPRA, it became clear that the evaluation of science and technology research programs, especially those involving basic research, created challenges for both the agencies and 11 

12 Evaluating Research Efficiency in EPA oversight bodies. That, of course, is true for many other fields and practices. In the particular case of science, especially basic research, a fundamental challenge is that the course of research cannot be planned or known in advance; research entails continual feedback from observation and experimentation, which leads to new directions. As Donald Stokes has written, “research proceeds by making choices. Al- though the activities by which scientific research develops new information or knowledge are exceedingly varied, they always entail a sequence of decisions or choices.” They include the choice of a problem, construction of theories or mod- els, development of instruments or metrics, and design of experiments or obser- vations (Stokes 1997). Stokes wrote that the defining quality of basic research is that it seeks to widen the understanding of phenomena in a scientific field. In any search for new understanding, the researcher cannot know in advance what that understanding will be and therefore cannot know how long it will take, how much it will cost, and what instrumentation will be required; so the ability to evaluate progress against benchmarks is slight. Applied research is similar to basic research in that it has the same underlying process of inquiry, but it is of- ten distinct from basic research in emphasizing the extension of fundamental understanding to “some individual or group or societal need or use” (Stokes 1997). The intended outcomes of applied research, which include methods de- velopment and monitoring, are usually well known in advance. The committee believes that the terms basic research, applied research, and development describe overlapping and complementary activities. The proc- ess of research might be visualized as the development and flow of knowledge within and across categorical boundaries through collaboration, feedback loops, and fortuitous insights. Agencies support many levels of research to sustain a needed flow of knowledge, respond quickly to current demands, and prepare for future challenges. Attempts to evaluate research in terms of efficiency may founder because of the very nature of research. Thus, a negative or unexpected result of a scien- tific test can have value even if the time and other resources consumed by the test might be judged “inefficient” by some metrics. In addition, much of the work of researchers involves building on, integrating, and replicating previous results and this might also appear “inefficient.” RESEARCH TERMS AT THE ENVIRONMENTAL PROTECTION AGENCY The Environmental Protection Agency (EPA) uses a particular nomencla- ture to describe its research, including the terms core research and problem- driven research. Those terms were coined by a National Research Council committee that recommended “that EPA’s research program maintain a balance between problem-driven research, targeted at understanding and solving particu- lar identified environmental problems and reducing the uncertainties associated 

Introduction 13 with them, and core research, which aims to provide broader, more generic in- formation to help improve understanding relevant to environmental problems for the present and the future” (NRC 1997). The report added that the distinction at EPA between core and problem-driven research is not always clear.1 EVALUATING RESEARCH UNDER THE GOVERNMENT PERFORMANCE AND RESULTS ACT In the late 1990s, the National Academies was asked for advice on how to evaluate the research programs of federal agencies, and the Committee on Sci- ence, Engineering, and Public Policy (COSEPUP) undertook a study (NRC 1999) that began with a review of how federal research agencies were address- ing GPRA. That committee determined that • The useful outcomes of basic research cannot be measured directly on an annual basis because their timing and nature are inherently unpredictable. • Meaningful criteria do exist by which the performance of basic re- search can be evaluated while the research is in progress: quality, relevance, and, when appropriate, leadership, as measured in the context of international standards. • Such evaluations are best performed by “expert-review panels,” which, in addition to experts in the field under review, include experts in related fields who may be drawn from academe, industry, government, and other appropriate sectors. • Measurements based on those criteria can be reported regularly to as- sure the nation a good return on its investments in basic research. Two years later, when more information about agencies’ efforts to comply with GPRA was available, a panel appointed by COSEPUP reiterated and ex- panded on the original recommendations.2 The panel focused on the five agen- cies that provide the majority of federal funding for research3 and found that all had made good-faith efforts to comply with the requirements of GPRA. It also determined that some oversight bodies and agencies needed clearer procedures to validate agency evaluations and that compliance techniques, communication 1 The report also pointed out that “the terms were not the same as basic vs applied re- search, fundamental vs directed research, or short-term vs long-term research, which are typically used by other federal agencies and researchers.” 2 In addition, COSEPUP has extended experience since the time of GPRA in helping OMB to interpret the application of government-wide criteria to agency research pro- grams. Workshops were organized by COSEPUP for the OMB in 2001-2002 on the R&D Investment Criteria, and in 2004 on the PART. 3 The five agencies are the Department of Defense, the Department of Energy, the Na- tional Aeronautics and Space Administration, the Department of Health and Human Ser- vices (specifically, the National Institutes of Health), and the National Science Founda- tion. 

14 Evaluating Research Efficiency in EPA with oversight bodies, and timing requirements of agencies varied widely (NRC 2001). THE RATIONALE AND FUNCTION OF THE PROGRAM ASSESSMENT RATING TOOL Although GPRA increased the information on the results and performance of federal agencies, the Office of Management and Budget (OMB) wanted to improve the usefulness of the process and therefore developed the Program As- sessment Rating Tool (PART) in 2002. PART was designed for use throughout the federal government in the larger context of “performance budgeting” and “performance measurement.” Performance budgeting seeks to design budgeting procedures that optimize efficiency and effectiveness, including the most effec- tive mechanisms for allocating available resources and holding program manag- ers accountable for results. The first draft of PART was released for comment in May 2002 (OMB 2004), and OMB announced its intention to review each fed- eral program every 5 years and to complete the first cycle in 2007 (OMB 2004). PART itself is a questionnaire of at least 25 questions, whose number var- ies slightly with the type of program being evaluated.4 The questions are ar- ranged in four categories by which programs are assessed: purpose and design, strategic planning, management, and results and accountability. The answers to the questions result in a cumulative numeric score of 0-100 (100 is the best). Depending on that score, the program’s rating is “effective,” “moderately effec- tive,” “adequate,” or “ineffective.” Programs that have not developed acceptable performance metrics or sufficient performance data generally receive a rating of “Results not demonstrated.” Box 1-1 shows the distribution of PART scores for the 1,016 programs rated. PART constitutes an important step in program assessment. As one Gen- eral Accountability Official told Congress, “PART may mark a new chapter in performance-based budgeting by more successfully stimulating demand for this information—that is, using the performance information generated through GPRA’s planning and reporting processes to more directly feed into executive branch budgetary decisions” (Posner 2004). The process of PART implementation is still new, and agencies are still developing their compliance methods. Some 19% of federal programs are rated “Results not demonstrated” (OMB 2007a). THE APPLICATION OF THE PROGRAM ASSESSMENT RATING TOOL TO RESEARCH For evaluating research programs, the PART system adopted the criteria of quality and relevance suggested by National Academies reports. Those crite- 4 See Appendix C for the full list of PART questions. 

Introduction 15 ria are addressed in multiple PART questions, especially the first two sections, which explore program purpose and design and strategic planning. In lieu of the “leadership” criterion, OMB developed a new criterion of “performance.”5 This is described in terms of both effectiveness (the ability to achieve useful results) and efficiency6 (the ability to achieve results with little waste). OMB states in its most recent PART guidance document that “because a program’s performance goals represent its definition of success, the quality of the performance goals and actual performance [in achieving] those goals are the primary determinants of an overall PART rating” (OMB 2007b, p. 7). An annual retrospective analysis of results is also required.7 Box 1-2 indicates how PART questions are scored. Re- ports and scores for EPA ORD programs can be found in OMB (2007a). It appears, from the outset of its planning for PART, OMB recognized that research programs “would pose particular challenges for performance as- sessments and evaluations. For instance, in both applied and basic research, pro- jects take several years to complete and require more time before their meaning for the field can be adequately understood and captured in performance report- ing systems” (Posner 2004). BOX 1-1 Distribution of PART Scores Rating Percent and Number Effective 19% (186) Moderately effective 31% (319) Adequate 29% (289) Ineffective 3% (27) Results not demonstrated 19% (195) According to OMB, “[PART] assumes that a program that cannot demonstrate positive results is no more entitled to funding, let alone an in- crease, than a program that is clearly failing.” The consequences of a failing PART grade therefore would be nontrivial for a public agency, especially such a regulatory agency as EPA, whose actions take place in a political context. Source: OMB 2007a. 5 This distinction drew from the Army Research Laboratory’s evaluation criteria of quality, relevance, and productivity. 6 Efficiency is the ratio of the outcome or output to the input of any program (OMB 2006). 7 According to OMB (2007b, p. 85), “programs must document performance against previously defined output and outcome measures, including progress toward objectives, decisions, and termination points or other transitions.” 

16 Evaluating Research Efficiency in EPA BOX 1-2 How PART Questions Are Scored The four assessment categories are weighted according to the follow- ing scheme: • Program purpose and design: 20% (5 questions). • Strategic planning: 10% (8 questions). • Program management: 20% (11 questions for R&D programs). • Results and accountability: 50% (5 questions). As a default, individual questions within a category are assigned equal weighting that total 100% for each section. However, weighting may be al- tered to emphasize key factors of the program. In its 2004 budget statement, OMB wrote that the difficulty was most relevant to its preferred evaluation approach, which was to measure the out- comes8 of research (OMB 2004): “It is preferable to have outcome measures, but such measures are often not very practical to collect or use on an annual basis. The fact is there are no ‘right’ measures for some programs. Developing good measures is critical for making sure the program is getting results and making an impact.” To assist agencies with significant research programs, additional instruc- tions were added to the PART guidance and titled the “Research and Develop- ment Program Investment Criteria.” The R&D Investment Criteria are found in Appendix C of the PART instructions (see Appendix G). The main body of the PART instructions applies to all federal agencies and programs, including those that perform R&D; the R&D Investment Criteria attempt to clarify OMB’s ex- pectations for R&D programs. However, a shortcoming of the Investment Crite- ria is that the section on Performance does not use the word efficiency, so that agencies have had to extrapolate from other sections in the guidance in evaluat- ing that criterion. THE ORGANIZATION AND PERFORMANCE OF RESEARCH AND DEVELOPMENT AT THE ENVIRONMENTAL PROTECTION AGENCY Although PART applies to all federal research programs, the present re- port is concerned specifically with EPA’s experience under PART. That agency has experienced difficulties in achieving PART compliance for its R&D activi- 8 Outcomes may be defined as the results of research that have been integrated, as- sessed, and given regulatory or otherwise practical shape through a variety of actions. For example, an outcome of research on particulate matter may be improved air quality. 

Introduction 17 ties. This section briefly reviews the purpose and organization of those activi- ties. EPA is primarily a regulatory agency charged with developing regulations that broadly affect human health and the environment, but its regulatory actions are intended to be based on the best possible scientific knowledge as developed both within and outside the agency. Like any other agency, EPA cannot generate all the research it needs, but several previous National Research Council reports have underscored the importance of maintaining an active and credible program of internal research (NRC 2000b, 2003). A 1992 EPA report also states that science is one of the soundest investments the nation can make for the fu- ture. Strong science provides the foundation for credible environmental decision making. With a better understanding of environmental risks to people and ecosystems, EPA can target the hazards that pose the greatest risks, anticipate environmental problems before they reach a critical level, and develop strategies that use the nation’s, and the world’s, environ- mental protection dollars wisely (EPA 1992). Research at federal agencies, like other activities, is organized to support an agency mission. EPA’s process is described in its strategic plan. EPA drew up its first strategic plan in 1996 in response to GPRA. That plan, which has been renewed every 3 years, stated that “the mission of the U.S. Environmental Protection Agency is to protect human health and to safeguard the natural envi- ronment—air, water, and land—upon which life depends” (EPA 1997a). The current strategic plan (2006-2011) [EPA 2006] has five principal goals, all of which have scientific underpinnings: • Clean air and addressing global climate change. • Clean and safe water. • Land preservation and restoration. • Healthy communities and ecosystems. • Compliance and environmental stewardship. The plan also lists three “cross-goal strategies” that describe values meant to guide planning for all five goals: results and accountability, innovation and collaboration, and best available science (see Appendix D).9 Research-related activities at EPA, both internal and external, are the re- sponsibility of the Office of Research and Development (ORD) as well as EPA’s program offices and regional laboratories. The committee chose to focus its re- view on ORD’s research program as this is where the controversy regarding the 9 See Appendix D for more information about EPA’s strategic planning and multi-year planning process. 

18 Evaluating Research Efficiency in EPA development of its efficiency measures arose.10 The ORD conducts research in its in-house laboratories, develops risk-assessment methods and regulatory crite- ria, and provides technical services in support of the agency’s mission and its program and regional offices. It is organized in three national laboratories, four national centers, and two offices in 14 facilities around the country and its head- quarters in Washington, DC. ORD also has an extramural budget for grants, cooperative and interagency agreements, contracts, and fellowships that has ac- counted for 40-50% of its total budget in recent years.11 Its first strategic plan set forth a straightforward vision (EPA 1996, 1997b): “ORD will provide the scien- tific foundation to support EPA’s mission.” In 1999, ORD began to organize its scientific activities in multi-year plans to improve continuity and strategic integration. The multi-year plans, typically covering 5 years, are developed by research teams in ORD laboratories and cen- ters and are peer-reviewed. They cover 16 subjects (such as drinking water, safe food, and ecologic research) and are updated annually (EPA 2008). USES OF RESULTS OF THE ENVIRONMENTAL PROTECTION AGENCY RESEARCH The results of EPA research are used both by the agency itself and by various others outside the agency. The explicit purpose of both ORD research and extramural research is to provide scientific bases of EPA actions. The re- search may lead to end outcomes when results are integrated, assessed, and given regulatory or otherwise practical shape through actions in or outside EPA. Another important goal of EPA’s work, however, is to provide knowledge outputs for diverse organizations that have environmental interests and respon- sibilities, such as state and local governments, nongovernment organizations, international organizations, and community groups. Such entities interpret and use ORD outputs for their own planning and regulatory purposes. SUMMARY In the last decade and a half, two important efforts to evaluate the work of government agencies have been developed: GPRA, passed into law by Congress in 1993, and PART, designed in 2002 by OMB and first applied in 2003. Both efforts apply to all federal programs, including R&D programs. Evaluating the efficiency of R&D, required under PART, has proved to be challenging for all 10 For this reason, the committee did not address economic analysis, as this work is not conducted by ORD. A report summarizing the history of economics research at EPA through the 1990’s can be found at: http://yosemite.epa.gov/ee/epa/eed.nsf/webpages/ EconomicsResearchAtEPA.html. 11 For the FY 2007 enacted budget, 43% ($239,168,600 of $555,383,000) was budg- eted to be spent extramurally. [EPA, ORD, personal communication, 2007] 

Introduction 19 research-intensive agencies.12 One of those is EPA, which has sought the assis- tance of the National Academies in its effort to comply with the efficiency ques- tions of PART. The next chapter examines in some detail the complex process of PART compliance and how various agencies have responded. REFERENCES EPA (U.S. Environmental Protection Agency). 1992. Safeguarding the Future: Credible Science, Credible Decisions. The Report of the Expert Panel on the Role of Sci- ence at EPA. EPA/600/9-91/050. U.S. Environmental Protection Agency, Wash- ington, DC. EPA (U.S. Environmental Protection Agency). 1996. Strategic Plan for the Office of Research and Development. EPA/600/R-96/059. Office of Research and Devel- opment, U.S. Environmental Protection Agency, Washington, DC. EPA (U.S. Environmental Protection Agency). 1997a. EPA Strategic Plan. EPA/190-R- 97-002. Office of the Chief Financial Officer, U.S. Environmental Protection Agency, Washington, DC. EPA (U.S. Environmental Protection Agency). 1997b. Update to ORD’s Strategic Plan. EPA/600/R-97/015. Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC. EPA (U.S. Environmental Protection Agency). 2006. 2006-2011 EPA Strategic Plan: Charting Our Course. Office of the Chief Financial Officer, U.S. Environmental Protection Agency. September 2006 [online]. Available: http://www.epa.gov/cfo/ plan/2006/entire_report.pdf [accessed Feb. 7, 2008]. EPA (U.S. Environmental Protection Agency). 2008. Research Directions: Multi-Year Plans. Office of Research and Development, U.S. Environmental Protection Agency [online]. Available: http://www.epa.gov/ord/htm/aboutord.htm [accessed Feb. 7, 2008]. NRC (National Research Council). 1997. Building a Foundation for Sound Environ- mental Decisions. Washington, DC: National Academy Press. NRC (National Research Council). 1999. Evaluating Federal Research Programs: Re- search and the Government Performance and Results Act. Washington, DC: The National Academies Press. NRC (National Research Council). 2000a. Experiments in International Benchmarking of U.S. Research Fields. Washington, DC: National Academy Press. NRC (National Research Council). 2000b. Strengthening Science at the U.S. Environ- mental Protection Agency. Washington, DC: National Academy Press. NRC (National Research Council). 2001. Pp. 2-3 in Implementing the Government Per- formance and Results Act for Research Programs: A Status Report. Washington, DC: The National Academies Press. NRC (National Research Council). 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency’s Science to Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. 12 In this report, research-intensive is used to describe agencies of whose mission’s re- search is an essential, though not necessarily dominant, aspect. For example, research is important at EPA but not the primary function of the agency, as it is for the National Institutes of Health or the National Science Foundation. 

20 Evaluating Research Efficiency in EPA OMB (Office of Management and Budget). 2004. Pp. 49-52 in Rating the Performance of Federal Programs. The Budget for Fiscal Year 2004. Office of Management and Budget [online]. Available: http://www.gpoaccess.gov/usbudget/fy04/pdf/budget/ performance.pdf [accessed Nov. 7, 2007]. OMB (Office of Management and Budget). 2007a. ExpectMore.gov. Office of Manage- ment and Budget [online]. Available: http://www.whitehouse.gov/omb/expect more/ [accessed Nov. 7, 2007]. OMB (Office of Management and Budget). 2007b. Guide to the Program Assessment Rating Tool (PART). Office of Management and Budget. January 2007 [online]. Available: http://stinet.dtic.mil/cgi-bin/GetTRDoc?AD=ADA471562&Location= U2&doc=GetTRDoc.pdf [accessed Nov. 7, 2007]. Posner, P.L. 2004. Performance Budgeting: OMB’s Performance Rating Tool Presents Opportunities and Challenges for Evaluating Program Performance: Testimony be- fore the Subcommittee on Environment, Technology, and Standards, Committee on Science, House of Representatives, March 11, 2004. GAO-04-550T. Washing- ton, DC: U.S. General Accounting Office [online]. Available: http://www.gao.gov/ new.items/d04550t.pdf [accessed Nov. 7, 2007]. Stokes, D.E. 1997. Pp. 6-8 in Pasteur’s Quadrant: Basic Science and Technological Inno- vation. Washington, DC: Brookings Institution.