Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 21
Evaluating Research Efficiency in the U.S. Environmental Protection Agency 2 Efficiency Metrics Used by the Environmental Protection Agency and Other Federal Research and Development Programs The questions in the Program Assessment Rating Tool (PART) address many aspects of programs of the federal government, but the charge to this committee refers specifically to questions used to evaluate efficiency. The charge asks What efficiency measures are currently used for EPA R&D programs and other federally-funded R&D programs? Are these efficiency measures sufficient? Outcome based? What principles should guide the development of efficiency measures for federally-funded R&D programs? What efficiency measures should be used for EPA’s basic and applied R&D programs? This chapter addresses primarily the first question in the charge. To answer that question, the committee examined many of the efficiency metrics proposed to comply with PART by EPA and other federal agencies engaged in research. The committee reviewed documents, interviewed agency personnel, and heard presentations during a workshop in April 2007 that was attended by most of the research-intensive agencies and several large corporations that emphasize research.1 This chapter summarizes some of the challenges of evaluating research efficiency and ways in which agencies have approached those challenges. 1 The workshop was held on April 24, 2007, at the National Academies, 2101 Constitution Avenue, Washington, DC 20418. A workshop summary appears in Appendix B.
OCR for page 22
Evaluating Research Efficiency in the U.S. Environmental Protection Agency EVALUATING RESEARCH AND DEVELOPMENT Research is difficult to evaluate by any mechanism. Useful evaluation requires substantial elapsed time because research on a given scientific question may span 3-5 years from initiation of laboratory or field experiments to analysis and publication of results. Substantial time may also be required for training of EPA staff or the scientific community in scientific and technical advancements prior to the conduct of the research. Considerably more time may elapse before the broader impacts of published research are apparent (NRC 2003). Although the committee was asked specifically to render advice on how EPA could best comply with the efficiency questions of PART, it concluded that more general suggestions on the evaluation of research would also have value for research-intensive agencies, for the Office of Management and Budget (OMB) and the Office of Science and Technology Policy (OSTP), and for Congress. It therefore examined the particular details of PART (Chapters 2-4), proposed principles that can be used to evaluate the results of research in any federal agency, and provided recommendations for EPA that other agencies also may find useful (Chapter 5). THE PROGRAM ASSESSMENT RATING TOOL AND EFFICIENCY Efficiency is a common enough concept, as illustrated by familiar dictionary definitions: “effective operation as measured by a comparison of production with cost (as in energy, time, and money)” and “the ratio of the useful energy delivered by a dynamic system to the energy supplied to it.”2 The PART approach to efficiency is explained this way by OMB (OMB 2007a, p. 9): Efficiency measures reflect the economical and effective acquisition, utilization, and management of resources to achieve program outcomes or produce program outputs. Efficiency measures may also reflect ingenuity in the improved design, creation, and delivery of goods and services to the public, customers, or beneficiaries by capturing the effect of intended changes made to outputs aimed to reduce costs and/or improve productivity, such as the improved targeting of beneficiaries, redesign of goods or services for simplified customer processing, manufacturability, or delivery. APPLYING EFFICIENCY TO INPUTS, OUTPUTS, AND OUTCOMES Any definition of efficiency depends on the process to which it is applied. Of relevance to this report is its application to the processes of research and development, which, as described by OMB, are complex and variable and involve 2 Merriam Webster Online, http://www.m-w.com/.
OCR for page 23
Evaluating Research Efficiency in the U.S. Environmental Protection Agency inputs, outputs, and outcomes that vary by agency, program, and laboratory. Those terms are discussed by OMB (OMB 2007a) as follows: Inputs for purposes of PART are any agency resources that support research, which may include “overhead, intramural/extramural spending, infrastructure, and human capital” (OMB 2007a, p. 76). Outputs “describe the level of activity that will be provided over a period of time, including a description of the characteristics (e.g., timeliness) established as standards for the activity. Outputs refer to the internal activities of a program (i.e., the products and services delivered)” (OMB 2007a, p. 83). Outputs that have been used by agencies to comply with PART include research findings, papers published or cited, grants awarded, adherence to a projected schedule, and variance from cost and time schedules (OMB 2007b). Outcomes, according to OMB guidance, “describe the intended result of carrying out a program or activity. They define an event or condition that is external to the program or activity and that is of direct importance to the intended beneficiaries and/or the public” (OMB 2007a, p. 8). OMB gives the example of a tornado-warning system, whose outcomes “could be the number of lives saved and property damage averted” (OMB 2007a, p. 9). An outcome of research in support of the mission of a regulatory agency, such as EPA, may be the consequence of a regulation or other change that brings about some improvement in health or environmental quality. THE PROGRAM ASSESSMENT RATING TOOL GRADING SYSTEM The overall structure of PART was introduced in Chapter 1; additional aspects are described briefly here. Although this report focuses on the criterion of efficiency, the PART questions, taken in their entirety, attempt to address a broad range of issues with the goal of a thorough evaluation of federal programs. For example, section 1 of PART asks for general information about purpose and design. The second section, on strategic planning, asks such questions as whether a program has a long-term plan (2.1) and annual plans (2.2), both of which concern the relevance of research to the agency mission; it then asks whether the program is meeting long-term targets (2.3) and annual targets (2.4) concerning the quality of the research. The requirements for annual plans and annual targets often present problems for research managers. As discussed in Chapter 1, programs engaged in core research may be unable to specify the nature, timing, or benefits of their work annually because the results of core research can seldom be planned or analyzed in 1-year increments. Many of these sections also assume that evaluation be based on ultimate outcomes. For example, the PART guidance for question 2.2 (Does the program have ambitious targets and timeframes for its long-term measures?) states as follows: “For R&D programs, a Yes answer would require that the program pro-
OCR for page 24
Evaluating Research Efficiency in the U.S. Environmental Protection Agency vides multi-year R&D objectives. Where applicable, programs must provide schedules with annual milestones, highlighting any changes from previous schedules. Program proposals must define what outcomes would represent a minimally effective program and a successful program.” [Additional examples of the emphasis on ultimate outcomes can be seen in the sections of the guidance included in Appendix I.] In Section 3, which deals with management, question 3.4 specifies quantitative efficiency metrics as follows: “Does the program have procedures (e.g., competitive sourcing/cost comparison, IT improvements, and appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?” To win approval, a program must have “regular procedures to achieve efficiencies and cost effectiveness and at least one efficiency measure that uses a baseline and targets” (OMB 2007a, p. 26). Question 4.3 also addresses efficiency but appears in Section 4, on results and accountability: “Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?” To pass question 4.3, a program must have passed question 3.4 (that is, have “at least one efficiency measure that uses a baseline and targets”) and have demonstrated improved efficiency or cost effectiveness over the prior year. In discussions with the committee, OMB officials were consistent in supporting the use of outcome-based measures to evaluate efficiency. They also acknowledged that efforts to do so had not yet been successful.3 Because agencies are expected to provide a satisfactory answer to every PART question and several questions require answers on efficiency and annual achievements, this focus on efficiency and annual achievements can lead to a poor PART grade even if the relevance, quality, and effectiveness of a research program are demonstrated. That and other difficulties are addressed further in Chapter 3. THE USE OF “EXPERT REVIEW” AT THE ENVIRONMENTAL PROTECTION AGENCY EPA uses multiple mechanisms to evaluate its R&D activities, including internal processes of strategic plans, multi-year plans, and annual performance goals (mentioned in Chapter 1). The multi-year plans provide a means for tracking and, when necessary, adjusting research activities as they progress toward long-term goals. 3 As one example, the committee heard the following from OMB: “The requirement for 3.4 is that they have an efficiency measure. The highest standard is outcome. However, if that is not achievable, then they can have an output efficiency measure… But we are pushing for the outcomes, because if we just focus on the activities that we do, we don’t necessarily have the ability to find out if those activities and strategies are effective. That’s why another key component of the PART is evaluation” (from April 2007 workshop discussion; see summary in Appendix B).
OCR for page 25
Evaluating Research Efficiency in the U.S. Environmental Protection Agency To gain independent external perspective, EPA uses several standing “expert review” boards. Expert review is a broadened version of peer review, the mechanism by which researchers’ work is traditionally judged by other researchers in the same field.4 Expert review groups may include not only experts in the field under review but members from other fields and appropriate “users” of research results, who may represent the private sector, other agencies, non-government organizations (NGOs), state governments, labor unions, and other relevant bodies (NRC 1999). At EPA, expert reviewers are chosen for their skills, experience, and ability to judge not only the quality, relevance, and effectiveness of a program but whether it is being efficiently planned, managed, and revised in response to new knowledge—that is, whether it is efficient. Each panel should include members who have successfully run research programs themselves and are able to recognize good performance. The Science Advisory Board One of EPA’s long-standing review bodies is the Science Advisory Board (SAB), which includes a mix of scientists and engineers in academe, industry, state government, advisory bodies, and NGOs. The SAB was established by Congress in 1978 under a broad mandate to advise the agency on technical matters, including the quality and relevance of information used as the basis of regulations. The panel includes experts in science and technology policy, environmental-business planning processes, environmental economics, toxicology, resource management, environmental decision-making, ecotoxicology, risk perception and communication, decision analysis, risk assessment, civil and environmental engineering, epidemiology, radiologic health, air-quality modeling, public health, and environmental and occupational health (EPASAB 2007). According to the Overview of the Panel Formation Process at the Environmental Protection Agency Science Advisory Board (EPASAB 2002), EPA uses the following criteria in evaluating an individual panelist to serve on the SAB: Expertise, knowledge, and experience (primary factors). Availability and willingness to serve. 4 According to one definition, “peer review is a widely used, time-honored practice in the scientific and engineering community for judging and potentially improving a scientific or technical plan, proposal, activity, program, or work product through documented critical evaluation by individuals or groups with relevant expertise who had no involvement in developing the object under review” (NRC 2000). Expert review was also recommended by the Committee on Science, Engineering, and Public Policy panel cited in Chapter 1 for evaluating research to comply with the Government Performance and Results Act.
OCR for page 26
Evaluating Research Efficiency in the U.S. Environmental Protection Agency Scientific credibility and impartiality. Skills working in committees and advisory panels. The Board of Scientific Counselors The other principal EPA expert-review panel for ORD is the Board of Scientific Counselors (BOSC), a body of nongovernment scientists and engineers established in 1996 to provide advice, information, and recommendations to ORD. It has up to 15 members, and they meet three to five times a year. The BOSC reviews are relevant to this discussion because EPA is experimenting with their use as a mechanism for reviewing various aspects of research effectiveness. In 2004, the BOSC review process was restructured to focus on the three evaluation criteria of PART and to include both prospective and retrospective reviews of research programs. In 2006, three charge questions were developed for use in BOSC’s summary assessment of each program’s long-term goals: How appropriate is the research used to achieve each long-term goal? Is the program still asking the right questions, or have they been superseded by advancements in the field? (Relevance) How good is the technical quality of the program’s research products? (Quality) How much are the program results being used by environmental decision-makers to inform decisions and achieve results? (Performance) The BOSC review process also feeds into PART through several other questions. For example, the BOSC review is submitted in response to question 4.5, “Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?”5 In spring 2006, a BOSC panel added the charge questions discussed above to its evaluation criteria; it used them for the first time early in 2007. Although there is much overlap between the BOSC investigations and the PART questions regarding results of a program, OMB had not by the time of this committee’s investigation determined whether the use of BOSC’s revised charge met the requirements of PART.6 During the 2005 PART review for EPA’s drinking-water program, EPA experimented with a quantitative version of the BOSC evaluation. It involved nine questions, for each of which the committee gave a rating of 1-5 to provide a 5 For additional material about BOSC, see EPA’s Draft Board of Scientific Counselors Handbook for Subcommittee Chairs, Appendix B, p. 18. 6 During the July 2007 workshop, committee members discussed this issue with representatives of OMB and EPA. The EPA representatives described current efforts to develop a quantitative system for use by BOSC.
OCR for page 27
Evaluating Research Efficiency in the U.S. Environmental Protection Agency numerical grade. That process was not accepted as scientifically valid by the BOSC. Two BOSC reviews were in progress at the time of this report: on pesticides and toxics and on sustainability research. EPA staff noted that the reviews will serve as baselines for later reviews. EPA is providing the BOSC review committees with two kinds of data, among others, to evaluate the performance of research programs: pilot surveys that evaluate how the research is being used and bibliometric analyses (P. Juengst, EPA, personal communication, 2007). Box 2-1 provides an example of a BOSC expert review. According to EPA staff, there is increasing pressure from OMB to focus on outcome-based efficiency metrics, but the agency has been unable to establish such metrics for research. EMERGING ISSUES One important role of expert review is to complement the ability of program managers and agency leaders to anticipate important emerging issues. Strategic effectiveness rises when the agency plans for the “next big thing,” rather than awaiting its sudden arrival. The program managers necessarily focus their attention on the day-to-day demands of administration, but expert reviewers can survey agency research in a wider context. To the degree that an agency can position itself at the forefront of a new field, it can increase its research relevance, quality, and performance. METRICS PROPOSED BY THE ENVIRONMENTAL PROTECTION AGENCY EPA, like other agencies, proposed quantitative metrics to measure various kinds of efficiency in its PART compliance, and many of them were accepted by OMB. Metrics considered or used by EPA included the use of its research results to support regulations, surveys to gauge client satisfaction with its products, average time spent in producing assessments, overhead as a fraction of research, and citations per dollar invested. Such metrics fit well with the existing strategic and multi-year planning that provides annual milestones against which to evaluate them. Like other agencies, EPA has proposed that an increase in the number of peer-reviewed publications produced per full-time equivalent (FTE) complies with the PART guidance that “efficiency measures could focus on how to produce a given output level with fewer resources” (OMB 2006a, p. 10). That was not accepted by OMB examiners for question 3.4 as an efficiency metric for the Water Quality Research Program. OMB found that the lack of a tight linkage between publications and budget made it hard to determine whether money was being spent appropriately. Publications might have been far ahead of schedule and over budget, for example, or behind schedule and under budget (K. Neyland, OMB, personal communication, 2007).
OCR for page 28
Evaluating Research Efficiency in the U.S. Environmental Protection Agency BOX 2-1 BOSC: An Example of Expert Review One example of the composition and function of a BOSC expert-review panel is its Human Health Subcommittee, which issued a report on EPA’s Human Health Research Program (HHRP) in 2005. The panel included eight members in academe, industry, and government.7 The panel met for 3 days and stated its purpose as follows: “The objective of this review is to evaluate the relevance, quality, performance, and scientific leadership of the Office of Research and Development’s (ORD’s) Human Health Research Program.” It evaluated the overall program’s relevance, quality, performance, and leadership relative to each of its four long-term goals: Use of mechanistic data in risk assessment. Aggregate and cumulative risk assessment. Evaluation of risk to susceptible subpopulations. Evaluation of public-health outcomes. The subcommittee visited the HHRP’s main facility, in Research Triangle Park, North Carolina, where it heard from EPA offices and programs regarding the utility of research products developed by ORD scientists in the HHRP. The expert-review panel received extensive confirmation that ORD scientists were helpful to the various EPA regions in hosting regional scientists in ORD laboratories, collaborating with the regions on regional environmental problems, providing scientific consultation to the regions to help to ameliorate their environmental problems, and providing scientific consultation to the regions on specific problems in environmental toxicology. Earned-Value Management EPA staff also approached OMB to discuss the use of EVM as an efficiency metric for its ecologic research program. EVM measures the degree to which research outputs conform to scheduled costs along a timeline. It is used by agencies and other organizations in many management settings, such as construction projects and facilities operations, where the outcome (such as a new 7 The members of the Human Health Subcommittee, according to the BOSC Web site, had “considerable expertise in the area of human health research, including formal education, training, and research experience in biology, chemistry, biochemistry, environmental carcinogenesis, pharmacology, molecular biology and molecular mechanisms of carcinogenicity and toxicity, toxicology, physiologically based pharmacokinetic (PBPK) modeling, exposure modeling, risk assessment, epidemiology, biomarkers and biological monitoring, and public health, with additional expertise in the areas of children’s health, community-based human exposure studies, and clinical experience” (EPA 2005, p. 1).
OCR for page 29
Evaluating Research Efficiency in the U.S. Environmental Protection Agency laboratory or optimal use of facilities) is well known in advance, and progress can be plotted against milestones. Although EPA and other agencies have found value in using EVM to measure the efficiency of some processes, they have not found a way to apply it to research outcomes. METRICS THAT DID NOT PASS THE PROGRAM ASSESSMENT RATING TOOL PROCESS Since 2004, when PART grading began, several major research-based EPA programs have been rated “ineffective” or “results not demonstrated.” One was the Ecological Research Program (ERP), which in 2005 was given an “ineffective” rating,8 including a “no” on question 3.4. At that time, it was noted that the program lacked an acceptable efficiency measure, but was working to develop one. In EPA’s view, the agency failed to provide an acceptable efficiency metric because it could not measure the outcome efficiency of its research (M. Peacock, EPA, personal communication, 2007). It was also given a zero score on question 4.3 because a program that fails to have an acceptable efficiency metric on question 3.4 cannot demonstrate annual increases in efficiency. Other programs that have not passed the efficiency questions include the National Ambient Air Quality Standards Program and the Ground Water and Drinking Water Program, for similar reasons. In examining OMB documents and the ExpectMore.gov Web site, the committee found that for most research-intensive agencies other than EPA, OMB accepted efficiency measures similar to those proposed by EPA, such as scheduled regulatory decision-making activities. No agency has responded to PART questions 3.4 and 4.3 by using outcome-based efficiency measures for R&D programs. The Appeals Process EPA appealed the denial of using publications per FTE as an efficiency metric for the Water Quality Research Program to an OMB Appeals Board, which accepted the appeal on several conditions. One was that “the program should include a follow-up action in its PART improvement plan relating to developing an outcome-oriented efficiency metric,” and another was that the program “must have a baseline and targets” (OMB 2006b). Even though the use of “outcome-oriented efficiency metrics” is not required by the PART guidance or the R&D Investment Criteria, it is strongly preferred by OMB and sometimes required by the examiner and/or during the appeals process. In both EPA and other research-intensive agencies, concerns have been voiced by those involved 8 PART rated 3% of federal programs “ineffective.” It rated 19% as “results not demonstrated” (OMB 2007c).
OCR for page 30
Evaluating Research Efficiency in the U.S. Environmental Protection Agency in PART compliance that the application of rules sometimes seems inconsistent or confusing.9 In addition, OMB continues to encourage development of a version of EVM that will prove satisfactory for the purpose of evaluating the efficiency of R&D programs, but at the time of the present committee’s study the issue had not been resolved (B. Kleinman, OMB, personal communication, 2007). THE CONSEQUENCES OF A “NO” ANSWER TO A PROGRAM ASSESSMENT RATING TOOL QUESTION Because PART was initiated in 2003 and has not yet examined all research programs of federal agencies, there is little information about the effects of low ratings on agencies. However, as demonstrated above by the example of the ERP, a program may do poorly in the PART process if it does not have an acceptable measure of efficiency even if it has high marks for relevance and quality. A rating of “ineffective” for research cannot be helpful for a regulatory agency like EPA, whose authority rests in part on its reputation for sound scientific research. Indeed, after the “ineffective” PART ratings were applied to the ERP in 2005, the program suffered substantial erosion of support (Morgan 2007).10 According to figures cited by EPA’s Risk Policy Report (Sarvana 2007), Congress reduced extramural funding for “ecology and global change” through EPA’s National Center for Environmental Research from about $32 million in FY 2002 to less than $25 million in FY 2003. The program received a similar cut from FY 2004, when it received about $24 million, to FY 2005, when it received only about $8 million.11 9 According to one study of the early application of PART, “The patterns of rating programs are not very clear regarding the FY 2004 process, largely because of variability among the OMB budget examiners. The variability was pointed out by GAO in its assessment of the process” (Radin 2006, p. 123). See also comments by NASA representative on p. 33 below. 10 In his testimony before the Subcommittee on Energy and Environment, Committee on Science and Technology, U.S. House of Representatives, March 2007, M. Granger Morgan, of Carnegie Mellon University (CMU), noted the agency’s difficulty in carrying out the work necessary to comply with PART while its budget was reduced. He stated that “it appears seriously misguided to raise the bar for comprehensive cost-effective or benefit-cost justification for environmental science research, while simultaneously shrinking the resources devoted to the types of research needed to assess the net social benefits of the outcomes of environmental science research.” Morgan is chair of CMU’s Department of Engineering and Public Policy, chair of the EPA SAB, and an expert in risk analysis and uncertainty (Morgan 2007, p. 4). 11 Most recently, the ERP received a positive rating, again as reported by InsideEPA.com: “The rating was part of OMB’s Program Assessment Rating Tool (PART) process, which rates federal programs’ performance and helps set budget levels. It is the third PART review of ERP since OMB launched the initiative in 2002, and the first to give the program a positive score. Previous PART reviews criticized ERP for not fully
OCR for page 31
Evaluating Research Efficiency in the U.S. Environmental Protection Agency An “ineffective” PART rating also affects program ratings under the President’s Management Agenda (PMA). The PMA is relevant to this document, even though the committee’s charge did not specify it, because PART is a component of the PMA. The PMA is generally an agency-level effort with five initiatives, one of which is Budget-Performance Integration (BPI). All programs assessed by PART must have acceptable efficiency measures for the agency to receive a “green” score in the BPI.12 EVALUATION MECHANISMS USED BY OTHER AGENCIES The committee and staff have consulted with other agencies about their PART evaluation processes. They also invited research-intensive agencies’ representatives to a workshop at the Academies in April 2007 to describe their processes and compare results (see Appendix B for the workshop summary). This section is based on the workshop presentations and followup conversations. Metrics of Efficiency Accepted by the Office of Management and Budget OMB makes clear its general preference for “outcome efficiency” in PART compliance (OMB 2006a), but the mechanisms proposed by agencies are metrics of output (process) efficiency. The committee compiled and reviewed an “efficiency measures table” of efficiency metrics used for research programs by 11 federal agencies, including EPA, in following the PART guidance13 (see Appendix E for Table E-1). The table also includes information gathered from four corporations that have R&D programs. The following list is a sample of the common types of metrics proposed by the agencies, many of which have been accepted by OMB: Time to process and award grants. Time to respond to information requests. Publications per FTE (or per dollar). Percentage of budget that is overhead. Percentage of work that is peer-reviewed. Average cost per measurement or analysis. Cost-sharing. demonstrating the results of programmatic and research efforts—and resulted in ERP funding cuts” (Inside EPA Risk Policy Report 2007). 12 The PMA awards agencies a green, yellow, or red rating. As noted in PART guidance (OMB 2007a, p. 9): “The President’s Management Agenda (PMA) Budget and Performance Integration (BPI) Initiative requires agencies to develop efficiency measures to achieve Green status.” 13 The complete table is in Appendix E.
OCR for page 32
Evaluating Research Efficiency in the U.S. Environmental Protection Agency Quality or cost of equipment and other inputs. Variance from schedule and cost. Chapter 3 asks whether those metrics are “sufficient” for evaluating research efficiency. Department of Energy The Department of Energy (DOE) uses PART to evaluate many management processes of its Office of Science. For evaluating the quality and relevance of research, DOE depends on peer review of all portfolios every second or third year by “committees of visitors.” That was found to be fairly cost-effective, allowing the agency to look at what was proposed and how well it was performed, to identify ideas that lack merit, to discontinue inefficient processes, to redirect R&D, or to terminate a poorly performing project. With the creation of PART, a committee was established to test appropriate metrics, but the committee has not found a way to assign value to a basic-research portfolio. A DOE representative commented that the work is valued according to its societal and mission accomplishments; this has to be done by working closely with the scientific community. A DOE representative said that the director of the President’s Office of Science and Technology Policy had established a committee charged with developing a mechanism for measuring the value of research and estimating the cost of compliance. National Science Foundation Like DOE, the National Science Foundation (NSF) uses external committees of visitors to perform peer review (called merit review) on programs or portfolios every 3 years. Merit review is a detailed and long examination of technical merit and broader impacts of research. NSF tracks efficiency primarily in two ways. One measures the time to decision on research awards; the second measures facility costs, schedules, and operations, with specific goals for each. National Aeronautics and Space Administration The National Aeronautics and Space Administration (NASA) uses PART exercises in evaluating the efficiency of repetitive, stable, and baseline processes and some aspects of R&D, such as financial management, contracting, travel processing, and capital-assets tracking. The agency has been using PART metrics to track and evaluate the complex launch process and to find safe ways to reduce the size of the Space Shuttle workforce. Other uses were planned, such as
OCR for page 33
Evaluating Research Efficiency in the U.S. Environmental Protection Agency increasing the on-time availability and operation of ground test facilities and reducing the cost per minute of operating space network support for missions. Like other agencies, NASA does not find PART useful for evaluating the efficiency of research, especially unrepeatable projects, such as discoveries dictated by science or the development of prototypes. A NASA representative notes that the PART process depended heavily on the PART examiners, who tended to vary widely in attitudes and experience. Although the NASA examiners typically had scientific or engineering backgrounds, this was not necessarily the case for OMB policy-makers crafting the PART policies and guidance. The representative notes that because the OMB policy-makers generally did not have research backgrounds, NASA spends considerable time in educating them about the relevant differences in R&D programs, as these differences are not well considered in PART guidance. The representative suggests more flexibility in the actions of the reviewers—for example, in recognizing that short-term decreases in efficiency might lead to long-term efficiency gains and in seeing the need to balance efficiency with effectiveness.14 National Institutes of Health National Institutes of Health (NIH) staff described using PART on research and research-support activities. For example, the extramural research program has achieved cost savings through improved grant administration. The intramural research program has used it to reallocate laboratory resources, and the building and facilities program has monitored its property condition index. The extramural construction program has achieved economies by expanding the use of electronic management tools to monitor construction and occupancy for 20-years post-completion. NIH staff notes that the PART approach to efficiency is the same as a business model which emphasizes time, cost, and deliverables. With such a model, efficiency can be increased by improving any variable, so long as the other two do not worsen. This approach does not fit the scientific discovery process. Some 99% of the NIH portfolio has been subjected to PART; 95% percent of the programs are rated as effective, and the other 5% as moderately effective. External research (close to 90% of the budget), which is not under NIH’s direct control, is excluded, although the agency coordinates with awardees to ensure performance. 14 In a case study of the PART process at the Department of Health and Human Services, a similar difficulty was described. “In some cases, OMB budget examiners were willing to deal with multiple elements of programs as a package; in other cases, the examiner insisted that a small program would require individual PART submissions. It was not always clear to HHS staff why a particular program received the rating it was given; OMB policy officials did not appear to have a consistent view of the PART process” (Radin 2006, p. 142).
OCR for page 34
Evaluating Research Efficiency in the U.S. Environmental Protection Agency NIH staff notes that, in scientific discovery, variables are largely unknown because the outcome is unpredictable knowledge, and the inputs of time, cost, and resources are difficult to estimate. Research does not fit easily into a business model, for other reasons cited: In research, high-risk projects are strongly associated with innovative outcomes that may initially fail even though the scientific approach was sound. A research outcome may be unexpected or lead to an unexpected benefit. Changing direction, which may look like poor management in the context of a business model, may be good research practice in order to conduct good science. A business model does not capture the null hypothesis, or a serendipitous finding that gives valuable information (Duran 2007). These types of research results provide valuable information, which is likely not credited in a business model approach. National Institute for Occupational Safety and Health The National Institute for Occupational Safety and Health (NIOSH) functions as the research partner of the Occupational Safety and Health Administration. NIOSH uses independent expert review to evaluate its 30 research programs, which exist in a “matrix” with substantial overlaps. A model for evaluating a research program was provided by a framework developed for NIOSH by the National Academies with a blend of quantitative and qualitative elements. A central feature of the framework is that it adds to outputs and outcomes a third metric, “intermediate outcomes.” The categories were associated with the following kinds of results (Sinclair 2007): Outputs. Peer-reviewed publications, NIOSH publications, communications to regulatory agencies or Congress, research methods, control technologies and patents, and training and information products. Intermediate outcomes. Regulations, guidance, standards, training and education programs, and pilot technologies. End outcomes. Reductions in fatalities, injuries, illnesses, and exposures to hazards. The Academies’ review of NIOSH procedures included several approaches to program evaluation, including the following general road map for characterizing the evaluation process as a whole (NRC 2007): Gather appropriate information from NIOSH and other sources. Determine timeframe for the evaluation.
OCR for page 35
Evaluating Research Efficiency in the U.S. Environmental Protection Agency Identify program-subject challenges and objectives. Identify subprograms and major projects in the research program. Evaluate the program and subprogram components sequentially (this will involve qualitatively assessing each phase of a research program). Evaluate the research program’s potential outcomes not yet appreciated. Evaluate and score the program’s potential outcomes and important subprogram outcomes specifically for contributions to the environment and health. Evaluate and score the overall program for quality, using a numerical scale. Evaluate and score the overall program for relevance, using a numerical scale. Evaluate and score the overall program for performance (effectiveness and efficiency), using a numeric scale. Identify important emerging research. Prepare report. One attribute of that approach, said a NIOSH representative, is that it is flexible enough to apply to R&D programs in which efficiency metrics are appropriate for some functions but not for others. It allows evaluation of a research program by focusing on outputs agreed to in the multi-year plan, and the outcomes are embedded in the plan itself. The plan can thus be evaluated by year-over-year results that lay the foundation for the outcomes. The NIOSH plan could also make use of a numeric scale for performance. For example, expert-review panels could be asked to assess performance on a scale of 1-5. METHODS USED BY INDUSTRY Representatives of four industries that perform research participated in the committee’s public workshop and described how research delivers value to their companies. They all had clear rationales for investing in research and spent substantial amounts (typically about 1% of sales) on research activities, but none used EVM, none evaluated research in terms of efficiency, and none described direct links to outcomes (sales).15 Chapter 3 offers additional perspective on evaluating research efficiency in industry. SUMMARY OMB has required that federal agencies measure the efficiency of their 15 See workshop summary in Appendix B.
OCR for page 36
Evaluating Research Efficiency in the U.S. Environmental Protection Agency activities to comply with PART. This chapter has summarized how agencies with substantial research programs, including EPA, have attempted to comply with that requirement. In the next chapter, we discuss whether the metrics that agencies have proposed and are using are “sufficient” for evaluating the performance of research, including its efficiency, and discuss a balanced approach to such evaluation. REFERENCES Duran, D. 2007. Presentation at the Workshop on Evaluating the Efficiency of Research and Development Programs at the Environmental Protection Agency, April 24, 2007, Washington, DC. EPA (U.S. Environmental Protection Agency). 2005. Program Review Report of the Board Scientific Counselors: Human Health Research Program. Office of Research and Development, U.S. Environmental Protection Agency [online]. Available: http://www.epa.gov/osp/bosc/pdf/hh0507rpt.pdf [accessed Nov. 8, 2007]. EPASAB (U.S. Environmental Protection Agency Science Advisory Board ). 2002. Overview of the Panel Formation Process at the Environmental Protection Agency. EPA-SAB-EC-02-010. Office of the Administrator, Science Advisory Board, U.S. Environmental Protection Agency, Washington, DC. September 2002 [online]. Available: http://www.epa.gov/sab/pdf/ec02010.pdf [accessed Nov. 8, 2007]. EPASAB (U.S. Environmental Protection Agency Science Advisory Board ). 2007. U.S. Environmental Protection Agency Chartered Science Advisory Board FY 2007 Member Biosketches. Science Advisory Board, U.S. Environmental Protection Agency, Washington, DC [online]. Available: http://www.epa.gov/sab/pdf/board_biosketches_2007.pdf [accessed Nov. 8, 2007]. Inside EPA’s Risk Policy Report. 2007. Improved OMB Rating May Help Funding for EPA Ecological Research. Inside EPA’s Risk Policy Report 14(39). September 25, 2007. Morgan, G.M. 2007. Testimony M. Granger Morgan, Chair U.S. Environmental Protection Agency Science Advisory Board Before the Subcommittee on Energy and Environment Committee on Science and Technology, U.S. House of Representatives, March 14, 2007 [online]. Available: http://www.epa.gov/sab/pdf/2gm_final_written_testimony_03-14-07.pdf [accessed Nov. 8, 2007]. NRC (National Research Council). 1999. Evaluating Federal Research Programs: Research and the Government Performance and Results Act. Washington, DC: National Academy Press. NRC (National Research Council). 2000. P. 99 in Strengthening Science at the U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices. Washington, DC: The National Academy Press. NRC (National Research Council). 2003. P. 3 in The Measure of STAR: Review of the U.S. Environmental Protection Agency’s Science to Achieve Results (STAR) Research Grants Program. Washington DC: The National Academies Press. NRC (National Research Council). 2007. Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health. Aug. 10, 2007. OMB (Office of Management and Budget). 2006a. Guide to the Program Assessment Rating Tool (PART). Office of Management and Budget. March 2006 [online].
OCR for page 37
Evaluating Research Efficiency in the U.S. Environmental Protection Agency Available: http://www.whitehouse.gov/omb/part/fy2006/2006_guidance_final.pdf [accessed Nov. 7, 2007]. OMB (Office of Management and Budget). 2006b. PART Appeals Board Decisions-Environmental Protection Agency. Letter from Clay Johnson, III, Deputy Director for Management, Office of Management and Budget, Washington, DC, to Marcus C. Peacock, Deputy Administrator, U.S. Environmental Protection Agency, Washington, DC. August 28, 2006. OMB (Office of Management and Budget). 2007a. Guide to the Program Assessment Rating Tool (PART). Office of Management and Budget. January 2007 [online]. Available: http://stinet.dtic.mil/cgi-bin/GetTRDoc?AD=ADA471562&Location=U2&doc=GetTRDoc.pdf [accessed Nov. 7, 2007]. OMB (Office of Management and Budget). 2007b. Circular A–11, Part 7, Planning, Budgeting, Acquisition and Management of Capital Assets. Office of Management and Budget, Executive Office of the President. July 2007 [online]. Available: http://www.whitehouse.gov/omb/circulars/a11/current_year/part7.pdf [accessed Nov. 9, 2007]. OMB (Office of Management and Budget). 2007c. ExpectMore.gov. Office of Management and Budget [online]. Available: http://www.whitehouse.gov/omb/expectmore/ [accessed Nov. 7, 2007]. Radin, B. 2006. Challenging the Performance Movement: Accountability, Complexity and Democratic Values. Washington, DC: Georgetown University Press. Sarvana, A. 2007. Science Board Joins House in Opposing EPA Ecosystem Research Cuts. Inside EPA’s Risk Policy Report 14 (37):1, 8-9. September 11, 2007. Sinclair, R. 2007. Presentation at the Workshop on Evaluating the Efficiency of Research and Development Programs at the Environmental Protection Agency, April 24, 2007, Washington, DC.