Cover Image


View/Hide Left Panel

The second central issue is the charge question of whether metrics used by federal agencies to measure the efficiency of research are “sufficient” and “outcome-based.” In approaching sufficiency, the committee gathered examples of methods used by agencies and organized them in nine categories. It found that most of the methods were insufficient for evaluating programs’ process efficiency either because they addressed only a portion of a program or because they addressed issues other than research, and all were insufficient for evaluating investment efficiency because they did not include the use of expert review.

In responding to the question of whether the metrics used are outcome-based, the committee determined that ultimate-outcome-based evaluations of the efficiency of research are neither achievable nor valid. The issue is discussed in Chapter 3.

Those two basic conclusions constitute the background of the major findings of this report. Findings 2, 4, 5, 6, and 7 are linked to specific charge questions, as indicated; findings 1 and 3 are more general.


  1. The key to research efficiency is good planning and implementation. EPA and its Office of Research and Development (ORD) have a sound strategic planning architecture that provides a multi-year basis for the annual assessment of progress and milestones for evaluating research programs, including their efficiency.

  2. All the metrics examined by the committee that have been proposed by or accepted by OMB to evaluate the efficiency of federal research programs have been based on the inputs and outputs of research-management processes, not on their outcomes.

  3. Ultimate-outcome-based efficiency metrics are neither achievable nor valid for this purpose.

  4. EPA’s difficulties in complying with PART questions about efficiency (questions 3.4 and 4.32) have grown out of inappropriate OMB requirements for outcome-based efficiency metrics.

  5. An “ineffective” (OMB 2007a)3 PART rating of a research program can have serious adverse consequences for the program or the agency.


Question 3.4 is “Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?” Question 4.3 is “Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?”


OMB (2007a) states that “programs receiving the Ineffective rating are not using tax dollars effectively. Ineffective programs have been unable to achieve results due to a lack of clarity regarding the program’s purpose or goals, poor management, or some other significant weakness. Ineffective programs are categorized as Not Performing.”

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement