Cover Image


View/Hide Left Panel

actions, including political and regulatory actions that are beyond the agency’s control and distant in time from the original research.

The committee believes that a link with ultimate outcomes is not the correct criterion for determining the sufficiency of metrics for evaluating research. Indeed, after analyzing agencies’ attempts to measure outcome-based efficiency, the committee concluded that for most research programs ultimate-outcome-based metrics for evaluating the efficiency of research are neither achievable nor valid. The committee considers this issue to be of such importance for its report that it amplifies its reasoning as follows:

  • There is often a large gap in time between the completion of research and the ultimate “outcome” of the research. In the case of EPA, for instance, the gap often is measured in years or even decades, commonly because the true outcome can be identified only by epidemiologic or ecologic studies that necessarily lag the original research itself. Thus, a retrospective outcome-based evaluation may be attempting to evaluate the “efficiency” of research conducted decades previously. Such an evaluation, if it can be done, may have little relevance to research being undertaken at the time of the evaluation.

  • A number of entities over which the research program has no control are responsible for translating research results into outcomes. In the case of EPA, such translation can involve multiple steps even for problem-driven research. The EPA program office has to convert research results into a risk-management strategy that complies with legislative requirements. That strategy undergoes substantial review and comment by other government agencies, the regulated community, and the public before it can be adopted. It may even be subjected to judicial review. When it is finally adopted, state agencies usually perform the implementation chores with their own corresponding risk-management strategies and programs. Even then, no ultimate outcomes appear until people, businesses, or other government units take action in response to the programs and their accompanying rules and incentives. The initial research program has no influence over any of those steps. If the initial activity is core research, the number and variety of organizations and individuals involved in producing outcomes may be even greater.

  • The results of research may change the nature of the outcome. The purpose of research is to produce knowledge, and new knowledge adds to the understanding of which outcomes are possible and desirable. To take another example of problem-driven research supported by EPA, suppose that results of a research project suggest that a particular chemical is toxic. That information may be only indicative, not definitive. EPA may launch a research program to confirm whether the chemical has toxic effects in humans or the natural environment. Results confirming toxicity would be expected to lead to a risk-management strategy that produces an ultimate outcome of reduced risk and improved health. In addition, EPA’s research on chemicals and development of toxicity screening tests provide industry with tools that impact their choice of

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement