Research is difficult to evaluate by any mechanism. Useful evaluation requires substantial elapsed time because research on a given scientific question may span 3-5 years from initiation of laboratory or field experiments to analysis and publication of results. Substantial time may also be required for training of EPA staff or the scientific community in scientific and technical advancements prior to the conduct of the research. Considerably more time may elapse before the broader impacts of published research are apparent (NRC 2003).
Although the committee was asked specifically to render advice on how EPA could best comply with the efficiency questions of PART, it concluded that more general suggestions on the evaluation of research would also have value for research-intensive agencies, for the Office of Management and Budget (OMB) and the Office of Science and Technology Policy (OSTP), and for Congress. It therefore examined the particular details of PART (Chapters 2-4), proposed principles that can be used to evaluate the results of research in any federal agency, and provided recommendations for EPA that other agencies also may find useful (Chapter 5).
Efficiency is a common enough concept, as illustrated by familiar dictionary definitions: “effective operation as measured by a comparison of production with cost (as in energy, time, and money)” and “the ratio of the useful energy delivered by a dynamic system to the energy supplied to it.”2 The PART approach to efficiency is explained this way by OMB (OMB 2007a, p. 9):
Efficiency measures reflect the economical and effective acquisition, utilization, and management of resources to achieve program outcomes or produce program outputs. Efficiency measures may also reflect ingenuity in the improved design, creation, and delivery of goods and services to the public, customers, or beneficiaries by capturing the effect of intended changes made to outputs aimed to reduce costs and/or improve productivity, such as the improved targeting of beneficiaries, redesign of goods or services for simplified customer processing, manufacturability, or delivery.
Any definition of efficiency depends on the process to which it is applied. Of relevance to this report is its application to the processes of research and development, which, as described by OMB, are complex and variable and involve
Merriam Webster Online, http://www.m-w.com/.