Because of the nature of research, it is not possible to measure the creation of new knowledge as it develops–or the pace at which research progresses or scientific breakthroughs occur. Demonstrating research contributions to outcomes is very challenging when federal agencies conduct research to support regulatory decisions, and then rely on third parties–such as state environmental agencies–to enforce the regulations and demonstrate environmental improvements. Typically, many years may be required for practical research applications to be developed and decades may be required for some research outcomes to be achieved in a measurable way.
Most of ORD’s environmental research programs investigate complex environmental problems and processes—combining use-inspired basic research with applied research, and integrating several scientific disciplines across a conceptual framework that links research to environmental decisions or environmental outcomes. In multidisciplinary research programs such as these, progress toward outcomes can not be measured by outputs created in a single year. Rather, research progress occurs over several years, as research teams explore hypotheses with individual studies, interpret research findings, and then develop hypotheses for future studies.
In designing and managing its research programs, ORD emphasizes the importance of identifying priority research questions or topics to guide its research. Similarly, ORD recommends that its programs develop a small number of performance goals that serve as indicators of progress to answer the priority questions and to accomplish outcomes. Short-term outcomes are accomplished when research is applied by specific clients, e.g., to strengthen environmental decisions. These decisions and resulting actions (e.g., the reduction of contaminant emissions or restoration of ecosystems) ultimately contribute to improved environmental quality and health.
In a comprehensive evaluation of science and research at EPA, the National Research Council recommended that the Agency substantially increase its efforts to both explain the significance of its research products and to assist clients inside and outside the Agency in applying them. In response to this recommendation, ORD has engaged science advisors from client organizations to serve as members of its research program teams. These teams help identify research contributions with significant decision making value and help plan for their transfer and application.
For ORD’s environmental research programs, periodic retrospective analysis at intervals of four or five years is needed to characterize research progress, to assess how clients are applying research to strengthen environmental decisions, and to evaluate client feedback about the research. Conducting program evaluations at this interval enables assessment of: research progress, the scientific quality and decision-making value of the research, and whether research progress has resulted in short-term outcomes for specific clients.
A description of the OSTP/OMB Research and Development Investment Criteria is included in Appendix I.