and researchers may, by virtue of reading an evaluation report, gain an appreciation for the external factors that have shaped a research agenda, or they may see common threads across a set of evaluations that identify the need for new agency-wide strategic objectives. These less direct, longer term uses of evaluation products can function as a way of gradually infusing an organization with new ideas from the outside to the inside. Furthermore, external stakeholders or advocacy groups can use an evaluation report to call attention to needed research and priorities.
The committee commends NIOSH for undertaking the recent external review of a series of their programs by the National Academies, and in particular for requesting that the focus of the evaluations be on the relevance and impact in reducing work-related illnesses, injuries, or hazardous exposures. Many evaluations, particularly of research programs, stop with assessments of outputs, such as the number of peer-reviewed publications, and do not take into account the research transfer steps, the external factors that influence program activities and outcomes, or the need to use the program’s impact on intermediate or end outcomes as the metric for a successful program.
In an effort to continue the forward momentum, the committee recommends that NIOSH establish a system for periodic external evaluation of its programs. The system does not need to be modeled on the evaluations just carried out by the National Academies; indeed, many options are available, and NIOSH may want to incorporate elements of several in its overall approach. For example, the following types of evaluation approaches may be considered:
From time to time, an outside look at agency-wide processes is necessary.
Regular external review at the broad program level should be complemented with internal self-assessments on a more regular basis.
Formative evaluations at the program design or implementation stages can be useful for new programs.
Special studies of the long-term impacts of the program (such as the development of human capital) could be undertaken.
Whatever the composition or structure of external review, the research program should conduct a self-study prior to external review. This is analogous to the self-study that an academic program would conduct prior to accreditation. Much of the evaluation framework presented in Chapter 3, including the scoring system, may be useful in self-evaluations.
These approaches might emphasize different criteria for NIOSH effectiveness, balancing the appropriate focus on long-term impact in the workplace with attention to other public benefits. As noted above, in addition to research program evaluations, external evaluations of agency-wide initiatives or processes can be use-