From the outset of the evaluations, NIOSH leadership established the primary goal as program improvement, but the context for the evaluations also included the PART (Program Assessment Rating Tool) federal agency evaluation process.

The first step in this multiphase effort was to develop an evaluation framework that could be applied across the set of program evaluations to enhance cross-study consistency. An Institute of Medicine (IOM)/National Research Council (NRC) committee (the framework committee) was appointed to develop the evaluation framework. The resulting evaluation framework was then used by eight separately appointed ad hoc committees (evaluation committees) to assess NIOSH programs in hearing loss; mining; agriculture, forestry, and fishing; respiratory diseases; personal protective technology; traumatic injury; construction; and health hazard evaluation. Each evaluation committee produced an individual report (IOM and NRC, 2006, 2008, 2009; NRC and IOM, 2007, 2008a,b, 2009a,b).

This report provides the evaluation framework developed, implemented, and refined over the course of four years and eight evaluations. The framework uses a standard tool in program management and evaluation—the logic model—and provides details on the types of information that are needed and questions to be considered in each phase of the evaluation. This report has two goals: (1) to summarize the evaluation process and lessons learned in the development and use of the framework and (2) to provide recommendations for future evaluation efforts. The evaluation framework may prove applicable in evaluating other federal agency research programs.

SCOPE OF THE TASK

The framework and evaluation committees followed the same basic statement of task (Box 1-1). Although the statement of task was modified to clarify specific issues or to accommodate programs that were not specifically research programs,2 the basic objectives for the program evaluations remained the same:

  • An assessment of the relevance and impact of the NIOSH program’s contribution to reducing work-related hazardous exposures, illnesses, and injuries based on integer scales of 1 to 5, with text to support the rating;

  • Assessment of the program’s effectiveness in targeting new research areas and identification of emerging issues that the program should be prepared to address; and

  • Recommendations for program improvement.

2

The charges to the committees to evaluate the NIOSH Health Hazard Evaluation and Personal Protective Technology programs were each slightly modified to accommodate the unique standards-setting and investigative aspects of these programs.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement