available in the individual committee reports (IOM and NRC, 2006, 2008, 2009; NRC and IOM, 2007, 2008a,b, 2009a,b).

EVALUATION FRAMEWORK

Applying the Logic Model and Assessing Outcomes

As discussed in Chapter 2, the logic model is used increasingly by program evaluations to delineate what a program does and accomplishes into the categories of inputs, activities, outputs, and outcomes (see Figure 2-1). The focus of this series of NIOSH program evaluations was on assessing the relevance and impact of the NIOSH program on reducing work-related injuries, illnesses, and hazardous exposures (end outcomes). Initially a number of framework committee members had concerns about developing evaluation criteria that hold a research agency accountable for end outcomes when so many of the factors that affect end outcomes (e.g., workplace conditions, regulations) are outside of the agency’s authority or control. Many members of the evaluation committees shared similar concerns.

Several aspects of the framework that were ultimately adopted made the focus on end outcomes more acceptable. The first was the fact that the framework calls for the identification of external factors and provides a fairly extensive list of examples. This provides an opportunity for evaluators to identify obstacles to or promoters of research impact and underscores the changes necessary outside the research agency to improve worker safety and health. A more conventional approach to research evaluation that focuses on knowledge generation would not typically allow for making explicit statements about these obstacles. The second aspect was the clear direction that intermediate outcomes could be used as measures of success in the absence of end outcomes. This was particularly appropriate for exposure-illness relationships that have years of latency before diagnosis. Intermediate outcomes are observable results that could plausibly lead from outputs to end outcomes. Intermediate outcomes (e.g., policy change, adoption of NIOSH-developed technologies in the workplace) may serve as the short- and medium-term proxies for expected end outcomes.

Setting the metric for program success at demonstrating an impact on end outcomes is laudable. Evaluation and framework committee members give NIOSH a great deal of credit for holding their research programs accountable for real outcomes that affect life and health. Evaluations of research supported by other federal agencies often focus on output productivity and intermediate outcomes and do not hold the agency accountable for real-world impacts. This may be appropriate for the goals of some research programs, but NIOSH’s focus is on applied research



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement