Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps (2009)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Summary ABSTRACT The goal of research conducted by the National Institute for Occupational Safety and Health (NIOSH) is protecting and improving the health and safety of workers. The Institute of Medicine and the National Re- search Council conducted a series of evaluations of NIOSH research programs that assessed the relevance and impact of NIOSH’s work on meeting this goal. Eight NIOSH programs were evaluated based on a common framework. This report provides the lessons learned in the evaluation process, a revised evaluation framework, and the following recommendations for evaluating occupational health and safety research programs: continue systematic ex- ternal evaluations, bolster research translation efforts, enhance occupational health and safety surveillance, and integrate evaluations of intramural and extramural research. P reventing tractor rollovers on farmers, protecting construction workers from falls, improving the health of miners in dusty environments, reducing back injuries in nursing aides, developing substitute materials to eliminate haz- ardous chemical exposures, and designing work conditions to reduce fatigue and stress are among the issues critical to improving worker safety and health. In 2007, 5,657 fatal work injuries occurred in the United States, along with an estimated 4 million nonfatal occupational injuries and illnesses among workers in private industry. In addition to the human suffering involved, these statistics are also asso- ciated with high economic costs. One estimate puts the costs of occupational injury  

 E v a l u a t i n g O c c u p a t i o n a l H e a l t h a n d S a f e t y Research Programs and illnesses for all industries for 2005 at more than $160 billion. The continued atten­tion to further improve occupational health and safety through research is not only fully ­warranted but such research requires critical evaluation for its relevance and impact. The core mission of the National Institute for Occupational Safety and Health (NIOSH) is to conduct research to improve and protect the health and safety of workers. In September 2004, NIOSH contracted with The National Academies to con- duct a series of evaluations of individual NIOSH research programs. This set of in- dependent evaluations focused on the relevance and impact of each of eight NIOSH programs on reducing work-related injuries, illnesses, and hazardous exposures. From the outset of the evaluations, NIOSH leadership established the primary goal as program improvement but the context for the evaluations also included the PART (Program Assessment Rating Tool) federal agency evaluation process. The first step in this multiphase effort was the appointment of a committee to develop an evaluation framework that was then used by eight separately appointed evaluation committees to assess NIOSH programs in hearing loss; mining; agri- culture, forestry, and fishing; respiratory diseases; personal protective technology; traumatic injury; construction; and health hazard evaluation. Individual reports were produced by each evaluation committee. At the conclusion of the eight studies, the framework committee held a public workshop in November 2008, “Evaluating NIOSH Programs: Lessons Learned and Next Steps,” where the discussions focused on the experiences gained in the evalua- tion process with NIOSH program and senior staff, members of the NIOSH Board on Scientific Counselors, evaluation committee members, and National Academies’ staff. This report provides the evaluation framework developed, implemented, and refined over the course of four years and eight program evaluations. The evalua- tion framework may prove applicable in evaluating other federal agency research programs. This report has two goals: (1) to summarize the evaluation process and lessons learned in the development and use of the framework and (2) to provide recommendations for future evaluation efforts. EVALUATION FRAMEWORK After examining different approaches to program evaluation, the framework committee decided to define the scope and stages of the evaluation process based on the logic model, a model that is widely used in program evaluation and planning. The logic model organizes the program and its efforts into inputs (e.g., budget, staffing, facilities), activities (e.g., research studies, surveillance, exposure measure- ment), outputs (e.g., reports, publications, conferences, training, patents), and outcomes (e.g., collaborations, policy changes, reductions in injuries and hazard- 

Summary  BOX S-1 Steps in the Evaluation Process   1. Gather appropriate information.   2. Assess external factors.   3. Identify time frame to be evaluated.   4. Identify major occupational health and safety challenges in program area.   5. Analyze program goals and objectives.   6. Identify major program components.   7. Evaluate program inputs, activities, outputs, and outcomes.   8. Determine scores for relevance and impact and provide the rationale.   9. Assess the program’s process for targeting priority research needs and provide the committee’s assessment of emerging issues. 10. Prepare report by using the template provided as a guide. ous exposures). The evaluation framework (outlined in Box S-1) developed for the evaluation of the NIOSH research programs provides criteria for assessing each component of the logic model. As requested by NIOSH, the charge to the National Academies included scoring each program (using integer rating scales of 1 to 5) on the relevance and on the impact of the NIOSH program in improving worker safety and health. To provide guidance on evaluating and scoring these measures after assessing each NIOSH program, the framework committee developed criteria and specific questions to be used in the assessment of each component of the logic model. Assessment of strategic goals and objectives, inputs, activities, and outputs largely defined the rel- evance of the program; the committee examined the adequacy of the inputs and the scope and targeting of the activities and outputs in achieving the program’s goals. Assessment of the intermediate and end outcomes largely defined the program’s impact. The evaluation framework also included specific sets of scoring criteria for rating the program’s relevance and impact on reducing work-related injuries, illnesses, or hazardous exposures (Boxes S-2 and S-3). Setting the metric for program success at demonstrating an impact on end outcomes is laudable. Evaluation and framework committee members give NIOSH a great deal of credit for holding their research programs accountable for real out- comes that affect life and health. Evaluations of research supported by other federal agencies often focus on output productivity and intermediate outcomes and do not hold the agency accountable for real-world impacts. This may be appropriate for the goals of some research programs, but NIOSH’s focus is on applied research and 

 E v a l u a t i n g O c c u p a t i o n a l H e a l t h a n d S a f e t y Research Programs BOX S-2 Scoring Criteria for Relevance 5  =   esearch is in high-priority subject areas and the NIOSH program is significantly R engaged in appropriate transfer activities for completed research projects/reported research results. 4  =   esearch is in high-priority subject areas and the NIOSH program is engaged in ap- R propriate transfer activities for completed research projects/reported research results; or research is in priority subject areas and the NIOSH program is significantly engaged in appropriate transfer activities for completed research projects/reported research results. 3  =   esearch is in high-priority subject areas, but the NIOSH program is not engaged in R appropriate transfer activities; or research is in priority subject areas but the NIOSH program is not significantly engaged in appropriate transfer activities; or research focuses on lesser priorities but the NIOSH program is significantly engaged in ap- propriate transfer activities. 2  =   esearch program is focused on lesser priorities and the NIOSH program is not sig- R nificantly engaged in appropriate transfer activities. 1  =   esearch program is not focused on priorities. R BOX S-3 Scoring Criteria for Impact 5  =   esearch program has made major contribution(s) to worker health and safety on the R basis of end outcomes or well-accepted intermediate outcomes. 4  =   esearch program has made some contributions to end outcomes or well-accepted R intermediate outcomes. 3  =   esearch program activities are ongoing and outputs are produced that are likely to R result in improvements in worker health and safety. Well-accepted outcomes have not been recorded. 2  =   esearch program activities are ongoing and outputs are produced that may result in R new knowledge or technology, but only limited application is expected. Well-accepted outcomes have not been recorded. 1  =   esearch activities and outputs do not result in or are not likely to have any R application. NA  =  Impact cannot be assessed; program is not mature enough. 

Summary  thus the end outcomes are appropriately focused on improving worker safety and health. The evaluation framework acknowledges that actions of those in industry, labor, regulatory entities, and others beyond NIOSH’s control are also necessary for NIOSH programs to have impacts. The challenge for the evaluation committees was to search for data on end outcomes while also thoroughly examining the multiple pathways leading to inter- mediate outcomes and from there to end outcomes. NIOSH staff and the evalua- tion committees used the logic model to document a wide range of intermediate outcomes. Particularly in the absence of good end outcome data, a frequent finding, attention was often focused on the most observable pathways between program activities and end outcomes. The scoring criteria for relevance and impact are provided in Boxes S-2 and S-3. The criteria for assessment of relevance are focused on determining whether the program appropriately set priorities among research needs as well as how engaged the program was in appropriate transfer activities to move research findings into the workplace. The criteria for assessment of impact are focused on completed work and the extent to which the work has directly or indirectly led to improve- ment in worker health and safety. The scoring systems serve as a starting point for descriptive text that explains what the score meant and describes the program’s strengths and limitations that led to the score. Although there was initial concern that the scores would be the only endpoint noted from the evaluation reports, the framework committee is pleased to see the detailed strategic plans and ac- tion plans that are being developed and implemented in NIOSH’s response to the recommendations. DATA FOR THE EVALUATION PROCESS NIOSH provided detailed evidence packages to each evaluation committee with information on the program including program descriptions, staffing levels, program goals and objectives, and details on the program and its accomplishments. Although the evidence package was the core input to each evaluation, it was one of many sources of information that the evaluation committees assessed. Other sources included committee requests to NIOSH for additional information, pre- sentations by NIOSH staff and academic researchers, stakeholder presentations and other input, and in some cases, site visits. Input from organizations and individuals with an interest in the mission of the program was vital to the evaluation process, particularly given the limited surveillance and other end-outcome data related to determining the program’s impact on reducing hazardous occupational exposures and worker injuries and illnesses. 

 E v a l u a t i n g O c c u p a t i o n a l H e a l t h a n d S a f e t y Research Programs IMPROVING THE EVALUATION PROCESS Throughout the course of the eight evaluations, the framework and evalua- tion committees exchanged information on the strengths and limitations of the framework and the evaluation process. A workshop at the conclusion of the eight studies provided an opportunity for NIOSH staff, National Academies’ staff, and evaluation and framework committee members to reflect on ways to improve the process. The report suggests a number of areas in which the evaluation process could be improved, including the provision of more information in the evidence packages on priority setting efforts and on budget and staffing; additional input on extramural research and the connections between the intramural and extra- mural program objectives; ensuring plenty of opportunities for input by external stakeholders and agency staff; considerations regarding the timeline for the evalu- ations; and options for other types of recommendations or ways of categorizing recommendations in the evaluation reports. Increased opportunities, particularly informal opportunities, for NIOSH staff to discuss issues with evaluation commit- tees would be helpful as would increased attention by evaluation committees to some of the more indirect measures of intermediate outcomes. The committee also recognized that logic models are quite linear and focus on readily observable short and medium term outcomes. Evaluation committees, therefore, need to be open to exploring less linear aspects of knowledge development and flow. Concerted efforts need to be made to include the more diffuse contributions of the program to the development of general knowledge and human capital in the field of occupational safety and health. RECOMMENDATIONS Drawing on the lessons learned in developing the evaluation framework and in applying the framework in eight program evaluations, the following recommenda- tions provide the framework committee’s thoughts on moving forward in program evaluation, particularly from a long-range perspective. These recommendations may also be informative for other federal agency program evaluations. Ongoing Evaluation Evaluation of research programs at regular intervals has become the norm, with the trend toward internal management reviews supplemented by periodic evaluation by external parties. Competently done external evaluation removes the unconscious bias of managers with regard to their programs; takes organizational competition out of the assessment; and usually provides new insights while rein- 

Summary  forcing some of what managers already knew, but could not act on. Whatever the composition or structure of external review, the research program should conduct a self-study prior to external review. This is analogous to the self-study an academic program would conduct prior to accreditation. Recommendation 1  Continue Systematic External Evaluations NIOSH should establish a system for periodic external evaluation com- plemented by internal self-assessments on a regular basis. Program or agency-wide evaluations should begin with strong self-evaluation ef- forts that allow the program or agency to assemble and analyze data and act on relevant findings concerning the program’s strengths, weaknesses, and opportunities. Continue to Bolster Research Translation Efforts From the perspective of NIOSH, research transfer or translation can be seen to encompass activities that staff and researchers engage in to increase the likelihood that results of research will be used to improve worker safety and health as well as studies of research translation conducted or funded to increase knowledge about which approaches are most effective. Research transfer is a commendable new em- phasis in the agency, and one that the evaluation committees noted has provided a number of positive intermediate outcomes. The committee believes NIOSH has a role to play not just in demonstrating and testing research-to-practice approaches, but also in documenting and testing its inverse, practice-to-research. Often the most effective research translation occurs through iterative learning. Practitioners can learn from researchers, but it is at least as important for researchers to learn from practitioners so that the new knowledge, practices, programs, and technolo- gies that researchers create are informed by real-world workplace conditions. Recommendation 2  Continue to Build and Improve Research Transla- tion Efforts NIOSH should continue to build and improve its research translation efforts with an emphasis on: • ongoing assessment and improvement of its research translation efforts through formative evaluation processes of listening to those in the workplace (workers and employers) and beyond (product designers, architects, health care providers, etc.), both to identify intervention needs and to provide early feedback regarding re- search translation products to improve the interventions; and 

 E v a l u a t i n g O c c u p a t i o n a l H e a l t h a n d S a f e t y Research Programs • building the capacity to implement and evaluate research translation efforts, both as research-to-practice and as practice-to-research. Enhance Occupational Health and Safety Surveillance The logic model approach to evaluation—used as the basis for the framework presented in this report—relies heavily on surveillance data on health outcomes and workplace exposures to evaluate strategic priorities and assess program im- pact. Surveillance data are also critical program inputs, and the extent to which research programs have considered surveillance data in setting research priorities is an important determinant of program relevance. Although a comprehensive system for tracking fatal occupational injuries in the United States is in place, the current approaches to surveillance of occupational illnesses and nonfatal occupa- tional injuries are fragmented and incomplete, and only limited surveillance data on exposure to hazards are available. Surveillance is a necessity for monitoring long-term progress in reducing hazardous exposures and work-related injuries and illnesses. Recommendation 3  Increase and Improve Surveillance to Benchmark Progress NIOSH should increase and improve surveillance of work-related inju- ries, illnesses, exposures, and working conditions so that information needed to assess program relevance and impact will be available for future evaluations. Enhanced surveillance should prove informative in balancing research priorities. Integrate Evaluations of Extramural and Intramural Research Obtaining the full picture of NIOSH’s work in a specific area of research re- quires examining the relevant intramural and the extramural research. However, the evaluation committees found that the extent to which the intramural and extramural components at NIOSH are currently separated makes it difficult to con- duct such an assessment. Several of the evaluation committees noted a disconnect between the intramural and extramural programs. Although the committee fully supports external scientific review to determine merit for funding investigator- initiated research, the evaluation committees noted that few avenues are currently available by which NIOSH staff can provide intramural input into the development of priorities for extramural research. 

Summary  Recommendation 4  Integrate Evaluations of Intramural and Extramu- ral Research Future evaluations should systematically consider intramural and ex- tramural research activities, in terms of both evaluating the impact and relevance of each type of research and assessing the extent to which in- tramural and extramural research are integrated in strategic planning. BOX S-4 Recommendations Recommendation 1  Continue Systematic External Evaluations NIOSH should establish a system for periodic external evaluation complemented by internal self-assessments on a regular basis. Program or agency-wide evaluations should begin with strong self-evaluation efforts that allow the program or agency to assemble and analyze data and act on relevant findings concerning the program’s strengths, weaknesses, and opportunities. Recommendation 2  Continue to Build and Improve Research Translation Efforts NIOSH should continue to build and improve its research translation efforts with an em- phasis on: •  ongoing assessment and improvement of its research translation efforts through formative evaluation processes of listening to those in the workplace (workers and employers) and beyond (product designers, architects, health care providers, etc.), both to identify intervention needs and to provide early feedback regarding research translation products to improve the interventions; and •  building the capacity to implement and evaluate research translation efforts, both as research-to-practice and as practice-to-research. Recommendation 3  Increase and Improve Surveillance to Benchmark Progress NIOSH should increase and improve surveillance of work-related injuries, illnesses, expo- sures, and working conditions so that information needed to assess program relevance and impact will be available for future evaluations. Enhanced surveillance should prove informative in balancing research priorities. Recommendation 4  Integrate Evaluations of Intramural and Extramural Research Future evaluations should systematically consider both intramural and extramural research activities, in terms of both evaluating the impact and relevance of each type of research and assessing the extent to which intramural and extramural research are integrated in strategic planning.