Summary

ABSTRACT The goal of research conducted by the National Institute for Occupational Safety and Health (NIOSH) is protecting and improving the health and safety of workers. The Institute of Medicine and the National Research Council conducted a series of evaluations of NIOSH research programs that assessed the relevance and impact of NIOSH’s work on meeting this goal. Eight NIOSH programs were evaluated based on a common framework. This report provides the lessons learned in the evaluation process, a revised evaluation framework, and the following recommendations for evaluating occupational health and safety research programs: continue systematic external evaluations, bolster research translation efforts, enhance occupational health and safety surveillance, and integrate evaluations of intramural and extramural research.

Preventing tractor rollovers on farmers, protecting construction workers from falls, improving the health of miners in dusty environments, reducing back injuries in nursing aides, developing substitute materials to eliminate hazardous chemical exposures, and designing work conditions to reduce fatigue and stress are among the issues critical to improving worker safety and health. In 2007, 5,657 fatal work injuries occurred in the United States, along with an estimated 4 million nonfatal occupational injuries and illnesses among workers in private industry. In addition to the human suffering involved, these statistics are also associated with high economic costs. One estimate puts the costs of occupational injury



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Summary ABSTRACT The goal of research conducted by the National Institute for Occupational Safety and Health (NIOSH) is protecting and improving the health and safety of workers. The Institute of Medicine and the National Re- search Council conducted a series of evaluations of NIOSH research programs that assessed the relevance and impact of NIOSH’s work on meeting this goal. Eight NIOSH programs were evaluated based on a common framework. This report provides the lessons learned in the evaluation process, a revised evaluation framework, and the following recommendations for evaluating occupational health and safety research programs: continue systematic ex- ternal evaluations, bolster research translation efforts, enhance occupational health and safety surveillance, and integrate evaluations of intramural and extramural research. P reventing tractor rollovers on farmers, protecting construction workers from falls, improving the health of miners in dusty environments, reducing back injuries in nursing aides, developing substitute materials to eliminate haz- ardous chemical exposures, and designing work conditions to reduce fatigue and stress are among the issues critical to improving worker safety and health. In 2007, 5,657 fatal work injuries occurred in the United States, along with an estimated 4 million nonfatal occupational injuries and illnesses among workers in private industry. In addition to the human suffering involved, these statistics are also asso- ciated with high economic costs. One estimate puts the costs of occupational injury 

OCR for page 1
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s and illnesses for all industries for 2005 at more than $160 billion. The continued attention to further improve occupational health and safety through research is not only fully warranted but such research requires critical evaluation for its relevance and impact. The core mission of the National Institute for Occupational Safety and Health (NIOSH) is to conduct research to improve and protect the health and safety of workers. In September 2004, NIOSH contracted with The National Academies to con- duct a series of evaluations of individual NIOSH research programs. This set of in- dependent evaluations focused on the relevance and impact of each of eight NIOSH programs on reducing work-related injuries, illnesses, and hazardous exposures. From the outset of the evaluations, NIOSH leadership established the primary goal as program improvement but the context for the evaluations also included the PART (Program Assessment Rating Tool) federal agency evaluation process. The first step in this multiphase effort was the appointment of a committee to develop an evaluation framework that was then used by eight separately appointed evaluation committees to assess NIOSH programs in hearing loss; mining; agri- culture, forestry, and fishing; respiratory diseases; personal protective technology; traumatic injury; construction; and health hazard evaluation. Individual reports were produced by each evaluation committee. At the conclusion of the eight studies, the framework committee held a public workshop in November 2008, “Evaluating NIOSH Programs: Lessons Learned and Next Steps,” where the discussions focused on the experiences gained in the evalua- tion process with NIOSH program and senior staff, members of the NIOSH Board on Scientific Counselors, evaluation committee members, and National Academies’ staff. This report provides the evaluation framework developed, implemented, and refined over the course of four years and eight program evaluations. The evalua- tion framework may prove applicable in evaluating other federal agency research programs. This report has two goals: (1) to summarize the evaluation process and lessons learned in the development and use of the framework and (2) to provide recommendations for future evaluation efforts. EVALUATION FRAMEWORK After examining different approaches to program evaluation, the framework committee decided to define the scope and stages of the evaluation process based on the logic model, a model that is widely used in program evaluation and planning. The logic model organizes the program and its efforts into inputs (e.g., budget, staffing, facilities), activities (e.g., research studies, surveillance, exposure measure- ment), outputs (e.g., reports, publications, conferences, training, patents), and outcomes (e.g., collaborations, policy changes, reductions in injuries and hazard-

OCR for page 1
summaRy  BOX S-1 Steps in the Evaluation Process   1. Gather appropriate information.   2. Assess external factors.   3. Identify time frame to be evaluated.   4. Identify major occupational health and safety challenges in program area.   5. Analyze program goals and objectives.   6. Identify major program components.   7. Evaluate program inputs, activities, outputs, and outcomes.   8. Determine scores for relevance and impact and provide the rationale.   9. A   ssess  the  program’s  process  for  targeting  priority  research  needs  and  provide  the  committee’s assessment of emerging issues. 10.  Prepare report by using the template provided as a guide. ous exposures). The evaluation framework (outlined in Box S-1) developed for the evaluation of the NIOSH research programs provides criteria for assessing each component of the logic model. As requested by NIOSH, the charge to the National Academies included scoring each program (using integer rating scales of 1 to 5) on the relevance and on the impact of the NIOSH program in improving worker safety and health. To provide guidance on evaluating and scoring these measures after assessing each NIOSH program, the framework committee developed criteria and specific questions to be used in the assessment of each component of the logic model. Assessment of strategic goals and objectives, inputs, activities, and outputs largely defined the rel- evance of the program; the committee examined the adequacy of the inputs and the scope and targeting of the activities and outputs in achieving the program’s goals. Assessment of the intermediate and end outcomes largely defined the program’s impact. The evaluation framework also included specific sets of scoring criteria for rating the program’s relevance and impact on reducing work-related injuries, illnesses, or hazardous exposures (Boxes S-2 and S-3). Setting the metric for program success at demonstrating an impact on end outcomes is laudable. Evaluation and framework committee members give NIOSH a great deal of credit for holding their research programs accountable for real out- comes that affect life and health. Evaluations of research supported by other federal agencies often focus on output productivity and intermediate outcomes and do not hold the agency accountable for real-world impacts. This may be appropriate for the goals of some research programs, but NIOSH’s focus is on applied research and

OCR for page 1
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s BOX S-2 Scoring Criteria for Relevance 5  =    esearch  is  in  high-priority  subject  areas  and  the  NIOSH  program  is  significantly  R engaged  in  appropriate  transfer  activities  for  completed  research  projects/reported  research results. 4  =    esearch is in high-priority subject areas and the NIOSH program is engaged in ap- R propriate transfer activities for completed research projects/reported research results;  or research is in priority subject areas and the NIOSH program is significantly engaged  in  appropriate  transfer  activities  for  completed  research  projects/reported  research  results. 3  =    esearch is in high-priority subject areas, but the NIOSH program is not engaged in  R appropriate transfer activities; or research is in priority subject areas but the NIOSH  program  is  not  significantly  engaged  in  appropriate  transfer  activities;  or  research  focuses  on  lesser  priorities  but  the  NIOSH  program  is  significantly  engaged  in  ap- propriate transfer activities. 2  =    esearch program is focused on lesser priorities and the NIOSH program is not sig- R nificantly engaged in appropriate transfer activities.  1  =    esearch program is not focused on priorities. R BOX S-3 Scoring Criteria for Impact 5  =    esearch program has made major contribution(s) to worker health and safety on the  R basis of end outcomes or well-accepted intermediate outcomes. 4  =    esearch program has made some contributions to end outcomes or well-accepted  R intermediate outcomes. 3  =    esearch program activities are ongoing and outputs are produced that are likely to  R result in improvements in worker health and safety. Well-accepted outcomes have not  been recorded. 2  =    esearch program activities are ongoing and outputs are produced that may result in  R new knowledge or technology, but only limited application is expected. Well-accepted  outcomes have not been recorded. 1  =    esearch  activities  and  outputs  do  not  result  in  or  are  not  likely  to  have  any  R application. NA  =  Impact cannot be assessed; program is not mature enough.

OCR for page 1
summaRy  thus the end outcomes are appropriately focused on improving worker safety and health. The evaluation framework acknowledges that actions of those in industry, labor, regulatory entities, and others beyond NIOSH’s control are also necessary for NIOSH programs to have impacts. The challenge for the evaluation committees was to search for data on end outcomes while also thoroughly examining the multiple pathways leading to inter- mediate outcomes and from there to end outcomes. NIOSH staff and the evalua- tion committees used the logic model to document a wide range of intermediate outcomes. Particularly in the absence of good end outcome data, a frequent finding, attention was often focused on the most observable pathways between program activities and end outcomes. The scoring criteria for relevance and impact are provided in Boxes S-2 and S-3. The criteria for assessment of relevance are focused on determining whether the program appropriately set priorities among research needs as well as how engaged the program was in appropriate transfer activities to move research findings into the workplace. The criteria for assessment of impact are focused on completed work and the extent to which the work has directly or indirectly led to improve- ment in worker health and safety. The scoring systems serve as a starting point for descriptive text that explains what the score meant and describes the program’s strengths and limitations that led to the score. Although there was initial concern that the scores would be the only endpoint noted from the evaluation reports, the framework committee is pleased to see the detailed strategic plans and ac- tion plans that are being developed and implemented in NIOSH’s response to the recommendations. DATA FOR THE EVALUATION PROCESS NIOSH provided detailed evidence packages to each evaluation committee with information on the program including program descriptions, staffing levels, program goals and objectives, and details on the program and its accomplishments. Although the evidence package was the core input to each evaluation, it was one of many sources of information that the evaluation committees assessed. Other sources included committee requests to NIOSH for additional information, pre- sentations by NIOSH staff and academic researchers, stakeholder presentations and other input, and in some cases, site visits. Input from organizations and individuals with an interest in the mission of the program was vital to the evaluation process, particularly given the limited surveillance and other end-outcome data related to determining the program’s impact on reducing hazardous occupational exposures and worker injuries and illnesses.

OCR for page 1
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s IMPROVING THE EVALUATION PROCESS Throughout the course of the eight evaluations, the framework and evalua- tion committees exchanged information on the strengths and limitations of the framework and the evaluation process. A workshop at the conclusion of the eight studies provided an opportunity for NIOSH staff, National Academies’ staff, and evaluation and framework committee members to reflect on ways to improve the process. The report suggests a number of areas in which the evaluation process could be improved, including the provision of more information in the evidence packages on priority setting efforts and on budget and staffing; additional input on extramural research and the connections between the intramural and extra- mural program objectives; ensuring plenty of opportunities for input by external stakeholders and agency staff; considerations regarding the timeline for the evalu- ations; and options for other types of recommendations or ways of categorizing recommendations in the evaluation reports. Increased opportunities, particularly informal opportunities, for NIOSH staff to discuss issues with evaluation commit- tees would be helpful as would increased attention by evaluation committees to some of the more indirect measures of intermediate outcomes. The committee also recognized that logic models are quite linear and focus on readily observable short and medium term outcomes. Evaluation committees, therefore, need to be open to exploring less linear aspects of knowledge development and flow. Concerted efforts need to be made to include the more diffuse contributions of the program to the development of general knowledge and human capital in the field of occupational safety and health. RECOMMENDATIONS Drawing on the lessons learned in developing the evaluation framework and in applying the framework in eight program evaluations, the following recommenda- tions provide the framework committee’s thoughts on moving forward in program evaluation, particularly from a long-range perspective. These recommendations may also be informative for other federal agency program evaluations. Ongoing Evaluation Evaluation of research programs at regular intervals has become the norm, with the trend toward internal management reviews supplemented by periodic evaluation by external parties. Competently done external evaluation removes the unconscious bias of managers with regard to their programs; takes organizational competition out of the assessment; and usually provides new insights while rein-

OCR for page 1
summaRy  forcing some of what managers already knew, but could not act on. Whatever the composition or structure of external review, the research program should conduct a self-study prior to external review. This is analogous to the self-study an academic program would conduct prior to accreditation. Recommendation 1 Continue Systematic External Evaluations NIOSH should establish a system for periodic external evaluation com- plemented by internal self-assessments on a regular basis. Program or agency-wide evaluations should begin with strong self-evaluation ef- forts that allow the program or agency to assemble and analyze data and act on relevant findings concerning the program’s strengths, weaknesses, and opportunities. Continue to Bolster Research Translation Efforts From the perspective of NIOSH, research transfer or translation can be seen to encompass activities that staff and researchers engage in to increase the likelihood that results of research will be used to improve worker safety and health as well as studies of research translation conducted or funded to increase knowledge about which approaches are most effective. Research transfer is a commendable new em- phasis in the agency, and one that the evaluation committees noted has provided a number of positive intermediate outcomes. The committee believes NIOSH has a role to play not just in demonstrating and testing research-to-practice approaches, but also in documenting and testing its inverse, practice-to-research. Often the most effective research translation occurs through iterative learning. Practitioners can learn from researchers, but it is at least as important for researchers to learn from practitioners so that the new knowledge, practices, programs, and technolo- gies that researchers create are informed by real-world workplace conditions. Recommendation 2 Continue to Build and Improve Research Transla- tion Efforts NIOSH should continue to build and improve its research translation efforts with an emphasis on: • ongoing assessment and improvement of its research translation efforts through formative evaluation processes of listening to those in the workplace (workers and employers) and beyond (product designers, architects, health care providers, etc.), both to identify intervention needs and to provide early feedback regarding re- search translation products to improve the interventions; and

OCR for page 1
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s • building the capacity to implement and evaluate research translation efforts, both as research-to-practice and as practice-to-research. Enhance Occupational Health and Safety Surveillance The logic model approach to evaluation—used as the basis for the framework presented in this report—relies heavily on surveillance data on health outcomes and workplace exposures to evaluate strategic priorities and assess program im- pact. Surveillance data are also critical program inputs, and the extent to which research programs have considered surveillance data in setting research priorities is an important determinant of program relevance. Although a comprehensive system for tracking fatal occupational injuries in the United States is in place, the current approaches to surveillance of occupational illnesses and nonfatal occupa- tional injuries are fragmented and incomplete, and only limited surveillance data on exposure to hazards are available. Surveillance is a necessity for monitoring long-term progress in reducing hazardous exposures and work-related injuries and illnesses. Recommendation 3 Increase and Improve Surveillance to Benchmark Progress NIOSH should increase and improve surveillance of work-related inju- ries, illnesses, exposures, and working conditions so that information needed to assess program relevance and impact will be available for future evaluations. Enhanced surveillance should prove informative in balancing research priorities. Integrate Evaluations of Extramural and Intramural Research Obtaining the full picture of NIOSH’s work in a specific area of research re- quires examining the relevant intramural and the extramural research. However, the evaluation committees found that the extent to which the intramural and extramural components at NIOSH are currently separated makes it difficult to con- duct such an assessment. Several of the evaluation committees noted a disconnect between the intramural and extramural programs. Although the committee fully supports external scientific review to determine merit for funding investigator- initiated research, the evaluation committees noted that few avenues are currently available by which NIOSH staff can provide intramural input into the development of priorities for extramural research.

OCR for page 1
summaRy  Recommendation 4 Integrate Evaluations of Intramural and Extramu- ral Research Future evaluations should systematically consider intramural and ex- tramural research activities, in terms of both evaluating the impact and relevance of each type of research and assessing the extent to which in- tramural and extramural research are integrated in strategic planning. BOX S-4 Recommendations Recommendation 1 Continue Systematic External Evaluations NIOSH should establish a system for periodic external evaluation complemented by internal  self-assessments on a regular basis. Program or agency-wide evaluations should begin with  strong  self-evaluation  efforts  that  allow  the  program  or  agency  to  assemble  and  analyze  data  and  act  on  relevant  findings  concerning  the  program’s  strengths,  weaknesses,  and  opportunities.  Recommendation 2 Continue to Build and Improve Research Translation Efforts NIOSH should continue to build and improve its research translation efforts with an em- phasis on:   •    ngoing assessment and improvement of its research translation efforts through  o formative evaluation processes of listening to those in the workplace (workers and  employers) and beyond (product designers, architects, health care providers, etc.),  both to identify intervention needs and to provide early feedback regarding research  translation products to improve the interventions; and   •    uilding the capacity to implement and evaluate research translation efforts, both  b as research-to-practice and as practice-to-research.  Recommendation 3 Increase and Improve Surveillance to Benchmark Progress NIOSH should increase and improve surveillance of work-related injuries, illnesses, expo- sures,  and  working  conditions  so  that  information  needed  to  assess  program  relevance  and  impact  will  be  available  for  future  evaluations.  Enhanced  surveillance  should  prove  informative in balancing research priorities. Recommendation 4 Integrate Evaluations of Intramural and Extramural Research Future evaluations should systematically consider both intramural and extramural research  activities,  in  terms  of  both  evaluating  the  impact  and  relevance  of  each  type  of  research  and assessing  the  extent  to  which intramural  and  extramural  research  are  integrated  in  strategic planning.

OCR for page 1