Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps (2009)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

5 Recommendations for Moving Forward D rawing on the lessons learned in developing the evaluation framework and in applying the framework during eight evaluations, this chapter provides the framework committee’s recommendations on next steps for National Institute for Occupational Safety and Health (NIOSH) program evalu- ations, particularly from a long-range perspective. These recommendations may also be informative for other federal agency program evaluations. ONGOING EVALUATION Evaluation of research programs at regular intervals has become the norm, with the trend toward internal management reviews supplemented by periodic evalu- ation by external parties (see Chapter 2). Competently done external evaluation removes the unconscious bias of managers with regard to their programs; takes or- ganizational competition out of the assessment; and usually provides new insights while reinforcing some of what managers already knew, but could not act on. Findings of program evaluations serve a variety of purposes. Direct and instru- mental uses of the evaluation report include specific modifications to the program through actions such as increasing or decreasing its budget, redefining its program objectives, or more obviously tying its work to the work being done in sister pro- grams. Evaluation reports can also be used by program staff to call attention to research problems or needs, thus raising the salience of an issue within the agency. Often, use occurs that is of a longer range and more diffuse nature. Agency staff 93 

94 E v a l u a t i n g O c c u p a t i o n a l H e a l t h a n d S a f e t y Research Programs and researchers may, by virtue of reading an evaluation report, gain an appreciation for the external factors that have shaped a research agenda, or they may see com- mon threads across a set of evaluations that identify the need for new agency-wide strategic objectives. These less direct, longer term uses of evaluation products can function as a way of gradually infusing an organization with new ideas from the outside to the inside. Furthermore, external stakeholders or advocacy groups can use an evaluation report to call attention to needed research and priorities. The committee commends NIOSH for undertaking the recent external review of a series of their programs by the National Academies, and in particular for requesting that the focus of the evaluations be on the relevance and impact in re- ducing work-related illnesses, injuries, or hazardous exposures. Many evaluations, particularly of research programs, stop with assessments of outputs, such as the number of peer-reviewed publications, and do not take into account the research transfer steps, the external factors that influence program activities and outcomes, or the need to use the program’s impact on intermediate or end outcomes as the metric for a successful program. In an effort to continue the forward momentum, the committee recommends that NIOSH establish a system for periodic external evaluation of its programs. The system does not need to be modeled on the evaluations just carried out by the National Academies; indeed, many options are available, and NIOSH may want to incorporate elements of several in its overall approach. For example, the following types of evaluation approaches may be considered: • From time to time, an outside look at agency-wide processes is necessary. • Regular external review at the broad program level should be comple- mented with internal self-assessments on a more regular basis. • Formative evaluations at the program design or implementation stages can be useful for new programs. • Special studies of the long-term impacts of the program (such as the de- velopment of human capital) could be undertaken. Whatever the composition or structure of external review, the research pro- gram should conduct a self-study prior to external review. This is analogous to the self-study that an academic program would conduct prior to accreditation. Much of the evaluation framework presented in Chapter 3, including the scoring system, may be useful in self-evaluations. These approaches might emphasize different criteria for NIOSH effectiveness, balancing the appropriate focus on long-term impact in the workplace with at- tention to other public benefits. As noted above, in addition to research program evaluations, external evaluations of agency-wide initiatives or processes can be use- 

R e c o m m e n dat i o n s f o r M o v i n g F o r w a r d 95 ful. The agency is more than the sum of its parts, and a broad review can provide perspective not gained in a cross-section of assessments of individual programs. Further, an emphasis on how the program contributes to the overall NIOSH pro- gram could be added, as the current framework is focused on issues specific to the program and its goals. Recommendation 1  Continue Systematic External Evaluations NIOSH should establish a system for periodic external evaluation com- plemented by internal self-assessments on a regular basis. Program or agency-wide evaluations should begin with strong self-evaluation ef- forts that allow the program or agency to assemble and analyze data and act on relevant findings concerning the program’s strengths, weaknesses, and opportunities. CONTINUE TO BOLSTER RESEARCH TRANSLATION EFFORTS From the perspective of NIOSH, research transfer or translation can be seen to encompass activities that staff and researchers engage in to increase the likelihood that the results of research will be used to improve worker safety and health as well as studies of research translation conducted or funded to increase knowledge about which approaches are most effective. Research transfer is a commendable new emphasis in the agency, and one that the evaluation committees noted has provided a number of positive intermediate outcomes. NIOSH refers to research translation as “r2p”—Research to Practice. However, the committee noted that the focus of NIOSH’s efforts on research-to-practice tends to be on interventions, demonstration projects, and control technology rather than the results of health effects research. NIOSH should expand its portfolio of r2p efforts to formulate an approach to translation of health effects research. The research observation of a new or more fully understood association between exposures and illness can lead to changes in use of existing technology or control programs and requires research translation efforts at a level similar to the develop- ment and application of new technology in the workplace to reduce exposure. Much remains to be learned about how to improve the likelihood that research translation efforts (e.g., site visits, demonstrations, partnerships) and products (e.g., websites, newsletters, CD-ROMs, training materials) will positively impact worksites. The systematic gathering and interpretation of early-stage feedback about research-to-practice processes and products prior to release or deployment is a form of research that focuses on the intended user, the user’s work context, and how improvements can be made to be most compatible with current work practices 

96 E v a l u a t i n g O c c u p a t i o n a l H e a l t h a n d S a f e t y Research Programs and conditions. These types of preintervention research efforts can be conducted with a small number of potential adopters and at little cost as long as respondents are somewhat representative of the larger worksite population that will be targeted. Feedback can then be used to identify which information channels are preferred by the target audience, which messages and images best communicate the positive attributes of the research results in question, which types of spokespeople are most effective at producing interest and inquiries by potential adopters, and how NIOSH innovations themselves can be redesigned to be more compatible with real-life workplace conditions and constraints. Conducting formative studies such as these can come to constitute a key NIOSH r2p skill set, although capacity building would be warranted. Further development of these skills within NIOSH may become an example of how social science can contribute to the agency’s worker protection and workplace safety mission. NIOSH is also warranted in soliciting and funding behavioral research about the dissemination of safety and occupational health outputs and intervention pro- grams, the barriers and facilitators that affect adoption of effective practices and processes in the workplace, and the study of effective implementation of NIOSH research results within organizations. The evaluation committee that assessed the NIOSH Agriculture, Forestry, and Fishing (AFF) Research Program made the following recommendation: “NIOSH should conduct research on the science of knowledge diffusion to identify effective methods for AFF research-to-practice programs” (NRC and IOM, 2008a, p. 12). Continued contributions by NIOSH to research on improving the effectiveness of translation efforts will ensure the consideration of the dynamics that characterize workplace safety and occupational health. The committee believes NIOSH has a role to play not just in demonstrating and testing research-to-practice approaches, but also in documenting and testing its inverse, practice-to-research. Often, the most effective research translation occurs through iterative learning. Practitioners can learn from researchers, but it is at least as important for researchers to learn from practitioners so that the new knowledge, practices, programs, and technolo- gies that researchers create are informed by real-world workplace conditions. Recommendation 2  Continue to Build and Improve Research Transla- tion Efforts NIOSH should continue to build and improve its research translation efforts with an emphasis on: • ongoing assessment and improvement of its research translation  efforts through formative evaluation processes of listening to those in the workplace (workers and employers) and beyond (product designers, architects, health care providers, etc.), both to identify 

R e c o m m e n dat i o n s f o r M o v i n g F o r w a r d 97 intervention needs and to provide early feedback regarding research translation products to improve the interventions; and • building the capacity to implement and evaluate research trans-  lation efforts, both as research-to-practice and as practice-to- research. ENHANCE OCCUPATIONAL HEALTH AND SAFETY SURVEILLANCE The logic model approach to evaluation of NIOSH programs—used as the basis for the framework presented in this report—relies heavily on surveillance data on health outcomes and workplace exposures to evaluate strategic priorities and assess program impact. Surveillance data are also critical program inputs, and the extent to which research programs have considered surveillance data in setting research priorities is an important determinant of program relevance. The impor- tance of surveillance extends far beyond research; it is also critical for effectively targeting and evaluating intervention activities at the national, state, and local lev- els and for the strategic planning needed to develop, implement, and assess these interventions. Although a comprehensive system for tracking fatal occupational injuries in the United States is in place, the current approaches to surveillance of occupational illnesses and nonfatal occupational injuries are fragmented and in- complete (Azaroff et al., 2002), and only limited surveillance data on exposure to hazards are available. All eight evaluation committee reviews of NIOSH programs completed to date identified the need for better surveillance, and seven of the eight evaluation reports included specific recommendations calling for improved surveil- lance and additional surveillance research (IOM and NRC, 2006, 2008, 2009; NRC and IOM, 2007, 2008a,b, 2009a,b). NIOSH includes surveillance in its mission and is engaged in a number of surveillance activities, both intramural and in collaboration with state partners, using a variety of data sources. Although NIOSH has made many important con- tributions to surveillance, the committee that evaluated the NIOSH Traumatic Injury Research Program noted that “these projects do not appear to be a part of a coordinated interagency strategy to improve national surveillance of traumatic nonfatal occupational injuries” (IOM and NRC, 2009, p. 52). The Survey of Occupational Injuries and Illnesses conducted by the Bureau of Labor Statistics (BLS), the official source of statistics on nonfatal, work-related in- juries and illnesses, is a valuable resource, but has many limitations. This ­employer- based data source currently excludes approximately 22 percent of the workforce and fails to capture most occupational illnesses (Leigh et al., 2004). Nonfatal injuries may be substantially underreported (Azaroff et al., 2002; Welch et al., 2007; Boden 

98 E v a l u a t i n g O c c u p a t i o n a l H e a l t h a n d S a f e t y Research Programs and Ozonoff, 2008), and there is concern that underreporting may vary by type of injury and worker or workplace characteristics. Important information about em- ployer health and safety practices that could identify correlations between employer practices and good health and safety records are not collected. It would be unfor- tunate if the recommendations for improved surveillance in individual NIOSH program reviews led to fragmented surveillance activities. The identification of the need for improved surveillance across NIOSH research programs underscores the need for a comprehensive and coordinated interagency plan for surveillance of work-related injuries, illnesses, and hazards. NIOSH, which has epidemiologic capacity and experience working with a wide range of health data sources, could play an important leadership role in coordi- nating efforts of relevant federal agencies, including but not limited to BLS, the Occupational Safety and Health Administration, Mine Safety and Health Admin- istration, National Center for Health Statistics, and Department of Transportation. States, which have the legal authority to require disease reporting and to collect health data, should be included as integral partners in developing and implement- ing a comprehensive surveillance plan. Such a plan should go beyond improve- ments in the existing employer-based data sources to include nonemployer-based data sources such as hospital and other medical data systems and population-based surveys such as the National Health Interview Survey and the Behavioral Risk Fac- tor Surveillance System. The potentially highly important role of electronic health records and new integrated data systems in surveillance efforts should be addressed. The plan should also include a surveillance research agenda and a proposed mecha- nism for ongoing interagency communication and coordination. Surveillance is a necessity for monitoring long-term progress in reducing haz- ardous exposures and work-related injuries and illnesses. Recommendation 3  Increase and Improve Surveillance to Benchmark Progress NIOSH should increase and improve surveillance of work-related inju- ries, illnesses, exposures, and working conditions so that information needed to assess program relevance and impact will be available for future evaluations. Enhanced surveillance should prove informative in balancing research priorities. INTEGRATE EVALUATIONS OF EXTRAMURAL AND INTRAMURAL RESEARCH Obtaining the full picture of NIOSH’s work in a specific area of research re- quires examining the relevant intramural and extramural research. However, the evaluation committees found that the extent to which the intramural and extramu- 

R e c o m m e n dat i o n s f o r M o v i n g F o r w a r d 99 ral components at NIOSH are currently separated makes it difficult to conduct such an assessment. Several of the evaluation committees noted a disconnect between the intramural and extramural programs. For example, the evaluation report on the NIOSH Hearing Loss Research Program stated, “In some cases, however, intramural researchers have not made themselves aware of relevant extramural research, which may have resulted in limited opportunities for effective collaboration” (IOM and NRC, 2006, pp. 122–123). Similarly, the report on the Traumatic Injury Research Program noted in a recommendation, “NIOSH should review its practices and take steps to improve the opportunities for intramural and extramural research- ers, including state occupational public health programs, to communicate and collaborate without excessively directing extramural research to the detriment of scientific creativity” (IOM and NRC, 2009, p. 14). Although the framework committee fully supports external scientific review to determine merit for funding investigator-initiated research, the evaluation commit- tees noted that few avenues are currently available by which NIOSH staff can pro- vide intramural input into the development of priorities for extramural research. Larger research agencies, such as the National Institutes of Health, fund full-time staff members to interface between the intramural and extramural programs, but this may not be a feasible option for NIOSH due to funding constraints. The evaluation report on the Personal Protective Technology (PPT) Program noted that improvements are being made in this area: “PPT Program staff members have also reported increased opportunities for dialogue with the NIOSH Office of Extramu- ral Programs in the past year regarding priorities for funding. The committee urges NIOSH to consider ways in which the PPT Program could have greater input into the extramural priority process at NIOSH and increased participation in drafting requests for grant applications” (IOM and NRC, 2008, p. 122). Future evaluation efforts need to focus on examining the relationship between intramural and extramural research in strategic planning for a cohesive research program that addresses program goals and the overall NIOSH mission. Recommendation 4  Integrate Evaluations of Intramural and Extramu- ral Research Future evaluations should systematically consider intramural and ex- tramural research activities, in terms of both evaluating the impact and relevance of each type of research and assessing the extent to which in- tramural and extramural research are integrated in strategic planning. ON THE HORIZON Over the course of its work during the past four years, the framework com- mittee has seen continuing developments in occupational safety and health that 

100 E v a l u a t i n g O c c u p a t i o n a l H e a l t h a n d S a f e t y Research Programs will have significant impacts in the decades ahead. The committee concludes this report with an overview of a few of the issues that may need to be considered in future evaluations of progress made in improving worker safety and health: • Defining “the workplace”: Traditional workplaces and job stability are rapidly changing. For example, some forms of work are increasingly con- ducted outside the physical workplace. Internet accessibility permits work to occur in homes and other locations through a variety of telecommuting arrangements. In addition, the mobility of some individual workers’ jobs prevents traditional approaches to monitoring these workers and their work risks in a reliable fashion. Appropriate surveillance for workplace in- juries, illnesses, exposures, and risks consequently has become much more complex and presents many new challenges for assessing and preventing work-related problems. • Occupational health and public health: The worksite is increasingly rec- ognized as a venue to improve population health. New emphasis is be- ing placed on changes in workplace policies and practices that promote healthy lifestyle choices and help prevent chronic diseases such as diabetes and stroke. As reflected in the NIOSH WorkLife Initiative, new integrated approaches in worksites that address both occupational and nonoccupa- tional risks are needed and will broaden considerations when evaluating occupational health and safety programs. • Ethics: Ethical issues relevant to occupational safety and health research are expanding beyond the bounds of institutional review boards and include issues of equity in addressing underserved and vulnerable populations and attention to working conditions in the small-business sector. • Worker demographics: The changing demographics of the workforce will need to be considered in future evaluations. For example, employment in the services sector continues to grow, the U.S. labor force continues to age, and the workforce is becoming more racially and ethnically diverse. • Economic challenges: Challenging economic times will likely bring as yet unknown changes to the workplace. A forward look at these changes will be necessary to keep pace with the needs for safety and health of the U.S. and global workforce. REFERENCES Azaroff, L. S., C. Levenstein, and D. H. Wegman. 2002. Occupational injury and illness surveillance: Conceptual filters explain underreporting. American Journal of Public Health 92(9):1421–1429. Boden, L., and A. Ozonoff. 2008. Capture-recapture estimates of nonfatal workplace injuries and illnesses. Annals of Epidemiology 18(6):500–506. 

R e c o m m e n dat i o n s f o r M o v i n g F o r w a r d 101 IOM and NRC (Institute of Medicine and National Research Council). 2006. Hearing loss research at NIOSH. Committee to Review the NIOSH Hearing Loss Research Program. Rpt. No. 1, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. IOM and NRC. 2008. The personal protective technology program at NIOSH. Committee to Review the NIOSH Personal Protective Technology Program. Rpt. No. 5, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. IOM and NRC. 2009. Traumatic injury research at NIOSH. Committee to Review the NIOSH Trau- matic Injury Research Program. Rpt. No. 6, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. Leigh, J. P., J. Marcin, and T. R. Miller. 2004. An estimate of the U.S. government’s undercount of non- fatal occupational injuries. Journal of Occupational and Environmental Medicine 46(1):10–18. NRC and IOM. 2007. Mining safety and health research at NIOSH. Committee to Review the NIOSH Mining Safety and Health Research Program. Rpt. No. 2, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Acad- emies Press. NRC and IOM. 2008a. Agriculture, forestry, and fishing research at NIOSH. Committee to Review the NIOSH Agriculture, Forestry, and Fishing Research Program. Rpt. No. 3, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. NRC and IOM. 2008b. Respiratory diseases research at NIOSH. Committee to Review the NIOSH Re- spiratory Diseases Research Program. Rpt. No. 4, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. NRC and IOM. 2009a. The health hazard evaluation program at NIOSH. Committee to Review the NIOSH Health Hazard Evaluation Program. Rpt. No. 7, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Acad- emies Press. NRC and IOM. 2009b. Construction research at NIOSH. Committee to Review the NIOSH Construc- tion Research Program. Rpt. No. 8, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. Welch, L. S., X. Dong, F. Carre, and K. Ringen. 2007. Is the apparent decrease in injury and illness rates in construction the result of changes in reporting? International Journal of Occupational and Environmental Health 13(1):39–45.