3
Evaluation Framework

As discussed throughout this report, an evaluation framework was developed to guide the NIOSH program evaluations and provide a set of criteria to be used in scoring the relevance and impact of each program’s efforts in reducing work-related hazardous exposures, illnesses, and injuries. The National Academies Committee for the Review of NIOSH Research Programs (referred to as the framework committee) developed an initial evaluation framework that was honed and refined to improve its utility, clarity, and emphasis as the eight evaluations proceeded. The framework presented in this chapter differs slightly from the versions that were used by the evaluation committees.1 Insights gained in the evaluation of the eight NIOSH programs (see Chapter 4) are included in this version of the framework, which is provided for consideration in future program evaluations. In addition to evaluating NIOSH programs, this analytic framework and approach may be applicable to the evaluation of other federal agency research programs or research programs in other organizations.

In conducting its evaluation, each committee was asked to determine whether the NIOSH program was undertaking high-priority, relevant research and transfer activities (relevance) and whether these efforts are improving health and safety in the workplace (impact). The evaluation committee was also tasked with (1) rating

1

The evaluation framework document used by the individual evaluation committee is provided as an appendix in each of the evaluation committee reports (IOM and NRC, 2006, 2008, 2009; NRC and IOM, 2007, 2008a,b, 2009a,b).



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 37
3 Evaluation Framework A s discussed throughout this report, an evaluation framework was developed to guide the NIOSH program evaluations and provide a set of criteria to be used in scoring the relevance and impact of each program’s efforts in reducing work-related hazardous exposures, illnesses, and injuries. The National Academies Committee for the Review of NIOSH Research Programs (referred to as the framework committee) developed an initial evaluation framework that was honed and refined to improve its utility, clarity, and emphasis as the eight evalua- tions proceeded. The framework presented in this chapter differs slightly from the versions that were used by the evaluation committees.1 Insights gained in the evalu- ation of the eight NIOSH programs (see Chapter 4) are included in this version of the framework, which is provided for consideration in future program evaluations. In addition to evaluating NIOSH programs, this analytic framework and approach may be applicable to the evaluation of other federal agency research programs or research programs in other organizations. In conducting its evaluation, each committee was asked to determine whether the NIOSH program was undertaking high-priority, relevant research and transfer activities (relevance) and whether these efforts are improving health and safety in the workplace (impact). The evaluation committee was also tasked with (1) rating 1 The evaluation framework document used by the individual evaluation committee is provided as an appendix in each of the evaluation committee reports (IOM and NRC, 2006, 2008, 2009; NRC and IOM, 2007, 2008a,b, 2009a,b). 

OCR for page 37
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s both the relevance and the impact of the NIOSH program using 1–5 integer scales, and (2) providing input about emerging areas of research and recommendations for program improvement. OVERVIEW OF THE EVALUATION FRAMEWORK After examining different approaches to program evaluation (see Chapter 2), the framework committee decided to define the scope and stages of the evaluation process based on the logic model (Williams et al., 2009). The resulting evaluation framework described in this chapter breaks the logic models developed by NIOSH (Figure 3-1) into discrete program components to be assessed by each evaluation committee. Criteria for evaluation of each component of the framework are de- tailed below. In the evaluation framework (overview provided in Figure 3-2), the assessment of strategic goals and objectives, inputs, activities, and outputs (B to E) largely define the relevance of the program, while assessment of intermediate and end outcomes (F and G) largely define the program impact. The following major components of each NIOSH program were assessed by the evaluation committees: • Major occupational safety and health challenges in the program area. • Goals and objectives as defined by NIOSH. • Inputs (e.g., budget; staff; facilities; and input from the program’s re - search management, the NIOSH Board of Scientific Counselors, and stakeholders). • Activities (efforts by NIOSH staff, contractors, and grantees; e.g., surveil- lance of injury, illness, and hazards; exposure assessment research; health- effects research; injury-risk factor research; intervention research; health services research; and technology transfer activities). • Outputs (NIOSH products; e.g., publications, reports, conferences, data- bases, tools, methods, guidelines, recommendations, education and train- ing, and patents). • Intermediate outcomes (actions by external stakeholders in response to NIOSH products; e.g., policy change, training and education, self-reported use or repackaging of NIOSH data by stakeholders, adoption of NIOSH- developed technologies, implemented guidelines, and licenses). • End outcomes (e.g., reduction in work-related injuries, illnesses, or hazard- ous exposures in the workplace). The framework committee understood that the efforts of any research pro- gram or the evaluation of that program will not be as linear as presented in either

OCR for page 37
OSHA, MSHA, other federal agencies; -Recommendations, Pilot and/or NIOSH reports, publications, Transfer: market-ready programs; workshops, databases, Employees, -Translation of technologies, Congress; state conferences; employers, research into training and and local industry, practice, products, education agencies; educators, -Training and education and technologies; programs, standards materials and regulators -Information guidance, bodies; labor, who reduce or demonstration programs, dissemination; regulations, trade, and trained professionals; prevent -Capacity building standards, professional hazardous through technical trade and associations; exposures or -Tools and methods, best assistance major media technology conditions practices, developmental (HHEs), training, releases, developers and technologies, licenses, and education websites manufacturers; patents other researchers; SH practitioners Conduct surveillance and evaluate intervention effectiveness FIGURE 3-1 The National Institute for Occupational Safety and Health logic model. Figure 3-1  R01502 copied from job #R01276 text is bitmapped and not editable

OCR for page 37
0 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s Exte rnal Factors C B D A Review and Analysis of Assessment Review and Goals and Identify M ajor of Inputs Assessme nt Objectives Driving Occupational Health of Activities Current Program and Safet y Challenges in Program Area Planning : e.g., surveillance Assessment of NIOSH and intervention data ; For example: surveillance, Independent assessment process to select stakeholder inputs health ef fects research, by commit tee to program goals, intervention research, compare with NIOSH evaluation of goals Production : e.g., intra- technology transfer program area goals selected by NIOSH, and ex tramural funding, ac tivities, health services comparison with staf fing, physical facilities, and other research assessment of management structure challenges (A) F E G Review and Assessment of Review and Review and Intermediate Assessme nt Assessment of Outcomes of Outputs End Outcomes Reduced injuries, For example: public policy For example: publications, illnesses, hazardous impact, training/ education, reports, databases, tools, exposures in the self-repor ted use and/ or methods, guidelines, workplac e repackaging by stakeholders, recommendations, implemente d guidelines licenses and patents Exte rnal Factors FIGURE 3-2 Overview of the evaluation process. Figure 3-2 por trait R01502 Figure 3-2 or Box 3-1; rather, they are iterative processes. Overlap necessarily occurs adapted from Figure A-2 in job #R01233 fully editable between the assessment of relevance and impact, particularly in the assessment of information transfer. Furthermore, components of any program may not fit perfectly into any one category. For example, training and development programs are appropriately defined as outputs by NIOSH in the logic model (Figure 3-1), but the framework committee found more value in focusing on the responses to these outputs as intermediate outcomes in the evaluation. Some NIOSH programs are organized using a matrix management approach as they span several NIOSH divisions or laboratories. Because resources within NIOSH are allocated in large part at the division level, rather than the program level, a matrix organization may have little control over the input portion of the logic model and therefore fewer resources within its direct control on which to base decisions. Following the suggested evaluation process ensured a level of consistency and comparability among all the evaluation committees. For future program evalua- tions, training on logic models and criteria for differentiating the various com-

OCR for page 37
e va l u at i o n f R a m e w o R k  BOX 3-1 Steps in the Evaluation Process   1.  Gather appropriate information.   2.  Assess external factors.   3.  Identify time frame to be evaluated.   4.  Identify major occupational health and safety challenges in program area.   5.  Analyze program goals and objectives.   6.  Identify major program components.   7.  Evaluate program inputs, activities, outputs, and outcomes.    8.  Determine scores for relevance and impact and provide the rationale.    9.    ssess  the  program’s  process  for  targeting  priority  research  needs  and  provide  the  A committee’s assessment of emerging issues.  10.  Prepare report by using the template provided as a guide. ponents of the logic model would be beneficial at the inception of the evaluation process both for NIOSH staff as they assemble the evidence packages and for evalu- ation committee members as they begin their assessment of the program. Drawing on the program logic model, the evaluation framework, and the evalu- ation committee members’ expertise, the evaluation committees began by examin- ing important inputs and external factors affecting the NIOSH research program’s agenda. Examples of external factors included research activities of industry and other federal agencies as well as the political and regulatory environment. The evaluation then focused on the program’s research activities, outputs, associated transfer activities, and resulting intermediate and end outcomes. Box 3-1 provides a summary of the evaluation process as suggested by the framework committee. Detailed guidance on each step is provided in later sections of this chapter. The following key factors were considered in assessing the relevance of NIOSH research programs: • The severity and/or frequency of health and safety hazards addressed and the number of people at risk (magnitude) for these hazards. • The extent to which NIOSH research programs identified and addressed gender-related issues and issues of vulnerable populations2 and the extent 2Vulnerable populations are defined as groups of workers who have biological, social, or economic characteristics that place them at increased risk for work-related conditions or on whom inadequate data have been collected. These populations include low-wage workers, disadvantaged minorities, disabled persons, and non-English-speakers for whom language or other barriers present health or safety risks.

OCR for page 37
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s to which NIOSH research programs addressed the health and safety needs of small businesses. • The stage of research on the problems being addressed. As the health ef- fects are understood, research efforts should shift from etiologic research to intervention research and then to intervention-effectiveness research. Gaps in the spectrum of prevention need to be addressed; for example, research on exposure assessment may be necessary before the next intervention steps can be taken. • The structure, in addition to the content, of the research program. A rel- evant research program is more than a set of unrelated research projects; it is an integrated program involving interrelated surveillance, research, and transfer activities. • Appropriate NIOSH consideration of external stakeholder input. The evaluation committees had the option to consider these and other relevant factors as they progressed through each stage of the evaluation. Data documenting end outcomes are often quite limited or are not available to quantify reductions in illness, injury, and hazardous exposures. Data documenting intermediate outcomes, although likely also limited, could serve as an appropriate proxy for end outcome data if the relationship between occupational exposures and health outcomes is well understood. For example, changes in regulations or procedures likely to result in reduced exposures are important measures of inter- mediate outcomes. Useful program evaluation requires specific questions and criteria for assessing each component of the program; a disciplined focus on a small number of questions or hypotheses typically related to program goals, performance criteria, and performance standards; a rigorous method of answering the questions or testing the hypotheses; and a credible procedure for developing qualitative and quantitative assessments. Because of the uniqueness of each NIOSH program, each evaluation committee determined the most reasonable way to apply the evaluation criteria. EVALUATION COMMITTEES The individual evaluation committees were formed in accordance with the rules of the National Academies that focus on ensuring a balanced committee. Each evaluation committee included: persons with expertise appropriate for the specific NIOSH research program under review (including researchers and representatives of stakeholder groups; e.g., worker organizations and industry), experts in technol - ogy and knowledge transfer, and experts in program evaluation.

OCR for page 37
e va l u at i o n f R a m e w o R k  The evaluation committee gathered appropriate information from the NIOSH research program under review, from external stakeholders affected directly by the NIOSH program research, and from relevant independent parties. The original contracts between NIOSH and the National Academies specified that each evalua- tion committee would consist of about 10 members, meet three times, and prepare a report due to NIOSH within 9 months of the first meeting of the evaluation committee. As noted in Chapter 4, future evaluations may consider extending the time frame to 12–14 months, depending on the size and complexity of the program being evaluated. STEPS IN THE EVALUATION PROCESS The evaluation process consists of 10 steps, described in the following sections and summarized in Box 3-1. A description is provided of how NIOSH programs were evaluated by the National Academies. This model can be applied to other program evaluations as well. 1. Gather Appropriate Information Each NIOSH program under review provided information to the relevant evaluation committee, including that outlined in Box 3-2. Some of the evaluation committees requested additional information from the program. Organizing the information listed in Box 3-2 by goal (or by subprogram if organizing by goal was not feasible) was helpful to the committees. In addition to the information provided by the NIOSH program, the evaluation committees independently collected additional information as deemed necessary for the evaluation, such as the perspectives of external stakeholders, including the Occupational Safety and Health Administration (OSHA), the Mine Safety and Health Administration (MSHA), workforces and their unions, and industry. In conducting the review, the evaluation committees examined how the inputs, activi- ties, outputs, and intermediate outcomes contribute to the impact and relevance of the program as a whole. Many NIOSH programs have been evaluated by internal and other external bodies as part of an overall assessment of NIOSH, such as the Performance As- sessment Rating Tool (PART) review,3 or through evaluations of specific research 3 PART focuses on assessing program-level performance and is one of the measures of success of the budget and performance integration initiative of the President’s management agenda (http://www. whitehouse.gov/omb/expectmore/summary/10002160.2004. html [accessed January 30, 2009]).

OCR for page 37
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s BOX 3-2 Evaluation Committee Information Needs •  Program background and overview: o  Program history o  Program management structure o  Major program challenges o  Program goals and objectives, past (for period under review) and current o  Process for developing and updating program strategic plans o  Enabling or authorizing legislation o  Major subprograms (if appropriate) o    esults of previous program reviews (e.g., annual review by NIOSH leadership team  R or external scientific program reviews) o  External factors affecting the program •  Interactions with external stakeholders and with other NIOSH programs: o    he role of program research staff in NIOSH policy setting, OSHA and MSHA stan- T dard setting, voluntary standard setting, and other government policy functions o  Interactions and working relationships with other NIOSH programs o    dentification  of  other  institutions  and  research  programs  with  similar  portfolios  I and an explanation of the relationship between NIOSH activities and those of other  institutions o    ey  partnerships  with  other  government  agencies,  employers,  labor,  academic  K institutions, nonprofit organizations, and international organizations •  Program inputs: o  Production inputs (program resources): ß  Funding by year for period under review ß  Funding by objective or subprogram ß    rogram staffing, full-time equivalents, and laboratory facilities, by subprogram  P (if indicated) ß  Percentage of program budget that is discretionary (beyond salaries) ß  Percentage of program budget that is earmarked ß  Significant contributions to the program from other sources (in kind or funds) o  Planning inputs: ß    urveillance  data,  inputs  from  the  Health  Hazard  Evaluation  and  the  Fatality  S Assessment and Control Evaluation programs, and intramural and extramural  research findings that influenced program goals and objectives ß    lanning inputs from stakeholders such as advisory groups; National Occupa- P tional  Research  Agenda  (NORA)  teams;  and  professional,  industry,  and  labor  groups (specify if any input comes from groups representing small business or  vulnerable populations) ß  Related OSHA and MSHA strategic plans or other input ß  Process for soliciting and approving intramural research ideas ß    rocess  for  soliciting  and  approving  program-supported  extramural  research  P activities

OCR for page 37
e va l u at i o n f R a m e w o R k  •  Program activities (more details provided in Box 3-3): o  Intramural: ß  Surveillance activities ß  Research activities ß    ransfer activities to encourage implementation of research results for improved  T occupational  safety  and  health  (e.g.,  information  dissemination,  technical  as- sistance, and technology and knowledge transfer) ß    ey collaborations in intramural activities (e.g., with other government agencies,  K academia, industry, and unions) o  Extramural funded by NIOSH: ß  Requests for applications developed by program ß    unded  projects:  grants,  cooperative  agreements,  and  contracts,  such  as  the  F following:    Surveillance activities   Research activities   Transfer activities   Capacity-building activities •  Outputs (products of the research program—more details provided in Box 3-4): o  Intramural: ß  Peer-reviewed publications, agency reports, alerts, and recommendations  ß    atabases,  websites,  tools,  and  methods  (including  education  and  training  D materials) ß  Technologies developed and patents ß  Sponsored conferences and workshops o  Extramural: ß  Program announcements ß  Requests for applications •  Intermediate outcomes:  o    tandards or guidelines issued by other agencies or organizations based in whole  S or in part on NIOSH research o    doption and use of control or personal protective technologies developed by NIOSH A o    vidence of  industry, employer,  or worker behavioral  changes in  response to  re- E search outputs o    se of NIOSH products by workers, industry, occupational health and safety profes- U sionals, healthcare providers, and others (including internationally) o  NIOSH website hits and document requests o  Unique staff or laboratory capabilities that serve as a national resource  o  Other intermediate outcomes, including those from extramural activities •  End outcomes: o    ata on program impact on rates and numbers of injuries, illnesses, and hazardous  D exposures in the workplace (including trend data, if available) o  Documentation of workplace risk reduction (quantitative, qualitative, or both) •    escription of current processes for setting research priorities and identifying emerging  D issues in the workplace

OCR for page 37
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s program elements. The evaluation committees were asked to review all prior evalu- ations of the program as an aid to understanding the evolution of the program and its elements. The National Academies committee evaluations, however, were independent of prior reviews and evaluations. 2. Assess External Factors As depicted in the logic model (Figure 3-1), reductions in work-related injury and illness or in hazardous exposures (end outcomes) are dependent on stakeholder activities (external factors). Actions beyond NIOSH’s control by those in labor, industry, regulatory entities, and elsewhere are necessary for NIOSH program ac- tivities to produce changes in end outcomes. Implementation of research findings may depend on existing or future policy considerations, economic conditions, and the public agenda. External factors were considered as forces beyond the control of the NIOSH program that may affect the evolution of the program. External factors influence progress through all phases of the logic model, from inputs to end outcomes (see Figure 3-1). Identification of external factors by an evaluation committee is essen- tial because it provides the context for evaluation of the program. External factors may be best assessed on the basis of the expert judgment of evaluation commit- tee members who have knowledge of the field of research. NIOSH program staff provided their ideas on external factors early in the evaluation process. Informa- tion regarding external factors was also sought from other NIOSH program and management staff, OSHA and MSHA staff, and from other external stakeholders. Additionally, each evaluation committee chose other approaches to assess external factors. Factors external to a program might help or hinder the achievement of certain outcomes or might present formidable obstacles. The evaluation commit- tees addressed both possibilities. Some external factors may constrain research activities related to specific target populations, methodological issues, or resource availability. Evaluation committees examined whether • Projects addressing a critical health need are technologically feasible. A workforce of appropriate size and with appropriate duration and distribu- tion of exposure for measuring a health effect may not exist; for example, no population of workers has been exposed for 30 years to formaldehyde at the current OSHA permissible exposure limit (PEL), so the related cancer mortality cannot yet be directly assessed. • Research is inhibited because NIOSH investigators are unable to access an adequate study population. Under current policy, NIOSH must either

OCR for page 37
e va l u at i o n f R a m e w o R k  obtain an invitation by management to study a workplace or seek a judicial order to provide authority to enter a worksite (cooperation under court order may well be insufficient for effective research). • Research is inhibited because the work environment, materials, and his- torical records cannot be accessed even with management and workforce cooperation. • Adequate or established methods exist for assessing the environment. • The NIOSH contribution to a particular field of research is reduced or difficult to estimate because other institutions are working in the same field. • NIOSH resources are inadequate to tackle key questions. Evaluation of the impact of NIOSH research outputs on worker health and safety also required consideration of external factors that might impede or aid implementation, measurement, and so on. Evaluation committees considered whether the following conditions exist and if so how this influences the research that NIOSH undertakes: • Regulatory changes and implementation are unachievable because of ob- stacles to regulation or because of differing priorities of the regulatory agencies. For example, there may be no implementation of recommenda- tions for improved respiratory protection programs for healthcare workers because of the lack or weakness of enforcement policies. • A feasible control for a known risk factor or exposure has not been imple- mented because the cost of implementation is too high or because current economic incentives do not favor such actions. • End outcomes are unobservable because baseline and continuing surveil- lance data are not available. For example, the current incidence of occu- pational noise-induced hearing loss is not known, although surveillance for a substantial threshold shift is feasible. (NIOSH conducts surveillance of some types of work-related illnesses, injuries, and hazards, but compre- hensive surveillance is not possible with existing resources.) • Reductions in adverse effects of chronic exposure cannot be measured. For example, 90 percent of identified work-related mortality is from diseases such as cancer that arise only after decades of latency after first exposure to a carcinogen. Effects of reducing exposure to a carcinogen therefore cannot be observed in the time frame of most interventions. • A promulgated regulation requires a technology that has been developed but is not widely used.

OCR for page 37
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s Assessment of end outcomes. The evaluation committee was asked to provide a qualitative assessment of program impact, discussing the evidence of reductions in injuries and illnesses or their appropriate proxies. Other outcomes. Regarding the NIOSH study, there may be as-yet unappreciated health and safety impacts or other beneficial social, economic, and environmental outcomes as a result of NIOSH activities. NIOSH study results may be influential outside the United States, and there may be evidence of implementation of NIOSH recommendations and training programs abroad. Questions considered in the evaluation of other outcomes 1. Is the program likely to produce a favorable change that has not yet oc- curred or not been appreciated? 2. Has the program been responsible for social, economic, security, or envi- ronmental outcomes? 3. Have program activities and outputs impacted occupational health and safety in other countries? Assessment of other outcomes. The evaluation committees were asked to con- sider other outcomes, including beneficial changes that are expected to occur; social, economic, security, or environmental outcomes; and the impact that the program has had on international occupational safety and health. 8. Determine Scores for Relevance and Impact and Provide the Rationale The evaluation committees assigned an integer score for the relevance of the research program to the improvement of occupational safety and health and another integer score for the impact of the program on such improvements. Using their expert judgment, the committees rated the relevance and impact of the overall research program by first summarizing their assessments of the major goals or subprograms and then appropriately weighting the goal areas (or subprograms) to determine the overall program ratings.5 Relevance and impact scores were based on 5-point categorical scales estab- lished by the framework committee (described below), in which 1 is the lowest and 5 the highest rating. The framework committee made an effort to establish mutu- 5 Inlight of substantial differences among the types of research programs, the framework commit- tee chose not to construct a single algorithm to use in weighting the goals or subprograms.

OCR for page 37
e va l u at i o n f R a m e w o R k  ally exclusive rating categories in the scales. The evaluation committee determined how individual goal areas (or subprograms) influenced final scores. Final program ratings consisted of integer scores for relevance and impact and prose justification of the scores. Box 3-5 provides an overview of the issues to be considered in determining ratings of relevance and impact. Evaluation committees were asked to consider items 1 through 4 in Box 3-5 for the overall program and to assess the relevance of the program by reviewing the committee’s responses to the questions evaluat- ing the program’s challenges, goals and objectives, inputs, activities, and outputs (Section 7). The evaluation committee evaluated separately the extent to which the program’s research efforts are in high-priority areas and the extent to which the program is involved in transfer activities. Transfer activities occur in two contexts: (1) efforts by the NIOSH program to translate intellectual products into practice and (2) stakeholder efforts to integrate NIOSH results into the workplace. To assess impact, each evaluation committee first needed to consider the avail- able evidence of changes in work-related risks and the adverse effects and external factors related to the changes. The evaluation committee reviewed the responses to the questions on the reviews of outputs, intermediate outcomes, and end out- comes and systematically assessed the impact of the research program. Items 2 to 7 in Box 3-5 address these areas. The evaluation committee needed to judge, for BOX 3-5 Overview of the Issues Assess the following for each program: 1.    elevance of current and recently completed research and transfer activities to objec- R tive improvements in workplace safety and health.  2.    ontributions of the NIOSH program’s research and transfer activities to changes in  C work-related practices and reduction in workplace exposures, illnesses, or injuries. 3.    ontributions of the NIOSH program’s research and transfer activities to improve- C ments in work-related practices. 4.    ontributions of the NIOSH program’s research to productivity, security, or environ- C mental quality (beneficial side effects). 5.    vidence of policy, technological, behavioral, and other changes that would reduce  E risk in the workplace (intermediate outcomes).  6.    vidence of reduction in workplace exposures, illnesses, or injuries (end outcomes).  E 7.    vidence of external factors that prevented translation of NIOSH research results into  E intermediate or end outcomes.

OCR for page 37
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s example, whether outcomes occurred earlier than they would have or are better than they would have been in the absence of the research program, or whether outcomes would have occurred were it not for external factors beyond the control of the NIOSH program. Scoring of Relevance As discussed in previous sections, numerous factors may be considered in as- sessing relevance. The scoring criteria focus on the evaluation committee’s assess- ment of whether the program appropriately set priorities among research needs as well as how engaged the program was in appropriate transfer activities to move research findings into the workplace. Since the evaluation of NIOSH programs in- cluded assessment of research activities and knowledge transfer activities, both are considered in the final relevance score. With respect to research, the key indicator is the extent to which the program’s research is in priority subject areas (high prior- ity, priority, lesser priority, or not focused on priorities); with respect to transfer, the key indicator is the level of engagement in appropriate transfer activities (in this case, significantly engaged, engaged, or not engaged). This approach resulted in a complex scoring system that tries to address the best and worst cases and any variations in between. Box 3-6 lists the criteria for scoring the overall relevance BOX 3-6 Scoring Criteria for Relevance 5  =    esearch  is  in  high-priority  subject  areas  and  the  NIOSH  program  is  significantly  R engaged  in  appropriate  transfer  activities  for  completed  research  projects/reported  research results. 4  =    esearch is in high-priority subject areas and the NIOSH program is engaged in ap- R propriate transfer activities for completed research projects/reported research results;  or research is in priority subject areas and the NIOSH program is significantly engaged  in  appropriate  transfer  activities  for  completed  research  projects/reported  research  results. 3  =    esearch is in high-priority subject areas, but the NIOSH program is not engaged in  R appropriate transfer activities; or research is in priority subject areas but the NIOSH  program  is  not  significantly  engaged  in  appropriate  transfer  activities;  or  research  focuses  on  lesser  priorities  but  the  NIOSH  program  is  significantly  engaged  in  ap- propriate transfer activities. 2  =    esearch program is focused on lesser priorities and the NIOSH program is not sig- R nificantly engaged in appropriate transfer activities.  1  =    esearch program is not focused on priorities. R

OCR for page 37
e va l u at i o n f R a m e w o R k  TABLE 3-1 Guidance for Weighting Research Priority and Transfer Activities Assessment of Engagement in Appropriate Research Priority Transfer Activities Applicable Score High priority Significantly engaged 5 High priority Engaged 4 Priority Significantly engaged 4 High priority Not engaged 3 Priority Engaged or not engaged 3 Lesser priority Significantly engaged 3 Lesser priority Engaged or not engaged 2 Not focused on priorities Any level of engagement 1 of the NIOSH research program. Table 3-1 provides guidance regarding how the committee may weight research priorities and transfer levels when determining relevance scores. The evaluation committee considered both completed research and research that is in progress in its assessment of relevance. The committee kept in mind how well the program has considered the frequency and severity of the problems being addressed; whether appropriate attention has been directed to issues regarding both sexes, vulnerable populations, or hard-to-reach workplaces; and whether the different needs of large and small businesses have been considered. Each committee determined how to consider external factors in assigning program scores. Scoring of Impact Box 3-7 provides the criteria established for the rating of impact. The evalu- ation committee primarily considered completed research outputs. In assigning a score for impact, it is important to recognize that a “major contribution” (required for a score of 5) does not imply that the NIOSH program was solely responsible for observed improvements in worker safety and health. Many factors may be required to effect improvements. The committee could say that the NIOSH program made “major contributions” if the improvements would not have occurred when they did without the program’s efforts. The framework committee had some concern that the imposed scoring criteria for impact might be considered a promotion of the conventional occupational- health research paradigm that focuses on health-effects and technology research

OCR for page 37
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s BOX 3-7 Scoring Criteria for Impact 5  =    esearch program has made major contribution(s) to worker health and safety on the  R basis of end outcomes or well-accepted intermediate outcomes. 4  =    esearch program has made some contributions to end outcomes or well-accepted  R intermediate outcomes. 3  =    esearch program activities are ongoing and outputs are produced that are likely to  R result in improvements in worker health and safety. Well-accepted outcomes have not  been recorded. 2  =    esearch program activities are ongoing and outputs are produced that may result in  R new knowledge or technology, but only limited application is expected. Well-accepted  outcomes have not been recorded. 1  =    esearch  activities  and  outputs  do  not  result  in  or  are  not  likely  to  have  any  R application. NA = Impact cannot be assessed; program is not mature enough. without much emphasis on the socioeconomic, policy, surveillance, and diffusion research (as opposed to diffusion activities) needed to effect change. The evalua- tion committees were asked to remember that not all intermediate outcomes occur in the workplace. Important outcomes that NIOSH can affect also occur much farther out on the causal chain. NIOSH, for example, has an important role in generating knowledge that may contribute to changing norms in the insurance industry, in healthcare practice, in public health practice, and in the community at large. The evaluation committees considered whether some of those issues need to be addressed and considered as external factors that facilitate or limit applica- tion of more traditional research findings. Given the rapidly changing nature of work and the workforce and the intractable problems in manufacturing, mining, and other fields, the evaluation committees were encouraged to think beyond the conventional paradigm. 9. Assess the Program’s Process for Targeting Priority Research Needs and Provide the Committee’s Assessment of Emerging Issues Among the most challenging aspects of research in illness and injury preven- tion are the identification of new or emerging needs and trends and the formula-

OCR for page 37
e va l u at i o n f R a m e w o R k  tion of a research response that uses scarce resources to best effect in anticipation of them. The second charge to the evaluation committee was assessment of the re - search program’s effectiveness in targeting new research and identifying emerging issues in occupational safety and health most relevant to future improvements in workplace protection. The evaluation committee was asked to provide a quali - tative narrative assessment of the program’s process for determining priorities for research and emerging workplace issues. The committee also independently identified emerging workplace issues that the NIOSH program should be pre- pared to address. The evaluation committees reviewed the procedures that the NIOSH program has in place to identify needed research relevant to the NIOSH mission and re- viewed the success that the NIOSH program has had in identifying and addressing research related to emerging issues. For example, the program should be involved in examining leading indicators from other federal agencies (e.g., EPA, Department of Labor, National Institute of Standards and Technology, NIH, DoD, and Depart- ment of Commerce) that track or provide data on new technologies, new products, new processes, and disease or injury trends. The NIOSH HHE program offers a potential source for the identification of emerging research needs. The evaluation committee needed to determine whether the program under review appropriately considered pertinent HHE investigation findings. Additional emerging issues may have been revealed through consideration of NIOSH and NIOSH-funded FACE reports, AOEC reports, U.S. Chemical Safety Board investigations, and the Sentinel Event Notification System for Occupational Risks and other state-based surveillance programs. Appropriate federal advisory committees and other stakeholder groups were also consulted to provide qualita- tive information. The evaluation committee systematically assessed how the research program targets new research by evaluating each goal area or subprogram for the items listed in Box 3-8. Questions Considered in Identifying Emerging Issues 1. What information does the NIOSH program review to identify emerging research needs? • What is the process for review? • How often does the process take place? • How are NIOSH staff scientists and leadership engaged? • What is the process for moving from ideas to formal planning and resource allocation?

OCR for page 37
0 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s BOX 3-8 Targeting of New Research and Identification of Emerging Issues Assess the following: 1.  Past and present effectiveness in targeting most relevant research needs. 2.    ffectiveness in targeting research in fields most relevant to future improvements in  E occupational safety and health. 2. How are external stakeholders involved? • What advisory or stakeholder groups are asked to identify emerging research targets? • How often are such groups consulted, and how are suggestions fol- lowed up? 3. What new research targets have been identified for future development in the program under evaluation? • How were they identified? • Were lessons learned that could help to identify other emerging issues? • Does the evaluation committee agree with the issues identified and selected as important and with the NIOSH program’s response, or were important issues overlooked? • Is there evidence of unwise expenditure of resources on unimportant issues? The evaluation committee members used their expert judgment both to evalu- ate the emerging research targets identified by the NIOSH program and to provide recommendations on improvements to the program or additional research that NIOSH had not yet identified. Recommendations included a brief statement of their rationale. 10. Prepare Report by Using the Template Provided as a Guide Consistency and comparability among evaluation committee report formats was desirable, but the framework committee recognized that each NIOSH research program is different and that each evaluation committee was independent. The outline provided in Box 3-9 flows from the framework committee’s review of the generalized NIOSH logic model (Figure 3-1) and the overviews of the evaluation process (Box 3-1, Figure 3-2). The evaluation committees were free to use or adapt

OCR for page 37
e va l u at i o n f R a m e w o R k  BOX 3-9 Suggested Outline for Evaluation Committee Reports I. Introduction This  section  should  be  a  brief  descriptive  summary  of  the  history  of  the  program  being  evaluated with respect to pre-NORA, NORA 1, and current and future plans of the research  program presented by the NIOSH program. It should present the context for the research on  safety and health; goals, objectives, and resources; groupings of goal areas or subprograms;  and any other important pertinent information (a list of the NIOSH materials reviewed should  be provided in Appendix C). II. Evaluation of the Program (Charge 1) A.  Evaluation  summary:  should  include  a  brief  summary  of  the  evaluation  with  respect to impact and relevance, scores for impact and relevance, and summary  statements B.  Strategic goals and objectives: should describe assessment of the extent to which  program strategic plans reflect program relevance C.  Assessment of inputs: should describe adequacy of inputs to achieve goals D.  Assessment  of  activities:  should  describe  assessment  of  the  relevance  of  the  activities E.  Assessment  of  research  program  outputs:  should  describe  assessment  of  rel- evance and potential usefulness of the research program’s outputs F.  Assessment  of  intermediate  outcomes  and  causal  impact:  should  describe  as- sessment of the intermediate outcomes and the program’s contribution to them;  should include the likely impacts and recent outcomes in the assessment G.  Assessment of end outcomes: should describe the end outcomes related to health  and safety and provide an assessment of the type and degree of attribution to the  NIOSH program H.  Assessment  of  other  outcomes:  should  discuss  health  and  safety  impacts  that  are expected to occur; beneficial social, economic, and environmental outcomes;  and international dimensions and outcomes  I.  Summary of ratings and rationale III. NIOSH Targeting of New Research and Identification of Emerging Issues (Charge 2) The evaluation committee should assess the progress that the NIOSH program has made in  targeting new research in occupational safety and health. The evaluation committee should  assess whether the NIOSH program has identified important emerging issues that appear  especially important in terms of relevance to the mission of NIOSH. The evaluation commit- tee should respond to NIOSH’s perspective and add its own recommendations. IV. Recommendations for Program Improvement On the basis of the review and evaluation of the program, the evaluation committee may  provide recommendations for improving the relevance of the NIOSH research program to  (continued)

OCR for page 37
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s BOX 3-9 Continued safety and health conditions in the workplace and the impact of the research program on  safety and health in the workplace. Appendix A Framework Document Appendix B Methods and Information Gathering Appendix C List of NIOSH and Related Materials Collected in the Process of the Evaluation this outline as necessary when organizing their final reports. The framework com- mittee encouraged each evaluation committee to look at prior evaluation commit- tee reports for organizational ideas. REFERENCES IOM and NRC (Institute of Medicine and National Research Council). 2006. Hearing loss research at NIOSH. Committee to Review the NIOSH Hearing Loss Research Program. Rpt. No. 1, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. IOM and NRC. 2008. The personal protective technology program at NIOSH. Committee to Review the NIOSH Personal Protective Technology Program. Rpt. No. 5, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. IOM and NRC. 2009. Traumatic injury research at NIOSH. Committee to Review the NIOSH Trau- matic Injury Research Program. Rpt. No. 6, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. NRC and IOM. 2007. Mining safety and health research at NIOSH. Committee to Review the NIOSH Mining Safety and Health Research Program. Rpt. No. 2, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Acad- emies Press. NRC and IOM. 2008a. Agriculture, forestry, and fishing research at NIOSH. Committee to Review the NIOSH Agriculture, Forestry, and Fishing Research Program. Rpt. No. 3, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. NRC and IOM. 2008b. Respiratory diseases research at NIOSH. Committee to Review the NIOSH Re- spiratory Diseases Research Program. Rpt. No. 4, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press.

OCR for page 37
e va l u at i o n f R a m e w o R k  NRC and IOM. 2009a. The health hazard evaluation program at NIOSH. Committee to Review the NIOSH Health Hazard Evaluation Program. Rpt. No. 7, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Acad- emies Press. NRC and IOM. 2009b. Construction research at NIOSH. Committee to Review the NIOSH Construc- tion Research Program. Rpt. No. 8, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. Rutstein, D. D., R. J. Mullan, T. M. Frazier, W. E. Halperin, J. M. Melius, and J. P. Sestito. 1983. Sentinel health events (occupational): A basis for physician recognition and public health surveillance. American Journal of Public Health 73(9):1054–1062. Williams, V. L., E. Eiseman, E. Landree, D. M. Adamson. 2009. Demonstrating and communicating research impact: Preparing NIOSH programs for external review. Santa Monica, CA: RAND.

OCR for page 37