|
||||||||||||||||
|
||||||||||||||||
The following HTML text is provided to enhance online readability. Many aspects of typography translate only awkwardly to HTML. Please use the page image as the authoritative form to ensure accuracy. 3
|
||||||||||||||||
 |
 |
Page 37 |
 |
 |
 |
|
| ||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 37
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
3
Evaluation Framework
As discussed throughout this report, an evaluation framework was developed to guide the NIOSH program evaluations and provide a set of criteria to be used in scoring the relevance and impact of each program’s efforts in reducing work-related hazardous exposures, illnesses, and injuries. The National Academies Committee for the Review of NIOSH Research Programs (referred to as the framework committee) developed an initial evaluation framework that was honed and refined to improve its utility, clarity, and emphasis as the eight evaluations proceeded. The framework presented in this chapter differs slightly from the versions that were used by the evaluation committees.1 Insights gained in the evaluation of the eight NIOSH programs (see Chapter 4) are included in this version of the framework, which is provided for consideration in future program evaluations. In addition to evaluating NIOSH programs, this analytic framework and approach may be applicable to the evaluation of other federal agency research programs or research programs in other organizations.
In conducting its evaluation, each committee was asked to determine whether the NIOSH program was undertaking high-priority, relevant research and transfer activities (relevance) and whether these efforts are improving health and safety in the workplace (impact). The evaluation committee was also tasked with (1) rating
1
The evaluation framework document used by the individual evaluation committee is provided as an appendix in each of the evaluation committee reports (IOM and NRC, 2006, 2008, 2009; NRC and IOM, 2007, 2008a,b, 2009a,b).
OCR for page 38
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
both the relevance and the impact of the NIOSH program using 1–5 integer scales, and (2) providing input about emerging areas of research and recommendations for program improvement.
OVERVIEW OF THE EVALUATION FRAMEWORK
After examining different approaches to program evaluation (see Chapter 2), the framework committee decided to define the scope and stages of the evaluation process based on the logic model (Williams et al., 2009). The resulting evaluation framework described in this chapter breaks the logic models developed by NIOSH (Figure 3-1) into discrete program components to be assessed by each evaluation committee. Criteria for evaluation of each component of the framework are detailed below. In the evaluation framework (overview provided in Figure 3-2), the assessment of strategic goals and objectives, inputs, activities, and outputs (B to E) largely define the relevance of the program, while assessment of intermediate and end outcomes (F and G) largely define the program impact.
The following major components of each NIOSH program were assessed by the evaluation committees:
Major occupational safety and health challenges in the program area.
Goals and objectives as defined by NIOSH.
Inputs (e.g., budget; staff; facilities; and input from the program’s research management, the NIOSH Board of Scientific Counselors, and stakeholders).
Activities (efforts by NIOSH staff, contractors, and grantees; e.g., surveillance of injury, illness, and hazards; exposure assessment research; health-effects research; injury-risk factor research; intervention research; health services research; and technology transfer activities).
Outputs (NIOSH products; e.g., publications, reports, conferences, databases, tools, methods, guidelines, recommendations, education and training, and patents).
Intermediate outcomes (actions by external stakeholders in response to NIOSH products; e.g., policy change, training and education, self-reported use or repackaging of NIOSH data by stakeholders, adoption of NIOSH-developed technologies, implemented guidelines, and licenses).
End outcomes (e.g., reduction in work-related injuries, illnesses, or hazardous exposures in the workplace).
The framework committee understood that the efforts of any research program or the evaluation of that program will not be as linear as presented in either
OCR for page 39
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
FIGURE 3-1 The National Institute for Occupational Safety and Health logic model.
OCR for page 40
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
FIGURE 3-2 Overview of the evaluation process.
Figure 3-2 or Box 3-1; rather, they are iterative processes. Overlap necessarily occurs between the assessment of relevance and impact, particularly in the assessment of information transfer. Furthermore, components of any program may not fit perfectly into any one category. For example, training and development programs are appropriately defined as outputs by NIOSH in the logic model (Figure 3-1), but the framework committee found more value in focusing on the responses to these outputs as intermediate outcomes in the evaluation.
Some NIOSH programs are organized using a matrix management approach as they span several NIOSH divisions or laboratories. Because resources within NIOSH are allocated in large part at the division level, rather than the program level, a matrix organization may have little control over the input portion of the logic model and therefore fewer resources within its direct control on which to base decisions.
Following the suggested evaluation process ensured a level of consistency and comparability among all the evaluation committees. For future program evaluations, training on logic models and criteria for differentiating the various com-
OCR for page 41
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
BOX 3-1
Steps in the Evaluation Process
Gather appropriate information.
Assess external factors.
Identify time frame to be evaluated.
Identify major occupational health and safety challenges in program area.
Analyze program goals and objectives.
Identify major program components.
Evaluate program inputs, activities, outputs, and outcomes.
Determine scores for relevance and impact and provide the rationale.
Assess the program’s process for targeting priority research needs and provide the committee’s assessment of emerging issues.
Prepare report by using the template provided as a guide.
ponents of the logic model would be beneficial at the inception of the evaluation process both for NIOSH staff as they assemble the evidence packages and for evaluation committee members as they begin their assessment of the program.
Drawing on the program logic model, the evaluation framework, and the evaluation committee members’ expertise, the evaluation committees began by examining important inputs and external factors affecting the NIOSH research program’s agenda. Examples of external factors included research activities of industry and other federal agencies as well as the political and regulatory environment. The evaluation then focused on the program’s research activities, outputs, associated transfer activities, and resulting intermediate and end outcomes.
Box 3-1 provides a summary of the evaluation process as suggested by the framework committee. Detailed guidance on each step is provided in later sections of this chapter. The following key factors were considered in assessing the relevance of NIOSH research programs:
The severity and/or frequency of health and safety hazards addressed and the number of people at risk (magnitude) for these hazards.
The extent to which NIOSH research programs identified and addressed gender-related issues and issues of vulnerable populations2 and the extent
2
Vulnerable populations are defined as groups of workers who have biological, social, or economic characteristics that place them at increased risk for work-related conditions or on whom inadequate data have been collected. These populations include low-wage workers, disadvantaged minorities, disabled persons, and non-English-speakers for whom language or other barriers present health or safety risks.
OCR for page 42
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
to which NIOSH research programs addressed the health and safety needs of small businesses.
The stage of research on the problems being addressed. As the health effects are understood, research efforts should shift from etiologic research to intervention research and then to intervention-effectiveness research. Gaps in the spectrum of prevention need to be addressed; for example, research on exposure assessment may be necessary before the next intervention steps can be taken.
The structure, in addition to the content, of the research program. A relevant research program is more than a set of unrelated research projects; it is an integrated program involving interrelated surveillance, research, and transfer activities.
Appropriate NIOSH consideration of external stakeholder input.
The evaluation committees had the option to consider these and other relevant factors as they progressed through each stage of the evaluation.
Data documenting end outcomes are often quite limited or are not available to quantify reductions in illness, injury, and hazardous exposures. Data documenting intermediate outcomes, although likely also limited, could serve as an appropriate proxy for end outcome data if the relationship between occupational exposures and health outcomes is well understood. For example, changes in regulations or procedures likely to result in reduced exposures are important measures of intermediate outcomes.
Useful program evaluation requires specific questions and criteria for assessing each component of the program; a disciplined focus on a small number of questions or hypotheses typically related to program goals, performance criteria, and performance standards; a rigorous method of answering the questions or testing the hypotheses; and a credible procedure for developing qualitative and quantitative assessments. Because of the uniqueness of each NIOSH program, each evaluation committee determined the most reasonable way to apply the evaluation criteria.
EVALUATION COMMITTEES
The individual evaluation committees were formed in accordance with the rules of the National Academies that focus on ensuring a balanced committee. Each evaluation committee included: persons with expertise appropriate for the specific NIOSH research program under review (including researchers and representatives of stakeholder groups; e.g., worker organizations and industry), experts in technology and knowledge transfer, and experts in program evaluation.
OCR for page 43
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
The evaluation committee gathered appropriate information from the NIOSH research program under review, from external stakeholders affected directly by the NIOSH program research, and from relevant independent parties. The original contracts between NIOSH and the National Academies specified that each evaluation committee would consist of about 10 members, meet three times, and prepare a report due to NIOSH within 9 months of the first meeting of the evaluation committee. As noted in Chapter 4, future evaluations may consider extending the time frame to 12–14 months, depending on the size and complexity of the program being evaluated.
STEPS IN THE EVALUATION PROCESS
The evaluation process consists of 10 steps, described in the following sections and summarized in Box 3-1. A description is provided of how NIOSH programs were evaluated by the National Academies. This model can be applied to other program evaluations as well.
1.
Gather Appropriate Information
Each NIOSH program under review provided information to the relevant evaluation committee, including that outlined in Box 3-2. Some of the evaluation committees requested additional information from the program. Organizing the information listed in Box 3-2 by goal (or by subprogram if organizing by goal was not feasible) was helpful to the committees.
In addition to the information provided by the NIOSH program, the evaluation committees independently collected additional information as deemed necessary for the evaluation, such as the perspectives of external stakeholders, including the Occupational Safety and Health Administration (OSHA), the Mine Safety and Health Administration (MSHA), workforces and their unions, and industry. In conducting the review, the evaluation committees examined how the inputs, activities, outputs, and intermediate outcomes contribute to the impact and relevance of the program as a whole.
Many NIOSH programs have been evaluated by internal and other external bodies as part of an overall assessment of NIOSH, such as the Performance Assessment Rating Tool (PART) review,3 or through evaluations of specific research
3
PART focuses on assessing program-level performance and is one of the measures of success of the budget and performance integration initiative of the President’s management agenda (http://www.whitehouse.gov/omb/expectmore/summary/10002160.2004. html [accessed January 30, 2009]).
OCR for page 44
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
BOX 3-2
Evaluation Committee Information Needs
Program background and overview:
Program history
Program management structure
Major program challenges
Program goals and objectives, past (for period under review) and current
Process for developing and updating program strategic plans
Enabling or authorizing legislation
Major subprograms (if appropriate)
Results of previous program reviews (e.g., annual review by NIOSH leadership team or external scientific program reviews)
External factors affecting the program
Interactions with external stakeholders and with other NIOSH programs:
The role of program research staff in NIOSH policy setting, OSHA and MSHA standard setting, voluntary standard setting, and other government policy functions
Interactions and working relationships with other NIOSH programs
Identification of other institutions and research programs with similar portfolios and an explanation of the relationship between NIOSH activities and those of other institutions
Key partnerships with other government agencies, employers, labor, academic institutions, nonprofit organizations, and international organizations
Program inputs:
Production inputs (program resources):
Funding by year for period under review
Funding by objective or subprogram
Program staffing, full-time equivalents, and laboratory facilities, by subprogram (if indicated)
Percentage of program budget that is discretionary (beyond salaries)
Percentage of program budget that is earmarked
Significant contributions to the program from other sources (in kind or funds)
Planning inputs:
Surveillance data, inputs from the Health Hazard Evaluation and the Fatality Assessment and Control Evaluation programs, and intramural and extramural research findings that influenced program goals and objectives
Planning inputs from stakeholders such as advisory groups; National Occupational Research Agenda (NORA) teams; and professional, industry, and labor groups (specify if any input comes from groups representing small business or vulnerable populations)
Related OSHA and MSHA strategic plans or other input
Process for soliciting and approving intramural research ideas
Process for soliciting and approving program-supported extramural research activities
OCR for page 45
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
Program activities (more details provided in Box 3-3):
Intramural:
Surveillance activities
Research activities
Transfer activities to encourage implementation of research results for improved occupational safety and health (e.g., information dissemination, technical assistance, and technology and knowledge transfer)
Key collaborations in intramural activities (e.g., with other government agencies, academia, industry, and unions)
Extramural funded by NIOSH:
Requests for applications developed by program
Funded projects: grants, cooperative agreements, and contracts, such as the following:
Surveillance activities
Research activities
Transfer activities
Capacity-building activities
Outputs (products of the research program—more details provided in Box 3-4):
Intramural:
Peer-reviewed publications, agency reports, alerts, and recommendations
Databases, websites, tools, and methods (including education and training materials)
Technologies developed and patents
Sponsored conferences and workshops
Extramural:
Program announcements
Requests for applications
Intermediate outcomes:
Standards or guidelines issued by other agencies or organizations based in whole or in part on NIOSH research
Adoption and use of control or personal protective technologies developed by NIOSH
Evidence of industry, employer, or worker behavioral changes in response to research outputs
Use of NIOSH products by workers, industry, occupational health and safety professionals, healthcare providers, and others (including internationally)
NIOSH website hits and document requests
Unique staff or laboratory capabilities that serve as a national resource
Other intermediate outcomes, including those from extramural activities
End outcomes:
Data on program impact on rates and numbers of injuries, illnesses, and hazardous exposures in the workplace (including trend data, if available)
Documentation of workplace risk reduction (quantitative, qualitative, or both)
Description of current processes for setting research priorities and identifying emerging issues in the workplace
OCR for page 46
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
program elements. The evaluation committees were asked to review all prior evaluations of the program as an aid to understanding the evolution of the program and its elements. The National Academies committee evaluations, however, were independent of prior reviews and evaluations.
2.
Assess External Factors
As depicted in the logic model (Figure 3-1), reductions in work-related injury and illness or in hazardous exposures (end outcomes) are dependent on stakeholder activities (external factors). Actions beyond NIOSH’s control by those in labor, industry, regulatory entities, and elsewhere are necessary for NIOSH program activities to produce changes in end outcomes. Implementation of research findings may depend on existing or future policy considerations, economic conditions, and the public agenda.
External factors were considered as forces beyond the control of the NIOSH program that may affect the evolution of the program. External factors influence progress through all phases of the logic model, from inputs to end outcomes (see Figure 3-1). Identification of external factors by an evaluation committee is essential because it provides the context for evaluation of the program. External factors may be best assessed on the basis of the expert judgment of evaluation committee members who have knowledge of the field of research. NIOSH program staff provided their ideas on external factors early in the evaluation process. Information regarding external factors was also sought from other NIOSH program and management staff, OSHA and MSHA staff, and from other external stakeholders. Additionally, each evaluation committee chose other approaches to assess external factors. Factors external to a program might help or hinder the achievement of certain outcomes or might present formidable obstacles. The evaluation committees addressed both possibilities.
Some external factors may constrain research activities related to specific target populations, methodological issues, or resource availability. Evaluation committees examined whether
Projects addressing a critical health need are technologically feasible. A workforce of appropriate size and with appropriate duration and distribution of exposure for measuring a health effect may not exist; for example, no population of workers has been exposed for 30 years to formaldehyde at the current OSHA permissible exposure limit (PEL), so the related cancer mortality cannot yet be directly assessed.
Research is inhibited because NIOSH investigators are unable to access an adequate study population. Under current policy, NIOSH must either
OCR for page 47
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
obtain an invitation by management to study a workplace or seek a judicial order to provide authority to enter a worksite (cooperation under court order may well be insufficient for effective research).
Research is inhibited because the work environment, materials, and historical records cannot be accessed even with management and workforce cooperation.
Adequate or established methods exist for assessing the environment.
The NIOSH contribution to a particular field of research is reduced or difficult to estimate because other institutions are working in the same field.
NIOSH resources are inadequate to tackle key questions.
Evaluation of the impact of NIOSH research outputs on worker health and safety also required consideration of external factors that might impede or aid implementation, measurement, and so on. Evaluation committees considered whether the following conditions exist and if so how this influences the research that NIOSH undertakes:
Regulatory changes and implementation are unachievable because of obstacles to regulation or because of differing priorities of the regulatory agencies. For example, there may be no implementation of recommendations for improved respiratory protection programs for healthcare workers because of the lack or weakness of enforcement policies.
A feasible control for a known risk factor or exposure has not been implemented because the cost of implementation is too high or because current economic incentives do not favor such actions.
End outcomes are unobservable because baseline and continuing surveillance data are not available. For example, the current incidence of occupational noise-induced hearing loss is not known, although surveillance for a substantial threshold shift is feasible. (NIOSH conducts surveillance of some types of work-related illnesses, injuries, and hazards, but comprehensive surveillance is not possible with existing resources.)
Reductions in adverse effects of chronic exposure cannot be measured. For example, 90 percent of identified work-related mortality is from diseases such as cancer that arise only after decades of latency after first exposure to a carcinogen. Effects of reducing exposure to a carcinogen therefore cannot be observed in the time frame of most interventions.
A promulgated regulation requires a technology that has been developed but is not widely used.
OCR for page 64
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
Assessment of end outcomes. The evaluation committee was asked to provide a qualitative assessment of program impact, discussing the evidence of reductions in injuries and illnesses or their appropriate proxies.
Other outcomes. Regarding the NIOSH study, there may be as-yet unappreciated health and safety impacts or other beneficial social, economic, and environmental outcomes as a result of NIOSH activities. NIOSH study results may be influential outside the United States, and there may be evidence of implementation of NIOSH recommendations and training programs abroad.
Questions considered in the evaluation of other outcomes
Is the program likely to produce a favorable change that has not yet occurred or not been appreciated?
Has the program been responsible for social, economic, security, or environmental outcomes?
Have program activities and outputs impacted occupational health and safety in other countries?
Assessment of other outcomes. The evaluation committees were asked to consider other outcomes, including beneficial changes that are expected to occur; social, economic, security, or environmental outcomes; and the impact that the program has had on international occupational safety and health.
8.
Determine Scores for Relevance and Impact and Provide the Rationale
The evaluation committees assigned an integer score for the relevance of the research program to the improvement of occupational safety and health and another integer score for the impact of the program on such improvements. Using their expert judgment, the committees rated the relevance and impact of the overall research program by first summarizing their assessments of the major goals or subprograms and then appropriately weighting the goal areas (or subprograms) to determine the overall program ratings.5
Relevance and impact scores were based on 5-point categorical scales established by the framework committee (described below), in which 1 is the lowest and 5 the highest rating. The framework committee made an effort to establish mutu-
5
In light of substantial differences among the types of research programs, the framework committee chose not to construct a single algorithm to use in weighting the goals or subprograms.
OCR for page 65
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
ally exclusive rating categories in the scales. The evaluation committee determined how individual goal areas (or subprograms) influenced final scores. Final program ratings consisted of integer scores for relevance and impact and prose justification of the scores.
Box 3-5 provides an overview of the issues to be considered in determining ratings of relevance and impact. Evaluation committees were asked to consider items 1 through 4 in Box 3-5 for the overall program and to assess the relevance of the program by reviewing the committee’s responses to the questions evaluating the program’s challenges, goals and objectives, inputs, activities, and outputs (Section 7). The evaluation committee evaluated separately the extent to which the program’s research efforts are in high-priority areas and the extent to which the program is involved in transfer activities. Transfer activities occur in two contexts: (1) efforts by the NIOSH program to translate intellectual products into practice and (2) stakeholder efforts to integrate NIOSH results into the workplace.
To assess impact, each evaluation committee first needed to consider the available evidence of changes in work-related risks and the adverse effects and external factors related to the changes. The evaluation committee reviewed the responses to the questions on the reviews of outputs, intermediate outcomes, and end outcomes and systematically assessed the impact of the research program. Items 2 to 7 in Box 3-5 address these areas. The evaluation committee needed to judge, for
BOX 3-5
Overview of the Issues
Assess the following for each program:
Relevance of current and recently completed research and transfer activities to objective improvements in workplace safety and health.
Contributions of the NIOSH program’s research and transfer activities to changes in work-related practices and reduction in workplace exposures, illnesses, or injuries.
Contributions of the NIOSH program’s research and transfer activities to improvements in work-related practices.
Contributions of the NIOSH program’s research to productivity, security, or environmental quality (beneficial side effects).
Evidence of policy, technological, behavioral, and other changes that would reduce risk in the workplace (intermediate outcomes).
Evidence of reduction in workplace exposures, illnesses, or injuries (end outcomes).
Evidence of external factors that prevented translation of NIOSH research results into intermediate or end outcomes.
OCR for page 66
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
example, whether outcomes occurred earlier than they would have or are better than they would have been in the absence of the research program, or whether outcomes would have occurred were it not for external factors beyond the control of the NIOSH program.
Scoring of Relevance
As discussed in previous sections, numerous factors may be considered in assessing relevance. The scoring criteria focus on the evaluation committee’s assessment of whether the program appropriately set priorities among research needs as well as how engaged the program was in appropriate transfer activities to move research findings into the workplace. Since the evaluation of NIOSH programs included assessment of research activities and knowledge transfer activities, both are considered in the final relevance score. With respect to research, the key indicator is the extent to which the program’s research is in priority subject areas (high priority, priority, lesser priority, or not focused on priorities); with respect to transfer, the key indicator is the level of engagement in appropriate transfer activities (in this case, significantly engaged, engaged, or not engaged). This approach resulted in a complex scoring system that tries to address the best and worst cases and any variations in between. Box 3-6 lists the criteria for scoring the overall relevance
BOX 3-6
Scoring Criteria for Relevance
5 =
Research is in high-priority subject areas and the NIOSH program is significantly engaged in appropriate transfer activities for completed research projects/reported research results.
4 =
Research is in high-priority subject areas and the NIOSH program is engaged in appropriate transfer activities for completed research projects/reported research results; or research is in priority subject areas and the NIOSH program is significantly engaged in appropriate transfer activities for completed research projects/reported research results.
3 =
Research is in high-priority subject areas, but the NIOSH program is not engaged in appropriate transfer activities; or research is in priority subject areas but the NIOSH program is not significantly engaged in appropriate transfer activities; or research focuses on lesser priorities but the NIOSH program is significantly engaged in appropriate transfer activities.
2 =
Research program is focused on lesser priorities and the NIOSH program is not significantly engaged in appropriate transfer activities.
1 =
Research program is not focused on priorities.
OCR for page 67
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
TABLE 3-1 Guidance for Weighting Research Priority and Transfer Activities
Assessment of Research Priority
Engagement in Appropriate Transfer Activities
Applicable Score
High priority
Significantly engaged
5
High priority
Engaged
4
Priority
Significantly engaged
4
High priority
Not engaged
3
Priority
Engaged or not engaged
3
Lesser priority
Significantly engaged
3
Lesser priority
Engaged or not engaged
2
Not focused on priorities
Any level of engagement
1
of the NIOSH research program. Table 3-1 provides guidance regarding how the committee may weight research priorities and transfer levels when determining relevance scores.
The evaluation committee considered both completed research and research that is in progress in its assessment of relevance. The committee kept in mind how well the program has considered the frequency and severity of the problems being addressed; whether appropriate attention has been directed to issues regarding both sexes, vulnerable populations, or hard-to-reach workplaces; and whether the different needs of large and small businesses have been considered. Each committee determined how to consider external factors in assigning program scores.
Scoring of Impact
Box 3-7 provides the criteria established for the rating of impact. The evaluation committee primarily considered completed research outputs. In assigning a score for impact, it is important to recognize that a “major contribution” (required for a score of 5) does not imply that the NIOSH program was solely responsible for observed improvements in worker safety and health. Many factors may be required to effect improvements. The committee could say that the NIOSH program made “major contributions” if the improvements would not have occurred when they did without the program’s efforts.
The framework committee had some concern that the imposed scoring criteria for impact might be considered a promotion of the conventional occupational-health research paradigm that focuses on health-effects and technology research
OCR for page 68
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
BOX 3-7
Scoring Criteria for Impact
5 =
Research program has made major contribution(s) to worker health and safety on the basis of end outcomes or well-accepted intermediate outcomes.
4 =
Research program has made some contributions to end outcomes or well-accepted intermediate outcomes.
3 =
Research program activities are ongoing and outputs are produced that are likely to result in improvements in worker health and safety. Well-accepted outcomes have not been recorded.
2 =
Research program activities are ongoing and outputs are produced that may result in new knowledge or technology, but only limited application is expected. Well-accepted outcomes have not been recorded.
1 =
Research activities and outputs do not result in or are not likely to have any application.
NA = Impact cannot be assessed; program is not mature enough.
without much emphasis on the socioeconomic, policy, surveillance, and diffusion research (as opposed to diffusion activities) needed to effect change. The evaluation committees were asked to remember that not all intermediate outcomes occur in the workplace. Important outcomes that NIOSH can affect also occur much farther out on the causal chain. NIOSH, for example, has an important role in generating knowledge that may contribute to changing norms in the insurance industry, in healthcare practice, in public health practice, and in the community at large. The evaluation committees considered whether some of those issues need to be addressed and considered as external factors that facilitate or limit application of more traditional research findings. Given the rapidly changing nature of work and the workforce and the intractable problems in manufacturing, mining, and other fields, the evaluation committees were encouraged to think beyond the conventional paradigm.
9.
Assess the Program’s Process for Targeting Priority Research Needs and Provide the Committee’s Assessment of Emerging Issues
Among the most challenging aspects of research in illness and injury prevention are the identification of new or emerging needs and trends and the formula-
OCR for page 69
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
tion of a research response that uses scarce resources to best effect in anticipation of them.
The second charge to the evaluation committee was assessment of the research program’s effectiveness in targeting new research and identifying emerging issues in occupational safety and health most relevant to future improvements in workplace protection. The evaluation committee was asked to provide a qualitative narrative assessment of the program’s process for determining priorities for research and emerging workplace issues. The committee also independently identified emerging workplace issues that the NIOSH program should be prepared to address.
The evaluation committees reviewed the procedures that the NIOSH program has in place to identify needed research relevant to the NIOSH mission and reviewed the success that the NIOSH program has had in identifying and addressing research related to emerging issues. For example, the program should be involved in examining leading indicators from other federal agencies (e.g., EPA, Department of Labor, National Institute of Standards and Technology, NIH, DoD, and Department of Commerce) that track or provide data on new technologies, new products, new processes, and disease or injury trends.
The NIOSH HHE program offers a potential source for the identification of emerging research needs. The evaluation committee needed to determine whether the program under review appropriately considered pertinent HHE investigation findings. Additional emerging issues may have been revealed through consideration of NIOSH and NIOSH-funded FACE reports, AOEC reports, U.S. Chemical Safety Board investigations, and the Sentinel Event Notification System for Occupational Risks and other state-based surveillance programs. Appropriate federal advisory committees and other stakeholder groups were also consulted to provide qualitative information.
The evaluation committee systematically assessed how the research program targets new research by evaluating each goal area or subprogram for the items listed in Box 3-8.
Questions Considered in Identifying Emerging Issues
What information does the NIOSH program review to identify emerging research needs?
What is the process for review?
How often does the process take place?
How are NIOSH staff scientists and leadership engaged?
What is the process for moving from ideas to formal planning and resource allocation?
OCR for page 70
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
BOX 3-8
Targeting of New Research and Identification of Emerging Issues
Assess the following:
Past and present effectiveness in targeting most relevant research needs.
Effectiveness in targeting research in fields most relevant to future improvements in occupational safety and health.
How are external stakeholders involved?
What advisory or stakeholder groups are asked to identify emerging research targets?
How often are such groups consulted, and how are suggestions followed up?
What new research targets have been identified for future development in the program under evaluation?
How were they identified?
Were lessons learned that could help to identify other emerging issues?
Does the evaluation committee agree with the issues identified and selected as important and with the NIOSH program’s response, or were important issues overlooked?
Is there evidence of unwise expenditure of resources on unimportant issues?
The evaluation committee members used their expert judgment both to evaluate the emerging research targets identified by the NIOSH program and to provide recommendations on improvements to the program or additional research that NIOSH had not yet identified. Recommendations included a brief statement of their rationale.
10.
Prepare Report by Using the Template Provided as a Guide
Consistency and comparability among evaluation committee report formats was desirable, but the framework committee recognized that each NIOSH research program is different and that each evaluation committee was independent. The outline provided in Box 3-9 flows from the framework committee’s review of the generalized NIOSH logic model (Figure 3-1) and the overviews of the evaluation process (Box 3-1, Figure 3-2). The evaluation committees were free to use or adapt
OCR for page 71
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
BOX 3-9
Suggested Outline for Evaluation Committee Reports
Introduction
This section should be a brief descriptive summary of the history of the program being evaluated with respect to pre-NORA, NORA 1, and current and future plans of the research program presented by the NIOSH program. It should present the context for the research on safety and health; goals, objectives, and resources; groupings of goal areas or subprograms; and any other important pertinent information (a list of the NIOSH materials reviewed should be provided in Appendix C).
Evaluation of the Program (Charge 1)
Evaluation summary: should include a brief summary of the evaluation with respect to impact and relevance, scores for impact and relevance, and summary statements
Strategic goals and objectives: should describe assessment of the extent to which program strategic plans reflect program relevance
Assessment of inputs: should describe adequacy of inputs to achieve goals
Assessment of activities: should describe assessment of the relevance of the activities
Assessment of research program outputs: should describe assessment of relevance and potential usefulness of the research program’s outputs
Assessment of intermediate outcomes and causal impact: should describe assessment of the intermediate outcomes and the program’s contribution to them; should include the likely impacts and recent outcomes in the assessment
Assessment of end outcomes: should describe the end outcomes related to health and safety and provide an assessment of the type and degree of attribution to the NIOSH program
Assessment of other outcomes: should discuss health and safety impacts that are expected to occur; beneficial social, economic, and environmental outcomes; and international dimensions and outcomes
Summary of ratings and rationale
NIOSH Targeting of New Research and Identification of Emerging Issues (Charge 2)
The evaluation committee should assess the progress that the NIOSH program has made in targeting new research in occupational safety and health. The evaluation committee should assess whether the NIOSH program has identified important emerging issues that appear especially important in terms of relevance to the mission of NIOSH. The evaluation committee should respond to NIOSH’s perspective and add its own recommendations.
Recommendations for Program Improvement
On the basis of the review and evaluation of the program, the evaluation committee may provide recommendations for improving the relevance of the NIOSH research program to
OCR for page 72
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
safety and health conditions in the workplace and the impact of the research program on safety and health in the workplace.
Appendix A Framework Document
Appendix B Methods and Information Gathering
Appendix C List of NIOSH and Related Materials Collected in the Process of the Evaluation
this outline as necessary when organizing their final reports. The framework committee encouraged each evaluation committee to look at prior evaluation committee reports for organizational ideas.
REFERENCES
IOM and NRC (Institute of Medicine and National Research Council). 2006. Hearing loss research at NIOSH. Committee to Review the NIOSH Hearing Loss Research Program. Rpt. No. 1, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press.
IOM and NRC. 2008. The personal protective technology program at NIOSH. Committee to Review the NIOSH Personal Protective Technology Program. Rpt. No. 5, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press.
IOM and NRC. 2009. Traumatic injury research at NIOSH. Committee to Review the NIOSH Traumatic Injury Research Program. Rpt. No. 6, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press.
NRC and IOM. 2007. Mining safety and health research at NIOSH. Committee to Review the NIOSH Mining Safety and Health Research Program. Rpt. No. 2, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press.
NRC and IOM. 2008a. Agriculture, forestry, and fishing research at NIOSH. Committee to Review the NIOSH Agriculture, Forestry, and Fishing Research Program. Rpt. No. 3, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press.
NRC and IOM. 2008b. Respiratory diseases research at NIOSH. Committee to Review the NIOSH Respiratory Diseases Research Program. Rpt. No. 4, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press.
OCR for page 73
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
NRC and IOM. 2009a. The health hazard evaluation program at NIOSH. Committee to Review the NIOSH Health Hazard Evaluation Program. Rpt. No. 7, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press.
NRC and IOM. 2009b. Construction research at NIOSH. Committee to Review the NIOSH Construction Research Program. Rpt. No. 8, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press.
Rutstein, D. D., R. J. Mullan, T. M. Frazier, W. E. Halperin, J. M. Melius, and J. P. Sestito. 1983. Sentinel health events (occupational): A basis for physician recognition and public health surveillance. American Journal of Public Health 73(9):1054–1062.
Williams, V. L., E. Eiseman, E. Landree, D. M. Adamson. 2009. Demonstrating and communicating research impact: Preparing NIOSH programs for external review. Santa Monica, CA: RAND.
OCR for page 74
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
This page intentionally left blank.