A
Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health*

This is the second version of a document prepared by the National Academies Committee for the Review of NIOSH Research Programs1 also referred to as the Framework Committee. This document is not a formal report of the National Academies—rather, it is a framework proposed for use by multiple National Academies evaluation committees to review up to 15 National Institute for Occupational Safety and Health (NIOSH) research programs. It is a working document subject to modification by the Framework Committee on the basis of responses received from evaluation-committee members, NIOSH, stakeholders, and the general public during the course of the assessments.

*

Version of 8/10/07.

1

Members of the committee at the time this version was produced were David Wegman, Chair (University of Massachusetts Lowell School of Health and Environment), William Bunn III (International Truck and Engine Corporation), Carlos Camargo (Harvard Medical School), Susan Cozzens (Georgia Institute of Technology), Letitia Davis (Massachusetts Department of Public Health), James Dearing (Kaiser Permanente-Colorado), Fred Mettler, Jr. (University of New Mexico School of Medicine), Franklin Mirer (Hunter College School of Health Sciences), Jacqueline Nowell (United Food and Commercial Workers International Union), Raja Ramani (Pennsylvania State University), Jorma Rantanen (International Commission on Occupational Health), Rosemary Sokas (University of Illinois at Chicago School of Public Health), Richard Tucker (Tucker and Tucker Consultants, Inc., and University of Texas at Austin), and James Zuiches (North Carolina State University). Sammantha Magsino (National Academies staff) was the study director. Joseph Wholey (University of Southern California), former committee member, contributed to the first version of this document. Part V includes brief biographies of current committee members.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 137
A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health* T his is the second version of a document prepared by the National Academies Committee for the Review of NIOSH Research Programs1 also referred to as the Framework Committee. This document is not a formal report of the National Academies—rather, it is a framework proposed for use by multiple National Academies evaluation committees to review up to 15 National Institute for Occupational Safety and Health (NIOSH) research programs. It is a working document subject to modification by the Framework Committee on the basis of responses received from evaluation-committee members, NIOSH, stakeholders, and the general public during the course of the assessments. *Version of 8/10/07. 1 Members of the committee at the time this version was produced were David Wegman, Chair (University of Massachusetts Lowell School of Health and Environment), William Bunn III (Inter- national Truck and Engine Corporation), Carlos Camargo (Harvard Medical School), Susan Cozzens (Georgia Institute of Technology), Letitia Davis (Massachusetts Department of Public Health), James Dearing (Kaiser Permanente-Colorado), Fred Mettler, Jr. (University of New Mexico School of Medicine), Franklin Mirer (Hunter College School of Health Sciences), Jacqueline Nowell (United Food and Commercial Workers International Union), Raja Ramani (Pennsylvania State University), Jorma Rantanen (International Commission on Occupational Health), Rosemary Sokas (University of Illinois at Chicago School of Public Health), Richard Tucker (Tucker and Tucker Consultants, Inc., and University of Texas at Austin), and James Zuiches (North Carolina State University). Sammantha Magsino (National Academies staff) was the study director. Joseph Wholey (University of Southern California), former committee member, contributed to the first version of this document. Part V includes brief biographies of current committee members. 37

OCR for page 137
t R au m at I C I n j u Ry R e s e a R C h nIosh 38 at This version reflects several significant changes to the original framework document (version 12/19/05) that was used to guide the work of the first four evaluation committees (Hearing Loss; Mining; Agriculture, Forestry, and Fish- ing; and Respiratory Disease). Changes were made in response to feedback from members and staff of these committees, as well as other comments on the original framework, in order to make the document more useful to evaluation committees as they carry out their work. In particular, the following changes were made to the framework document during the revision process: • The wording of some of the relevance and impact scores were edited to make the wording more precise and to reduce situations where the original scores were non-unique or overlapping (revised scoring cri- teria are given in Boxes 2 and 3). • A new table was added to provide explicit guidance to evaluation com- mittees on how to weigh differences in the observed levels of “research priority” and “engagement in appropriate transfer activities” in arriv- ing at a single integer score for relevance (see Table 6). • The guidance on scoring was clarified to make more explicit that all scores are to be given as integers. • The NIOSH logic model was updated (see Figure 1). • The table on evaluation committee information needs (Table 2) was reorganized to be more consistent with the NIOSH logic model, and additional information needs identified by the first set of evaluation committees were added. • A worksheet to assist with the development of scores has been deleted and key components of the worksheet have been incorporated into appropriate sections throughout the document. • The organization of the document was modified to more closely follow the revised statement of task and to improve readability. • A number of sections of text originally presented in outline form were modified in tables or boxes to make the information more accessible. This second version of the framework document remains a working document subject to further modification by the Framework Committee on the basis of input received from evaluation committee members, NIOSH, stakeholders, and the gen- eral public during the course of the assessments.

OCR for page 137
appendIx a 39 CONTENTS Abbreviations and Acronyms I. Introduction I.A. Overview of Charge to Evaluation Committees I.B. Evaluation Committees I.C. NIOSH Strategic Goals and Operational Plan I.D. Evaluation Committees’ Information Needs I.E. Prior Evaluations II. Summary of Evaluation Process II.A. The Evaluation Flowchart (Figure 2) II.B. Steps in Program Evaluation III. Evaluation of a NIOSH Research Program—The Process III.A. Analysis of External Factors Relevant to the NIOSH Research Program III.B. Evaluating NIOSH Research Programs by Using the Flowchart III.B.1. Identifying the Period for Evaluation III.B.2. Identifying Major Challenges (Figure 2, Circle) III.B.3. Analysis of Research-Program Strategic Goals and Objectives (Figure 2, Box A) III.B.4. Review of Inputs (Figure 2, Box B) III.B.5. Review of Activities (Figure 2, Box C) III.B.6. Review of Outputs (Figure 2, Box D) III.B.7. Review of Intermediate Outcomes (Figure 2, Box E) III.B.8. Review of End Outcomes (Figure 2, Box F) III.B.9. Review of Potential Outcomes III.B.10. Summary Evaluation Ratings and Rationale III.C. Assessment of NIOSH Process for Targeting Priority Research Needs and Committee Assessment of Emerging Issues IV. Evaluation Committee Report Template Figure 1 The NIOSH operational plan presented as a logic model Figure 2 Flowchart for the evaluation of the NIOSH research program Table 1 NORA High-Priority Research by Category Table 2 Evaluation Committee Information Needs Table 3 Examples of NIOSH Program Research and Transfer Activities Table 4 Examples of Research-Program Outputs to Be Considered Table 5 Background Context for Program Relevance and Impact Table 6 Guidance for Weighting Research Priority and Engagement in Appropriate Transfer Activities in the Application of Relevance Score

OCR for page 137
t R au m at I C I n j u Ry R e s e a R C h nIosh 40 at Table 7 Targeting of New Research and Identification of Emerging Issues Box 1 The Evaluation Process Box 2 Scoring Criteria for Relevance Box 3 Scoring Criteria for Impact Box 4 Suggested Outline for Evaluation Committee Reports

OCR for page 137
appendIx a 4 ABBREVIATIONS AND ACRONYMS ABLES Adult Blood Lead Epidemiology and Surveillance AOEC Association of Occupational and Environmental Clinics BLS Bureau of Labor Statistics CDC Centers for Disease Control and Prevention CSTE Council of State and Territorial Epidemiologists DOD U.S. Department of Defense EC Evaluation Committee EPA Environmental Protection Agency FACE Fatality Assessment Control and Evaluation FC Framework Committee HHE Health Hazard Evaluation MSHA Mine Safety and Health Administration NIH National Institutes of Health NIOSH National Institute for Occupational Safety and Health NORA National Occupational Research Agenda NORA1 National Occupational Research Agenda 1996-2005 NORA2 National Occupational Research Agenda 2005-forward OSH Review Occupational Safety and Health Review Commission Commission OSHA Occupational Safety and Health Administration OSHAct Occupational Safety and Health Act of 1970 PART Performance Assessment Rating Tool PEL permissible exposure limit RFA request for applications SENSOR Sentinel Event Notification System of Occupational Risks

OCR for page 137
t R au m at I C I n j u Ry R e s e a R C h nIosh 42 at TMT tools, methods, or technologies USDA U.S. Department of Agriculture

OCR for page 137
appendIx a 43 I. INTRODUCTION In September 2004, the National Institute for Occupational Safety and Health (NIOSH) contracted with the National Academies to conduct a review of NIOSH research programs. The goal of this multiphase effort is to assist NIOSH in in- creasing the impact of its research efforts that are aimed at reducing workplace illnesses and injuries and improving occupational safety and health. The National Academies assigned the task to the Division on Earth and Life Studies and the Institute of Medicine. The National Academies appointed a committee of 14 members, including persons with expertise in occupational medicine and health, industrial health and safety, industrial hygiene, epidemiology, civil and mining engineering, sociology, program evaluation, communication, and toxicology; representatives of industry and of the workforce; and a scientist experienced in international occupational- health issues. The Committee on the Review of NIOSH Research Programs, re- ferred to as the Framework Committee (FC), prepared the first version of this document during meetings held on May 5-6, July 7-8, and August 15-16, 2005. This second version was finalized after the Framework Committee’s May 30-31, 2007, meeting, based on feedback received on the framework from the first two independent evaluation committees, NIOSH leadership, and National Academies’ staff, as well as discussions during an earlier FC meeting in April 2006. This document is not a report of the National Academies; rather, it presents the evaluation framework developed by the FC to guide and provide common structure for the reviews of as many as 15 NIOSH programs during a 5-year period by independent evaluation committees (ECs) appointed by various divisions and boards of the National Academies. It is a working document to be shared with NIOSH and the public. This version has been modified by the FC on the basis of responses from the ECs, NIOSH, NIOSH stakeholders, and the public; and it may be modified again. It is incumbent on the ECs to consult with the FC if portions of the evaluation framework presented here are inappropriate for specific programs under review. I.A. Overview of Charge to Evaluation Committees At the first meeting of the FC, Lewis Wade, NIOSH senior science adviser, emphasized that a review of a NIOSH program should focus on the program’s relevance to and impact on health and safety in the workplace. In developing a framework, the FC considered the following elements of the charge to the ECs:

OCR for page 137
t R au m at I C I n j u Ry R e s e a R C h nIosh 44 at 1. Assessment of the program’s contribution, through occupational safety and health research, to reductions in workplace hazardous exposures, illnesses, or injuries through a. an assessment of the relevance of the program’s activities to the improvement of occupational safety and health, and b. an evaluation of the impact that the program’s research has had in reducing work-related hazardous exposures, illnesses, and injuries. The evaluation committee will rate the performance of the program for its relevance and impact using an integer score of 1-5. Impact may be assessed directly (for example, on the basis of reductions in ill- nesses or injuries) or, as necessary, by using intermediate outcomes to estimate impact. Qualitative narrative evaluations should be included to explain the numerical ratings. 2. Assessment of the program’s effectiveness in targeting new research areas and identifying emerging issues in occupational safety and health most relevant to future improvements in workplace protection. The committee will provide a qualitative narrative assessment of the pro- gram’s efforts and suggestions about emerging issues that the program should be prepared to address. I.B. Evaluation Committees Individual ECs will be formed in accordance with the rules of the National Academies for the formation of balanced committees. Each EC will comprise per- sons with expertise appropriate for the specific NIOSH research program under review and may include representatives of stakeholder groups (such as labor unions and industry), experts in technology and knowledge transfer, and program evalu- ation. The EC will gather appropriate information from the sponsor (the NIOSH research program under review), stakeholders affected directly by NIOSH program research, and relevant independent parties. Each EC will consist of about 10 mem- bers, will meet about three times, and will prepare a report. The National Academies will deliver the report to NIOSH within 9 months of the first meeting of the EC. EC reports are subject to the National Academies report-review process. I.C. NIOSH Strategic Goals and Operational Plan As a prelude to understanding the NIOSH strategic goals and operational plan, NIOSH research efforts should be understood in the context of the Occupational

OCR for page 137
appendIx a 45 Safety and Health Act (OSHAct), under which it was created. The OSHAct identifies workplace safety and health as having high national priority and gives employers the responsibility for controlling hazards and preventing workplace injury and ill- ness. The act creates an organizational framework for doing that, assigning comple- mentary roles and responsibilities to employers and employees, the Occupational Safety and Health Administration (OSHA), the states, the Occupational Safety and Health (OSH) Review Commission, and NIOSH. The act recognizes NIOSH’s role and responsibilities to be supportive and indirect. NIOSH research, training programs, criteria, and recommendations are intended to be used to inform and assist those more directly responsible for hazard control (OSHAct Sections 2b, 20, and 22). Section 2b of the OSHAct describes 13 interdependent means of accom- plishing the national goal, one of which is “by providing for research . . . and by developing innovative methods . . . for dealing with occupational safety and health problems.” Sections 20 and 22 give the responsibility for that research to NIOSH. NIOSH is also given related responsibilities, including the development of criteria to guide prevention of work-related injury or illness; development of regulations for reporting on employee exposures to harmful agents; establish- ment of medical examinations, programs, or tests to determine illness incidence and susceptibility; publication of a list of all known toxic substances; assessment of potential toxic effects or risks associated with workplace exposure in specific settings; and conduct of education programs for relevant professionals to carry out the OSHAct purposes. NIOSH is also responsible for assisting the secretary of labor regarding education programs for employees and employers in hazard recognition and control. The NIOSH mission is “to provide national and world leadership to prevent work-related illness, injury, disability, and death by gathering information, con- ducting scientific research, and translating the knowledge gained into products and services.” To fulfill its mission, NIOSH has established the following strategic goals:2 Goal 1: Conduct research to reduce work-related illnesses and • injuries. ° Track work-related hazards, exposures, illnesses, and injuries for prevention. ° Generate new knowledge through intramural and extramural research programs. 2 See http://www.cdc.gov/niosh/docs/strategic/.

OCR for page 137
t R au m at I C I n j u Ry R e s e a R C h nIosh 46 at Develop innovative solutions for difficult-to-solve problems in ° high-risk industrial sectors. Goal 2: Promote safe and healthy workplaces through interventions, • recommendations, and capacity building. ° Enhance the relevance and utility of recommendations and guidance. ° Transfer research findings, technologies, and information into practice. ° Build capacity to address traditional and emerging hazards. Goal 3: Enhance global workplace safety and health through inter- • national collaborations. ° Take a leadership role in developing a global network of occupa- tional health centers. ° Investigate alternative approaches to workplace illness and injury reduction and provide technical assistance to put solutions in place. ° Build global professional capacity to address workplace hazards through training, information sharing, and research experience. In 1994, NIOSH embarked on a national partnership effort to identify research priorities to guide occupational health and safety research for the next decade. The National Occupational Research Agenda (NORA) identified 21 high-priority research subjects (see Table 1). The NORA was intended not only for NIOSH but for the entire occupational health community. In the second decade of the NORA, NIOSH is working with its partners to update the research agenda, using an ap- proach based on industry sectors. NIOSH and its partners are working through sector research councils to establish sector-specific research goals and objectives. The emphasis is on moving research to practice in workplaces through sector-based partnerships. Figure 1 is the NIOSH operational plan, presented as a logic model,3 of the path from inputs to outcomes for each NIOSH research program. The FC adapted the model to develop its framework. NIOSH will provide similar logic models ap- propriate to each research program evaluated by an EC. 3 Developed by NIOSH with the assistance of the RAND Corporation.

OCR for page 137
appendIx a 47 TABLE 1 NORA High-Priority Research by Category Category Priority Research Area Disease and injury Allergic and irritant dermatitis Asthma and chronic obstructive pulmonary disease Fertility and pregnancy abnormalities Hearing loss Infectious diseases Low-back disorders Musculoskeletal disorders of upper extremities Trauma Work environment and workforce Emerging technologies Indoor environment Mixed exposures Organization of work Special populations at risk Research tools and approaches Cancer research methods Control technology and personal protective equipment Exposure-assessment methods Health-services research Intervention-effectiveness research Risk-assessment methods Social and economic consequences of workplace illness and injury Surveillance research methods I.D. Evaluation Committees’ Information Needs Each NIOSH program under review will provide information to the relevant EC, including that outlined in Table 2. The EC may request additional informa- tion of NIOSH as needed, and NIOSH should provide it as quickly as is practical. NIOSH should consider organizing the information listed in Table 2 by subpro- gram or program as appropriate and to the extent possible. In addition to the information provided by NIOSH, the EC should indepen- dently collect additional information that it deems necessary for evaluation (for example, the perspectives of stakeholders, such as OSHA, MSHA, unions and work- forces, and industry). In conducting the review, the EC should continually examine how individual projects or activities contribute to the impact and relevance of a program as a whole.

OCR for page 137
appendIx a 7 so on) for specific purposes. Whether the products have resulted in changes in the workplace or in the reduction of risk should be discussed. The recognition accorded to the program or the facilities by its peers (such as recognition as a “center of excellence” by national and international communities) should be considered in the assessment. To be highly ranked, a program should have high performance in most of the relevant questions in this section. An aspect of the evaluation can be whether the same changes in stakeholder activities and behaviors would probably have occurred without NIOSH efforts. III.B.8. Review of End Outcomes (Figure 2, Box F) It is necessary for the EC to assess, to the greatest extent possible, NIOSH’s contribution to end outcomes—improvements in workplace health and safety (im- pact). For purposes of this evaluation, end outcomes are health-related changes that are a result of program activities, including decreases in injuries, illnesses, deaths and exposures or risk. Data on reductions in work-related injuries, illnesses, and hazardous exposures will be available for some programs, and in some cases they will be quantifiable. It is possible, however, to evaluate the impact of a NIOSH research program using either intermediate outcomes or end outcomes. If there is no direct evidence of improvements in health and safety, intermediate outcomes may be used as proxies for end outcomes in assessing impact as long as the EC qualifies its findings. The EC will describe the realized or potential benefits of the NIOSH program. Examples of realized intermediate outcomes are new regulations and widely accepted guidelines, work practices, and procedures, all of which may contribute measurably to enhancing health and safety in the workplace. The FC recognizes that assessing the causal relationship between NIOSH re- search and specific occupational health and safety outcomes is a major challenge because NIOSH does not have direct responsibility or authority for implement- ing its research findings in the workplace. Furthermore, the benefits of NIOSH research program outputs can be realized, potential, or limited to the knowledge gained. Studies that conclude with negative results may nevertheless have incor- porated excellent science and contribute to the knowledge base. The generation of important knowledge is a recognized form of outcome in the absence of measur- able impacts. The impact of an outcome depends on the existence of a “receptor” for research results, such as a regulatory agency, a professional organization, an employer, and an employee organization. The EC should consider questions related to the various stages that lead to outputs, such as these:

OCR for page 137
t R au m at I C I n j u Ry R e s e a R C h nIosh 72 at 1. Did NIOSH research identify a gap in protection or a means of reduc- ing risk? 2. Did NIOSH convey that information to potential users in a usable form? 3. Were NIOSH research results (for example, recommendations, tech- nologies) applied? 4. Did the applied results lead to desired outcomes? Quantitative data are preferable to qualitative, but qualitative analysis may be necessary. Sources of quantitative data include the following: • Bureau of Labor Statistics (BLS) data on fatal occupational injuries (the Census of Fatal Occupational Injuries) and nonfatal occupational injuries and illnesses (the annual Survey of Occupational Injury and Illnesses) • NIOSH intramural surveillance systems, such as the National Elec- tronic Injury Surveillance System, the coal-worker x-ray surveillance program, and agricultural-worker surveys conducted by NIOSH in collaboration with USDA • State-based surveillance systems, such as the NIOSH-funded ABLES, and the SENSOR programs (for asthma, pesticides, silicosis, noise- induced hearing loss, dermatitis, and burns) • Selected state worker-compensation programs • Exposure data collected in the OSHA Integrated Management Infor- mation System The FC is unaware of mechanisms for surveillance of many occupationally related chronic illnesses, such as cancers that arise from long exposure to chemicals and other stressors. The incidence and prevalence of many such outcomes are best evaluated by investigator-initiated research. Research that leads to new, effective surveillance concepts or programs warrants special recognition. The EC should recognize the strengths and weaknesses of outcome data sources. Quantitative accident, injury, illness, and employment data and databases are sub- ject to error and bias and should be used by the EC only for drawing inferences after critical evaluation and examination of available corroborating data. For example, it is widely recognized that occupational illnesses are poorly documented in the BLS Survey of Occupational Injuries and Illnesses, which captures only incident cases among active workers. It is difficult for health practitioners to diagnose work-relat- edness of most illnesses that may not be exclusively related to work; furthermore, few practitioners are adequately trained to make such an assessment. Many of those

OCR for page 137
appendIx a 73 illnesses have long latencies and do not appear until years after people have left the employment in question. Surveillance programs may systematically undercount some categories of workers, such as contingent workers. In addition to measures of illness and injury, measures of exposure to chemical and physical agents and to safety and ergonomic hazards can be useful. Exposure or probability of exposure can serve as an appropriate proxy for disease or injury when a well-described occupational exposure-health association exists. In such instances, a decrease in exposure can be accepted as evidence that the end outcome of reduced illness or injury is being achieved. That is necessary particularly when the latent period between exposure and disease outcome, as in the case of asbestos exposure and lung cancer, makes effective evaluation of the relevant end outcome infeasible. As an example of how an exposure level can serve as a proxy, reduction in the number of sites that exceed an OSHA PEL or an American Conference of Gov- ernmental Industrial Hygienists threshold limit value is a quantitative measure of improvement of occupational health awareness and reduction of risk. In addition to exposure level, the number of people exposed and the distribution of exposure levels are important. Those data are available from multiple databases and studies of exposure. Apart from air monitoring, such measures of exposure as biohazard controls, reduction in requirements for use of personal protective equipment, and reduction in ergonomic risks are important. Challenges posed by inadequate or inaccurate measurement systems should not drive programs out of difficult fields of study, and the EC will need to be aware of such a possibility. In particular, contingent and informal working arrangements that place workers at greatest risk are also those on which surveillance information is almost totally lacking, so novel methods for measuring impact may be required. The commitment of industry, labor, and government to health and safety are critical external factors. Several measures of that commitment can be useful for the EC: monetary commitments, attitude, staffing, and surveys of relative importance. To the extent that resources allocated to safety and health are limiting factors, the EC should explicitly assess NIOSH performance in the context of constraints. Questions to Guide the Evaluation Committee 1. What are the amounts and qualities of relevant end-outcomes data (such as injuries, illness, exposure, and productivity affected by health)? 2. What are the temporal trends in those data? 3. Is there objective evidence of improvement in occupational safety or health?

OCR for page 137
t R au m at I C I n j u Ry R e s e a R C h nIosh 74 at 4. To what degree is the NIOSH program or subprogram responsible for improvement in occupational safety or health? 5. If there is no time trend in the data, how do findings compare with data from other comparable U.S. groups or the corresponding populations in other countries? 6. What is the evidence that external factors have affected outcomes or outcome measures? 7. Has the program been responsible for outcomes outside the United States that have not been described in another category? Assessment The EC should provide a qualitative assessment of the program and subpro- gram impact, discussing the evidence of reductions in injuries and illnesses or their appropriate proxies. III.B.9. Review of Potential Outcomes There may be health and safety impacts not yet appreciated and other benefi- cial social, economic, and environmental outcomes as a result of NIOSH activities. NIOSH study results may be influential outside the United States, and there may be evidence of implementation of NIOSH recommendations and training programs abroad. Questions to Guide the Evaluation Committee 1. Is the program likely to produce a favorable change that has not yet occurred or not been appreciated? 2. Has the program been responsible for social, economic, security, or environmental outcomes? 3. Has the program’s work had an impact on occupational health and safety in other countries? Assessment The EC may discuss other outcomes, including beneficial changes that have not yet occurred; social, economic, security, or environmental outcomes; and the impact that NIOSH has had on international occupational safety and health.

OCR for page 137
appendIx a 75 III.B.10. Summary Evaluation Ratings and Rationale The EC should use its expert judgment to rate the relevance and impact of the overall research program by first summarizing its assessments of the major subprograms and then appropriately weighting the subprograms to determine the overall program ratings. Table 5 provides some background context to aid the EC in reaching overall ratings for relevance and impact. The EC could consider the items in Table 5 for each subprogram then for the overall program and assess the relevance of the re- search subprograms and program by reviewing earlier responses to the questions in Sections III.B.2. through III.B.5. (reviews of program challenges, strategic goals and objectives, inputs, and activities). Items 1-4 in Table 5 are pertinent to assess- ing relevance. To assess overall impact, the EC first needs to consider the available evidence of changes in work-related risks and adverse effects and external factors related to the changes. The EC should review the responses to the questions in Sections III.B.6. through III.B.8. (reviews of outputs, intermediate outcomes, and end outcomes) and systematically assess the impact of the research program and its subprograms. Items 5-7 in Table 5 will be helpful. The EC should evaluate separately the impact of the research and the impact of transfer activities. Transfer activities occur in two contexts: NIOSH efforts to translate intellectual products into practice and stakeholder efforts to integrate NIOSH results into the workplace. High impact assessments require the EC’s judgment that the research program has contributed to outcomes; for example, outcomes have occurred earlier than they would have or are better than they would have been in the absence of the research program, TABLE 5 Background Context for Program Relevance and Impact Assess the following for each subprogram: 1. Relevance of current and recently completed research and transfer activities to objective improvements in workplace safety and health. 2. Contributions of NIOSH research and transfer activities to changes in work-related practices and reduction in workplace exposures, illnesses, or injuries. 3. Contributions of NIOSH research and transfer activities to improvements in work-related practices. 4. Contributions of NIOSH research to productivity, security, or environmental quality (beneficial side effects). 5. Evidence of reduction of risk in the workplace (intermediate outcome). 6. Evidence of reduction in workplace exposure, illness, or injuries (end outcome). 7. Evidence of external factors that prevented translation of NIOSH research results into intermediate or end outcomes.

OCR for page 137
t R au m at I C I n j u Ry R e s e a R C h nIosh 76 at or outcomes would have occurred were it not for external factors beyond NIOSH’s control or ability to plan around. The EC must assign one overall integer score for the relevance of the research program to the improvement of occupational safety and health and one overall integer score for the impact of the program on the improvement of occupational safety and health. The EC will use its expert judgment, summary assessment of research-program elements, and any appropriate information to arrive at those two scores. In light of substantial differences among the types of research programs that will be reviewed and the challenge to arrive at a summative evaluation of both relevance and impact, the FC chose not to construct an algorithm to produce the two final ratings. Relevance and impact scores will be based on five-point categorical scales established by the FC (see Boxes 2 and 3) in which 1 is the lowest and 5 the highest rating. The FC has made an effort to establish mutually exclusive rating categories in the scales. When the basis of a rating fits more than one category, the highest applicable score should be assigned. It is up to the EC to determine how individual subprograms should influence final scores. Single integer values should be assigned. Final program ratings will consist of integer scores for relevance and impact and prose justification of the scores. Box 2 includes the criteria for scoring the overall relevance of the NIOSH research program. As discussed in previous sections, numerous factors can be con- sidered in assessing relevance. The scoring criteria focus on two: the EC assessment BOX 2 Scoring Criteria for Relevance 5 = Research is in high-priority subject areas and NIOSH is significantly engaged in appro- priate transfer activities for completed research projects/reported research results. 4 = Research is in priority subject areas and NIOSH is engaged in appropriate transfer activities for completed research projects/reported research results. 3 = Research is in high priority or priority subject areas, but NIOSH is not engaged in appropriate transfer activities; or research focuses on lesser priorities but NIOSH is engaged in appropriate transfer activities. 2 = Research program is focused on lesser priorities and NIOSH is not engaged in or plan- ning some appropriate transfer activities. 1 = Research program is not focused on priorities and NIOSH is not engaged in transfer activities.

OCR for page 137
appendIx a 77 BOX 3 Scoring Criteria for Impact 5 = Research program has made major contribution(s) to worker health and safety on the basis of end outcomes or well-accepted intermediate outcomes. 4 = Research program has made some contributions to end outcomes or well-accepted intermediate outcomes. 3 = Research program activities are ongoing and outputs are produced that are likely to result in improvements in worker health and safety (with explanation of why not rated higher). Well accepted outcomes have not been recorded. 2 = Research program activities are ongoing and outputs are produced that may result in new knowledge or technology, but only limited application is expected. Well accepted outcomes have not been recorded. 1 = Research activities and outputs do not result in or are NOT likely to have any application. of whether the program appropriately sets priorities among research needs and the EC assessment of how engaged the program is in appropriate transfer activities. Table 6 provides some guidance regarding how the EC may weight research priori- ties and transfer levels when determining relevance scores. The EC will consider both completed research and research that is in progress and related to likely future improvements in its assessment of relevance. The EC should keep in mind how well the program has considered the frequency and severity of the problems being addressed; whether appropriate attention has been directed to both sexes, vulnerable populations, or hard-to-reach workplaces; and whether the differ- ent needs of large and small businesses have been accounted for. It is up to the EC to determine how to consider external factors in assigning program scores. Box 3 includes the criteria established for the rating of impact. In general, the EC will consider completed research outputs during the assessment of impact. In assigning a score for impact, it is important to recognize that a “major contribu- tion” (required for a score of 5) does not imply that the NIOSH program was solely responsible for observed improvements in worker health and safety. Many factors may be required to effect improvements. The EC could say that NIOSH made “major contributions” if the improvements would not have occurred when they did without NIOSH efforts.

OCR for page 137
t R au m at I C I n j u Ry R e s e a R C h nIosh 78 at TABLE 6 Guidance for Weighting Research Priority and Engagement in Appropriate Transfer Activities in the Application of Relevance Score Assessment of Research Priority Engagement in Applicable Transfer Activities Applicable Score High priority Significantly engaged 5 High priority Engaged 4 High priority Not engaged 3 Priority Significantly engaged 4 Priority Engaged 4 Priority Not engaged 3 Lesser priority Significantly engaged 3 Lesser priority Engaged 3 Lesser priority Not engaged 2 Not focused on priorities Significantly engaged 2 Not focused on priorities Engaged 2 Not focused on priorities Not engaged 1 The FC has some concern that the imposed scoring criteria for impact might be considered a promotion of the conventional occupational-health research para- digm that focuses on health-effects and technology research without much empha- sis on the socioeconomic, policy, surveillance, and diffusion research (as opposed to diffusion activities) needed to effect change. The EC should remember that not all intermediate outcomes occur in the workplace. Important outcomes that NIOSH can effect also occur much farther out on the causal chain. NIOSH, for example, has an important role to play in generating knowledge that may contribute to changing norms in the insurance industry, in health-care practice, in public-health practice, and in the community at large. The EC may find that some of those issues need to be addressed and considered as external factors that facilitate or limit application of more traditional research findings. Given the rapidly changing nature of work and the workforce and some of the intractable problems in manufacturing, min- ing, and some other fields, the EC is encouraged to think beyond the traditional paradigm. III.C. Assessment of NIOSH Process for Targeting Priority Research Needs and Committee Assessment of Emerging Issues The second charge to the EC is the assessment of the research program’s effec- tiveness in targeting new research and identifying emerging issues in occupational safety and health most relevant to future improvements in workplace protection. The EC is also asked to provide a qualitative narrative assessment of the program’s

OCR for page 137
appendIx a 79 efforts and to make suggestions about emerging issues that the program should be prepared to address. Among the most challenging aspects of research in illness and injury prevention are the identification of new or emerging needs or trends and the formulation of a research response that appropriately uses scarce resources in anticipation of them. The EC should review the procedures that NIOSH and the research program have in place to identify needed research relevant to the NIOSH mission and should review the success that NIOSH has had in identifying and addressing research related to emerging issues. It should examine leading indicators from appropriate federal agencies, such as EPA, the Department of Labor, the National Institute of Standards and Technology, NIH, DOD, and the Department of Commerce. Those indicators should track new technologies, new products, new processes, and disease or injury trends. One source of information deserving particular attention is NIOSH HHE reports. The HHE program offers a potential mechanism for identifying emerging research needs that could be incorporated as input into each of the programs evalu- ated. The EC should determine whether the program under review appropriately considers pertinent HHE investigation findings. Additional emerging issues may be revealed through consideration of NIOSH and the NIOSH-funded FACE reports, the AOEC reports, the U.S. Chemical Safety Board investigations, and SENSOR and other state-based surveillance programs. Appropriate federal advisory commit- tees and other stakeholder groups should also be consulted to provide qualitative information. The EC should systematically assess how the research program and its subprograms target new research by evaluating each subprogram for the items listed in Table 7. The EC will have to determine how best to weight subprogram contributions in the program’s targeting of new research. TABLE 7 Targeting of New Research and Identification of Emerging Issues Assess the following for each subprogram: 1. Past and present effectiveness in targeting most relevant research needs. 2. Effectiveness in targeting research in fields most relevant to future improvements in occupational safety and health. 3. Contribution of NIOSH research to enhancement of capacity in government or other research institutions.

OCR for page 137
t R au m at I C I n j u Ry R e s e a R C h nIosh 80 at Questions to Guide the Evaluation Committee 1. What information does NIOSH review to identify emerging research needs? a. What is the process for review? b. How often does the process take place? c. How are NIOSH staff scientists and NIOSH leadership engaged? d. What is the process for moving from ideas to formal planning and resource allocation? 2. How are stakeholders involved? a. What advisory or stakeholder groups are asked to identify emerg- ing research targets? b. How often are such groups consulted, and how are suggestions followed up? 3. What new research targets have been identified for future development in the program under evaluation? a. How were they identified? b. Were lessons that could help to identify other emerging issues learned? c. Does the EC agree with the issues identified and selected as im- portant and with the NIOSH response, or were important issues overlooked? d. Is there evidence of unwise expenditure of resources on unimport- ant issues? The EC members should use their expert judgment both to evaluate the emerging research targets identified by NIOSH and to provide recommendations to NIOSH regarding additional research that NIOSH has not yet identified. Recom- mendations should include a brief statement of their rationale. IV. EVALUATION COMMITTEE REPORT TEMPLATE Consistency and comparability among EC report formats is desirable, but the FC recognizes that each NIOSH research program is different and that each EC is independent. The outline provided in Box 4 flows from the FC’s review of NIOSH’s generalized logic model (Figure 1), the evaluation flowchart (Figure 2), and the assessment model described earlier in this document. The EC should feel free to use or adapt this outline as necessary when organizing its final report. The FC encourages each EC to look at prior EC reports for organizational ideas.

OCR for page 137
appendIx a 8 BOX 4 Suggested Outline for Evaluation Committee Reports I. Introduction This section should be a brief descriptive summary of the history of the program (and subprograms) being evaluated with respect to pre-NORA, NORA 1, and current and future plans of the research program presented by NIOSH. It should present the context for the research on safety and health; goals, objectives, and resources; groupings of subprograms; and any other important pertinent information. (A list of the NIOSH materials reviewed should be provided in Appendix C.) II. Evaluation of Programs and Subprograms (Charge 1) A. Evaluation summary (should include a brief summary of the evaluation with respect to impact and relevance, scores for impact and relevance, and summary statements). B. Strategic goals and objectives: should describe assessment of the program and sub- programs for relevance. C. Review of inputs: should describe adequacy of inputs to achieve goals. D. Review of activities: should describe assessment of the relevance of the activities. E. Review of research-program outputs: should describe assessment of relevance and potential usefulness of the research program. F. Review of intermediate outcomes and causal impact: should describe assessment of the intermediate outcomes and the attribution to NIOSH; should include the likely impacts and recent outcomes in the assessment. G. Review of end outcomes: should describe the end outcomes related to health and safety and provides an assessment of the type and degree of attribution to NIOSH. H. Review of other outcomes: should discuss health and safety impacts that have not yet occurred; beneficial social, economic, and environmental outcomes; and international dimensions and outcomes. I. Summary of ratings and rationale. III. NIOSH Targeting of New Research and Identification of Emerging Issues (Charge 2) The EC should assess the progress that the NIOSH program has made in targeting new research in occupational safety and health. The EC should assess whether the NIOSH pro- gram has identified important emerging issues that appear especially important in terms of relevance to the mission of NIOSH. The EC should respond to NIOSH’s perspective and add its own recommendations. IV. Recommendations for Program Improvement On the basis of the review and evaluation of the program, the EC may provide recommen- dations for improving the relevance of the NIOSH research program to health and safety conditions in the workplace and the impact of the research program on health and safety in the workplace. Appendix A — Framework Document Appendix B — Methods and Information-Gathering Appendix C — List of NIOSH and Related Materials Collected in the Process of the Evaluation