Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
A Framework for the Review of Research Programs of the National Institute for Occupational Safety and Health* T his is the second version of a document prepared by the National Academies Committee for the Review of NIOSH Research Programs also referred to as the Framework Committee. This document is not a formal report of the National Academiesârather, it is a framework proposed for use by multiple National Academies evaluation committees to review up to 15 National Institute for Occupational Safety and Health (NIOSH) research programs. It is a working document subject to modification by the Framework Committee on the basis of responses received from evaluation-committee members, NIOSH, stakeholders, and the general public during the course of the assessments. â*Version of 8/10/07. âMembers of the committee at the time this version was produced were David Wegman, Chair (University of Massachusetts Lowell School of Health and Environment), William Bunn III (Inter- national Truck and Engine Corporation), Carlos Camargo (Harvard Medical School), Susan ÂCozzens (Georgia Institute of Technology), Letitia Davis (Massachusetts Department of Public Health), James Dearing (Kaiser Permanente-Colorado), Fred Mettler, Jr. (University of New Mexico School of Medicine), Franklin Mirer (Hunter College School of Health Sciences), Jacqueline Nowell (United Food and Commercial Workers International Union), Raja Ramani (Pennsylvania State University), Jorma Rantanen (International Commission on Occupational Health), Rosemary Sokas (University of Illinois at Chicago School of Public Health), Richard Tucker (Tucker and Tucker Consultants, Inc., and University of Texas at Austin), and James Zuiches (North Carolina State University). Sammantha Magsino (National Academies staff) was the study director. Joseph Wholey (University of Southern California), former committee member, contributed to the first version of this document. Part V includes brief biographies of current committee members. 137
138 T r a u m a t ic I n j u r y R e s e a rc h at N I OSH This version reflects several significant changes to the original framework document (version 12/19/05) that was used to guide the work of the first four evaluation committees (Hearing Loss; Mining; Agriculture, Forestry, and Fish- ing; and Respiratory Disease). Changes were made in response to feedback from members and staff of these committees, as well as other comments on the original framework, in order to make the document more useful to evaluation committees as they carry out their work. In particular, the following changes were made to the framework document during the revision process: â¢ The wording of some of the relevance and impact scores were edited to make the wording more precise and to reduce situations where the original scores were non-unique or overlapping (revised scoring cri- teria are given in Boxes 2 and 3). â¢ A new table was added to provide explicit guidance to evaluation com- mittees on how to weigh differences in the observed levels of âresearch priorityâ and âengagement in appropriate transfer activitiesâ in arriv- ing at a single integer score for relevance (see Table 6). â¢ The guidance on scoring was clarified to make more explicit that all scores are to be given as integers. â¢ The NIOSH logic model was updated (see Figure 1). â¢ The table on evaluation committee information needs (Table 2) was reorganized to be more consistent with the NIOSH logic model, and additional information needs identified by the first set of evaluation committees were added. â¢ A worksheet to assist with the development of scores has been deleted and key components of the worksheet have been incorporated into appropriate sections throughout the document. â¢ The organization of the document was modified to more closely follow the revised statement of task and to improve readability. â¢ A number of sections of text originally presented in outline form were modified in tables or boxes to make the information more accessible. This second version of the framework document remains a working document subject to further modification by the Framework Committee on the basis of Âinput received from evaluation committee members, NIOSH, stakeholders, and the gen- eral public during the course of the assessments.
A p p e n d i x A 139 CONTENTS Abbreviations and Acronyms I. Introduction I.A. Overview of Charge to Evaluation Committees I.B. Evaluation Committees I.C. NIOSH Strategic Goals and Operational Plan I.D. Evaluation Committeesâ Information Needs I.E. Prior Evaluations II. Summary of Evaluation Process II.A. The Evaluation Flowchart (Figure 2) II.B. Steps in Program Evaluation I II. Evaluation of a NIOSH Research ProgramâThe Process III.A. Analysis of External Factors Relevant to the NIOSH Research Program III.B. Evaluating NIOSH Research Programs by Using the Flowchart III.B.1. Identifying the Period for Evaluation III.B.2. Identifying Major Challenges (Figure 2, Circle) III.B.3. Analysis of Research-Program Strategic Goals and Objectives (Figure 2, Box A) III.B.4. Review of Inputs (Figure 2, Box B) III.B.5. Review of Activities (Figure 2, Box C) III.B.6. Review of Outputs (Figure 2, Box D) III.B.7. Review of Intermediate Outcomes (Figure 2, Box E) III.B.8. Review of End Outcomes (Figure 2, Box F) III.B.9. Review of Potential Outcomes III.B.10. Summary Evaluation Ratings and Rationale III.C. Assessment of NIOSH Process for Targeting Priority Research Needs and Committee Assessment of Emerging Issues IV. Evaluation Committee Report Template Figure 1 The NIOSH operational plan presented as a logic model Figure 2 Flowchart for the evaluation of the NIOSH research program Table 1 NORA High-Priority Research by Category Table 2 Evaluation Committee Information Needs Table 3 Examples of NIOSH Program Research and Transfer Activities Table 4 Examples of Research-Program Outputs to Be Considered Table 5 Background Context for Program Relevance and Impact Table 6 Guidance for Weighting Research Priority and Engagement in Appropriate Transfer Activities in the Application of Relevance Score
140 T r a u m a t ic I n j u r y R e s e a rc h at N I OSH Table 7 Targeting of New Research and Identification of Emerging Issues Box 1 The Evaluation Process Box 2 Scoring Criteria for Relevance Box 3 Scoring Criteria for Impact Box 4 Suggested Outline for Evaluation Committee Reports
A p p e n d i x A 141 ABBREVIATIONS AND ACRONYMS ABLES Adult Blood Lead Epidemiology and Surveillance AOEC Association of Occupational and Environmental Clinics BLS Bureau of Labor Statistics CDC Centers for Disease Control and Prevention CSTE Council of State and Territorial Epidemiologists DOD U.S. Department of Defense EC Evaluation Committee EPA Environmental Protection Agency FACE Fatality Assessment Control and Evaluation FC Framework Committee HHE Health Hazard Evaluation MSHA Mine Safety and Health Administration NIH National Institutes of Health NIOSH National Institute for Occupational Safety and Health NORA National Occupational Research Agenda NORA1 National Occupational Research Agenda 1996-2005 NORA2 National Occupational Research Agenda 2005-forward OSH Review Occupational Safety and Health Review Commission Commission OSHA Occupational Safety and Health Administration OSHAct Occupational Safety and Health Act of 1970 PART Performance Assessment Rating Tool PEL permissible exposure limit RFA request for applications SENSOR Sentinel Event Notification System of Occupational Risks
142 T r a u m a t ic I n j u r y R e s e a rc h at N I OSH TMT tools, methods, or technologies USDA U.S. Department of Agriculture
A p p e n d i x A 143 I. INTRODUCTION In September 2004, the National Institute for Occupational Safety and Health (NIOSH) contracted with the National Academies to conduct a review of NIOSH research programs. The goal of this multiphase effort is to assist NIOSH in in- creasing the impact of its research efforts that are aimed at reducing workplace illnesses and injuries and improving occupational safety and health. The National Academies assigned the task to the Division on Earth and Life Studies and the Institute of Medicine. The National Academies appointed a committee of 14 members, including persons with expertise in occupational medicine and health, industrial health and safety, industrial hygiene, epidemiology, civil and mining engineering, sociology, program evaluation, communication, and toxicology; representatives of industry and of the workforce; and a scientist experienced in international occupational- health issues. The Committee on the Review of NIOSH Research Programs, re- ferred to as the Framework Committee (FC), prepared the first version of this document during meetings held on May 5-6, July 7-8, and August 15-16, 2005. This second version was finalized after the Framework Committeeâs May 30-31, 2007, meeting, based on feedback received on the framework from the first two independent evaluation committees, NIOSH leadership, and National Academiesâ staff, as well as discussions during an earlier FC meeting in April 2006. This document is not a report of the National Academies; rather, it presents the evaluation framework developed by the FC to guide and provide common structure for the reviews of as many as 15 NIOSH programs during a 5-year period by independent evaluation committees (ECs) appointed by various divisions and boards of the National Academies. It is a working document to be shared with NIOSH and the public. This version has been modified by the FC on the basis of responses from the ECs, NIOSH, NIOSH stakeholders, and the public; and it may be modified again. It is incumbent on the ECs to consult with the FC if portions of the evaluation framework presented here are inappropriate for specific programs under review. I.A. Overview of Charge to Evaluation Committees At the first meeting of the FC, Lewis Wade, NIOSH senior science adviser, emphasized that a review of a NIOSH program should focus on the programâs relevance to and impact on health and safety in the workplace. In developing a framework, the FC considered the following elements of the charge to the ECs:
144 T r a u m a t ic I n j u r y R e s e a rc h at N I OSH 1. Assessment of the programâs contribution, through occupational safety and health research, to reductions in workplace hazardous exposures, illnesses, or injuries through a. an assessment of the relevance of the programâs activities to the improvement of occupational safety and health, and b. an evaluation of the impact that the programâs research has had in reducing work-related hazardous exposures, illnesses, and injuries. The evaluation committee will rate the performance of the program for its relevance and impact using an integer score of 1-5. Impact may be assessed directly (for example, on the basis of reductions in ill- nesses or injuries) or, as necessary, by using intermediate outcomes to estimate impact. Qualitative narrative evaluations should be included to explain the numerical ratings. 2. Assessment of the programâs effectiveness in targeting new research areas and identifying emerging issues in occupational safety and health most relevant to future improvements in workplace protection. The committee will provide a qualitative narrative assessment of the pro- gramâs efforts and suggestions about emerging issues that the program should be prepared to address. I.B. Evaluation Committees Individual ECs will be formed in accordance with the rules of the National Academies for the formation of balanced committees. Each EC will comprise per- sons with expertise appropriate for the specific NIOSH research program under review and may include representatives of stakeholder groups (such as labor unions and industry), experts in technology and knowledge transfer, and program evalu- ation. The EC will gather appropriate information from the sponsor (the NIOSH research program under review), stakeholders affected directly by NIOSH program research, and relevant independent parties. Each EC will consist of about 10 mem- bers, will meet about three times, and will prepare a report. The National Academies will deliver the report to NIOSH within 9 months of the first meeting of the EC. EC reports are subject to the National Academies report-review process. I.C. NIOSH Strategic Goals and Operational Plan As a prelude to understanding the NIOSH strategic goals and operational plan, NIOSH research efforts should be understood in the context of the Occupational
A p p e n d i x A 145 Safety and Health Act (OSHAct), under which it was created. The OSHAct identifies workplace safety and health as having high national priority and gives employers the responsibility for controlling hazards and preventing workplace injury and ill- ness. The act creates an organizational framework for doing that, assigning comple- mentary roles and responsibilities to employers and employees, the Occupational Safety and Health Administration (OSHA), the states, the OccuÂpational Safety and Health (OSH) Review Commission, and NIOSH. The act recognizes NIOSHâs role and responsibilities to be supportive and indirect. NIOSH research, training programs, criteria, and recommendations are intended to be used to inform and assist those more directly responsible for hazard control (OSHAct Sections 2b, 20, and 22). Section 2b of the OSHAct describes 13 interdependent means of accom- plishing the national goal, one of which is âby providing for research . . . and by developing innovative methods . . . for dealing with occupational safety and health problems.â Sections 20 and 22 give the responsibility for that research to NIOSH. NIOSH is also given related responsibilities, including the development of criteria to guide prevention of work-related injury or illness; development of regulations for reporting on employee exposures to harmful agents; establish- ment of medical examinations, programs, or tests to determine illness incidence and susceptibility; publication of a list of all known toxic substances; assessment of potential toxic effects or risks associated with workplace exposure in specific settings; and conduct of education programs for relevant professionals to carry out the OSHAct purposes. NIOSH is also responsible for assisting the secretary of labor regarding education programs for employees and employers in hazard recognition and control. The NIOSH mission is âto provide national and world leadership to prevent work-related illness, injury, disability, and death by gathering information, con- ducting scientific research, and translating the knowledge gained into products and services.â To fulfill its mission, NIOSH has established the following strategic goals: â¢ Goal 1: Conduct research to reduce work-related illnesses and injuries. Â° Track work-related hazards, exposures, illnesses, and injuries for prevention. Â° Generate new knowledge through intramural and extramural r Â esearch programs. âSee http://www.cdc.gov/niosh/docs/strategic/.
146 T r a u m a t ic I n j u r y R e s e a rc h at N I OSH Â° Develop innovative solutions for difficult-to-solve problems in high-risk industrial sectors. â¢ Goal 2: Promote safe and healthy workplaces through interventions, recommendations, and capacity building. Â° Enhance the relevance and utility of recommendations and guidance. Â° Transfer research findings, technologies, and information into practice. Â° Build capacity to address traditional and emerging hazards. â¢ Goal 3: Enhance global workplace safety and health through inter- national collaborations. Â° Take a leadership role in developing a global network of occupa- tional health centers. Â° Investigate alternative approaches to workplace illness and injury reduction and provide technical assistance to put solutions in place. Â° Build global professional capacity to address workplace hazards through training, information sharing, and research experience. In 1994, NIOSH embarked on a national partnership effort to identify research priorities to guide occupational health and safety research for the next decade. The National Occupational Research Agenda (NORA) identified 21 high-priority research subjects (see Table 1). The NORA was intended not only for NIOSH but for the entire occupational health community. In the second decade of the NORA, NIOSH is working with its partners to update the research agenda, using an ap- proach based on industry sectors. NIOSH and its partners are working through sector research councils to establish sector-specific research goals and objectives. The emphasis is on moving research to practice in workplaces through sector-based partnerships. Figure 1 is the NIOSH operational plan, presented as a logic model, of the path from inputs to outcomes for each NIOSH research program. The FC adapted the model to develop its framework. NIOSH will provide similar logic models ap- propriate to each research program evaluated by an EC. âDeveloped by NIOSH with the assistance of the RAND Corporation.
A p p e n d i x A 147 TABLE 1â NORA High-Priority Research by Category Category Priority Research Area Disease and injuryÂ Allergic and irritant dermatitis Asthma and chronic obstructive pulmonary disease Fertility and pregnancy abnormalities Hearing loss Infectious diseases Low-back disorders Musculoskeletal disorders of upper extremities Trauma Work environment and workforceÂ Emerging technologies Indoor environment Mixed exposures Organization of work Special populations at risk Research tools and approachesÂ Cancer research methods Control technology and personal protective equipment Exposure-assessment methods Health-services research Intervention-effectiveness research Risk-assessment methods Social and economic consequences of workplace illness and injury Surveillance research methods I.D. Evaluation Committeesâ Information Needs Each NIOSH program under review will provide information to the relevant EC, including that outlined in Table 2. The EC may request additional informa- tion of NIOSH as needed, and NIOSH should provide it as quickly as is practical. NIOSH should consider organizing the information listed in Table 2 by subpro- gram or program as appropriate and to the extent possible. In addition to the information provided by NIOSH, the EC should indepen- dently collect additional information that it deems necessary for evaluation (for example, the perspectives of stakeholders, such as OSHA, MSHA, unions and work- forces, and industry). In conducting the review, the EC should continually examine how individual projects or activities contribute to the impact and relevance of a program as a whole.
148 New Figure A-1, bitmapped FIGURE 1â The NIOSH operational plan presented as a logic model.
A p p e n d i x A 149 TABLE 2â Evaluation Committee Information Needs â¢ Program background and resources: Â° Program history. Â° Major program challenges. Â° Program strategic goals and objectives, past (for period under review) and current. Â° Major subprograms (if appropriate). Â° Results of previous program reviews (for example, annual review by NIOSH leadership team or external scientific program reviews). Â° External factors affecting the program. â¢ Interactions with stakeholders and with other NIOSH programs: Â° The role of program research staff in NIOSH policy-setting, OSHA and MSHA standard- setting, voluntary standard-setting and other government policy functions. Â° Interactions and working relationships with other NIOSH programs. Â° Identification of other institutions and research programs with overlapping or similar portfolios and an explanation of the relationship between NIOSH activities and those of other institutions. Â° Key partnerships with employers, labor, other government organizations, academic institutions, nonprofit organizations, and international organizations. â¢ Program inputs: Â° Program resources (also called production inputs). Ã Funding by year for period under review. Ã Funding by objective or subprogram. Ã Program staffing, FTEâs, and laboratory facilities, by subprogram (if indicated). Ã Percentage of program budget that is discretionary (beyond salaries). Ã Percentage of program budget that is earmarked. Ã Contributions from other agencies (in kind or funds). Â° Planning inputs. Ã Surveillance data, inputs from the Health Hazard Evaluation (HHE) or Fatality Assessment Control and Evaluation (FACE) program, or intramural and extramural research findings that influenced program goals and objectives. Ã Planning inputs from stakeholders, for example, advisory groups, NORA teams, and professional, industry, and labor groups (specify if any input from groups representing small business or vulnerable populations). Ã Related OSHA, Mine Safety and Health Administration (MSHA) strategic plans, or other input. Ã Process for soliciting and approving intramural research ideas. Ã Process for soliciting and approving program-supported extramural research activities. â¢ Program activities (more details provided in Table 3): Â° Intramural. Ã Surveillance activities. Ã Research activities (projects). continued
150 T r a u m a t ic I n j u r y R e s e a rc h at N I OSH TABLE 2â Continued Ã Transfer activities to encourage implementation of research results for improved occupational safety and health (for example, information dissemination, technical assistance, and technology and knowledge transfer). Ã Key collaborations in intramural activities (for example, with other government agencies, academe, industry, and unions). Â° Extramural funded by NIOSH. Ã Requests for applications (RFAs) developed by program. Ã Funded projects: grants, cooperative agreements, and contracts, such as â Surveillance activities. â Research activities. â Transfer activities. â Capacity-building activities. â¢ Outputs (products of the research programâmore details provided in Table 4): Â° Intramural. Ã Peer-reviewed publications, agency reports, alerts, and recommendations. Ã Databases, Web sites, tools, and methods (including education and training materials). Ã Technologies developed and patents. Ã Sponsored conferences and workshops. Â° Extramural (to the extent practical). â¢ Intermediate outcomes: Â° Standards or guidelines issued by other agencies or organizations based in whole or in part on NIOSH research. Â° Adoption and use of control or personal protective technologies developed by NIOSH. Â° Evidence of industry, employer, or worker behavioral changes in response to research outputs. Â° Use of NIOSH products by workers, industry, occupational health and safety professionals, health care providers, and so on (including internationally). Â° NIOSH Web-site hits and document requests. Â° Unique staff or laboratory capabilities that serve as a national resource. Â° Other intermediate outcomes. â¢ End outcomes: Â° Data on program impact on rates and numbers of injuries and illnesses and exposures in the workplace (including trend data, if available). Â° Documentation of workplace risk reduction (quantitative, qualitative, or both). â¢ Description of current processes for setting research priorities and identifying emerging issues in the workplace.
A p p e n d i x A 151 I.E. Prior Evaluations Several NIOSH programs have already been evaluated by internal and external bodies. The evaluations may have been part of an overall assessment of NIOSH, such as the 2005 Performance Assessment Rating Tool (PART) review, or the evaluation of specific research program elements, such as any external scientific- program review. NIOSH should inform of, and the ECs should review, all prior evaluations of the program under review as an aid to understanding the evolution of the program and its elements. The EC evaluations, however, are independent of prior reviews and evaluations. II. SUMMARY OF EVALUATION PROCESS The ECs will assess the relevance and impact of NIOSH research programs. In conducting their evaluations, the ECs should ascertain whether NIOSH is doing the right things (relevance) and whether these things are improving health and safety in the workplace (impact). II.A. The Evaluation Flowchart (Figure 2) To address its charge, the FC simplified the logic model of Figure 1 into a flowchart (Figure 2) that breaks the NIOSH logic model into discrete, sequential program components to be assessed by the EC. Each component of Figure 2 is a Â ddressed in greater detail in the indicated section of this document. The FC under- stands that the activities of any research program will not be as linear as presented in either Figures 1 or 2. The major components to be evaluated are â¢ major program challenges, â¢ strategic goals and objectives, â¢ inputs (such as budget, staff, facilities, the instituteâs research manage- ment, the NIOSH Board of Scientific Counselors, the NORA process, and NORA work groups), â¢ activities (efforts by NIOSH staff, contractors, and grantees, such as hazard surveillance; surveillance for injury, illness, and biomarkers of effect; exposure-measurement research; safety-systems research; injury-prevention research; health-effects research; intervention 4The PART focuses on assessing program-level performance and is one of the measures of success of the budget and performance integration initiative of the presidentâs management agenda (see CDC Occupational Safety and Health at http://www.whitehouse.gov/omb/budget/fy2006/pma/hhs.pdf).
152 bitmap image New Figure A-2.eps FIGURE 2â Flowchart for the evaluation of the NIOSH research program.
A p p e n d i x A 153 r Â esearch; health-services research; and technology and knowledge transfer activities), â¢ outputs (NIOSH products, such as publications, reports, conferences, databases, tools, methods, guidelines, recommendations, education and training, and patents), â¢ intermediate outcomes (responses by NIOSH stakeholders to NIOSH products, such as public or private policy change, training and educa- tion in the form of workshop or seminar attendance, self-reported use or repackaging of NIOSH data by stakeholders, adoption of NIOSH- developed technologies, implemented guidelines, licenses, and reduc- tion in workplace hazardous exposure), and â¢ end outcomes (such as reduction in work-related injuries or illnesses or hazardous exposures in the workplace). The flowchart summarizes the FCâs vision of how a program evaluation should occur. In evaluating each program or major subprogram, the EC must collect, analyze, and evaluate information on items described in each of the boxes of Figure 2, regardless of management structure (such as linear or matrix). The FC recognizes that the components of any program will not fit perfectly in any category in ÂFigure 1 or 2. For example, training and development programs were appropri- ately defined as outputs by NIOSH in the logic model (Figure 1), but the FC finds more value in focusing on the responses to these outputs as intermediate outcomes (Figure 2, Box E) in the flowchart. The committee further recognizes that matrix organizations may have little control over the input portion of the logic model and that matrix program management may have fewer resources of its own on which to base its decisions. Following the suggested evaluation procedures, however, should ensure a desired level of consistency and comparability among all the ECs. Drawing on the program logic model, the flowchart, and EC membersâ e Â xpertise, the ECs will delineate important inputs and external factors affecting the NIOSH research programâs agenda and the consequences of NIOSH research activities. Examples of external factors are research activities of industry and other federal agencies and the political and regulatory environment. For purposes of this review, the results of inputs and external factors are the program research activities, outputs, and associated transfer activities that may result in intermediate outcomes and possibly end outcomes. II.B. Steps in Program Evaluation The FC concludes that useful evaluation requires a disciplined focus on a small number of questions or hypotheses typically related to program goals, performance
154 T r a u m a t ic I n j u r y R e s e a rc h at N I OSH criteria, and performance standards; a rigorous method of answering the questions or testing the hypotheses; and a credible procedure for developing qualitative and quantitative assessments. The evaluation process developed by the FC is summa- rized in Box 1 and described in detail in Section III of this document. III. EVALUATION OF A NIOSH RESEARCH PROGRAMâTHE PROCESS III.A. Analysis of External Factors Relevant to the NIOSH Research Program As depicted in the logic model (Figure 1), reduction in injury and illness (end outcomes) or in exposure (intermediate outcome) is affected by stakeholder activi- ties (external factors). Actions of those in labor, industry, regulatory entities, and others beyond NIOSHâs control are necessary for the implementation of NIOSH recommendations. Implementation of research findings may depend on existing or future policy considerations. External factors may be considered as forces beyond the control of NIOSH that may affect the evolution of a program. External factors influence NIOSHâs progress through all phases of the logic model and flowchart; from inputs to end outcomes (see Figures 1 and 2). Identification of external factors by an EC is essen- tial because it provides the context for evaluation of the NIOSH program. External factors may be best assessed on the basis of the expert judgment of EC members who have knowledge of the field of research. Information regarding external fac- tors should also be sought from NIOSH, OSHA, and MSHA staff and from other stakeholders. The EC, however, may choose additional approaches to assess external factors. NIOSH should identify and describe external factors early in the evaluation sequence (see Table 2). Factors external to NIOSH might have been responsible for achieving some outcomes or might have presented formidable obstacles. The EC must address both possibilities. Some external factors may involve constraints on research activities related to target populations, methodologic issues, and resource availability. ECs might examine whether or not the following are true: â¢ Projects addressing a critical health need are technologically feasible. However, a workforce of appropriate size and with appropriate duration and distribution of exposure for measuring a health effect may not exist; for example, no population of workers has been exposed for 30 years to formaldehyde at the current OSHA permissible exposure limit (PEL), so the related cancer mortality cannot yet be directly assessed. â¢ Research is inhibited because NIOSH investigators are unable to Âaccess an adequate study population. Under current policy, NIOSH must
A p p e n d i x A 155 BOX 1 The Evaluation Process 1. Gather appropriate information from NIOSH and other sources (see Table 2). 2. Determine timeframe to be covered in the evaluation (see III.B.1.). 3. Identify major program area challenges and objectives (see III.B.2.). All NIOSH research programs are designed to be responsive to present or future work- place safety and health issues. Each research program should have its own objectives. Each EC will provide an independent assessment of the major workplace health and safety problems related to the program under review and determine whether they are consistent with the programâs stated goals and objectives. 4. Identify subprograms and major projects in the research program. Each EC must determine how to disaggregate a program to achieve a manageable and meaningful evaluation of its components, and of the overall program. A program may need to be broken down into several recognizable subprograms or major projects if an effective evaluation is to be organized. It may be advantageous for an EC to disag- gregate a program into subprograms that NIOSH identifies. 5. Evaluate the subprogram components sequentially (see III.B.2. through III.B.8.), using the flowchart (Figure 2) as a guide. This will involve a qualitative assessment of each component of the research program. ECs will use professional judgment to answer questions and follow the guidance pro- vided by the FC. 6. Evaluate the research programâs potential outcomes that are not yet appreciated (see III.B.9.). 7. Evaluate the important subprogram outcomes specifically for contributions to improve- ments in workplace safety and health. Guidance is provided with specific items for consideration (see III.B.10.). 8. Evaluate and score the overall program for relevance (see III.B.10.). Final program ratings will consist of an integer score and discussion of its rationale. 9. Evaluate and score the overall program for impact (see III.B.10.). Final program ratings will consist of an integer score and discussion of its rationale. 10. Identify success in targeting priority research and emerging issues (see III.C.). The EC should briefly discuss its assessment of the NIOSH programâs process for determining priorities for research and emerging workplace issues. The ECs should also independently identify emerging workplace issues for which the NIOSH program under review should be prepared. 11. Prepare report by using the template provided in Section IV as a guide.
156 T r a u m a t ic I n j u r y R e s e a rc h at N I OSH e Â ither obtain an invitation by management to study a workplace or seek a judicial order to provide authority to enter a worksite. (Co- operation under court order may well be insufficient for effective research.) â¢ Research is inhibited because the work environment, materials, and historical records cannot be accessed even with management and workforce cooperation. â¢ Adequate or established methods do not exist for assessing the environment. â¢ The NIOSH contribution to a particular field of research is reduced because other institutions are working in the same field. â¢ NIOSH resources are inadequate to tackle key questions. Evaluation of the impact of NIOSH research outputs on worker health and safety may require consideration of external factors that might impede or aide implementation, measurement, and so on. ECs might consider whether or not the following are true: â¢ Regulatory end points are unachievable because of obstacles to regula- tion or because of differing priorities of the regulatory agencies. For example, there may be no implementation of recommendations for improved respiratory protection programs for health-care workers because of enforcement policies or lack of acceptance by the health- care institution administrators. â¢ A feasible control for a known risk factor or exposure is unimple- mented because the costs of implementation are too high or because current economic incentives do not favor such actions. â¢ End outcomes are unobservable because baseline and continuing sur- veillance data are not available. For example, the current incidence of occupational noise-induced hearing loss is not known although surveillance for a substantial threshold shift is feasible. (NIOSH con- ducts surveillance of work-related illnesses, injuries, and hazards, but comprehensive surveillance is not possible with existing resources.) â¢ Reductions in adverse effects of chronic exposure cannot be measured. For example, 90% of identified work-related mortality is from diseases, such as cancer, that arise only after decades of latency after first expo- sure; therefore, effects of reducing exposure to a carcinogen cannot be observed in the timeframe of most interventions. â¢ A promulgated regulation requires a technology that was developed but not widely used.
A p p e n d i x A 157 â¢ Reductions in fatal traumatic injuries occur because more-hazardous manufacturing jobs are replaced by less-hazardous knowledge-based jobs. III.B. Evaluating NIOSH Research Programs by Using the Flowchart The FC used the NIOSH logic model (Figure 1) to define the scope and stages of an EC evaluation. The evaluation of the elements in the flowchart (Figure 2) summarizes the FCâs vision of how a program evaluation should proceed. FC mem- bers also identified numerous possible factors to consider in assessing the relevance of NIOSH research-program components, including the following: â¢ The severity or frequency of health and safety hazards addressed and the number of people at risk (magnitude) for these hazards. â¢ The extent to which NIOSH research programs identify and address gender-related issues and issues of vulnerable populations: Vulner- able populations are defined as groups of workers who have biologic, social, or economic characteristics that place them at increased risk for work-related conditions or on whom inadequate data have been collected. Vulnerable populations include disadvantaged minorities, disabled persons, low-wage workers, and non-English-speakers for whom language or other barriers present health or safety risks. â¢ The extent to which NIOSH research programs address the health and safety needs of small businesses. â¢ The âlife stageâ of problems being addressed: As the health effects are understood, efforts should shift to intervention research, from efficacy to intervention, and to intervention-effectiveness research. Gaps in the spectrum of prevention need to be addressed; for example, research on exposure assessment may be necessary before the next intervention steps can be taken. â¢ The structure, in addition to the content, of the research program: A relevant research program is more than a set of unrelated research projects; it is an integrated program involving interrelated surveillance, research, and transfer activities. â¢ Appropriate NIOSH consideration of stakeholder input. The ECs may consider those and other important factors that bear on relevance as they progress through each stage of an evaluation. The following subsections are intended to guide the EC through the evaluation process and flowchart in Figure 2. Each begins with a definition of the component
158 T r a u m a t ic I n j u r y R e s e a rc h at N I OSH being evaluated, provides questions for the EC to consider during the course of its evaluation, and provides some guidance regarding the assessment of the com- ponent. The FC admittedly provides little guidance regarding the evaluation of programs that are organized in a matrix structure or programs that have large extramural research components. Because of the uniqueness of each program, each EC must determine the most reasonable way to apply the criteria established in this document. III.B.1. Identifying the Period for Evaluation By studying materials presented by the NIOSH research program and other sources, the EC will become familiar with the history of the research program b Â eing evaluated and its major subprograms, goals, objectives, resources, and other pertinent information. Having that information, the EC should choose the period most appropriate for the evaluation. EC efforts should focus on the impact and relevance of the NIOSH program in the most recent appropriate period. As a start- ing point, the ECs might consider three general timeframes: â¢ 1970-1995, the period from the founding of NIOSH to the initiation of NORA (pre-NORA period) â¢ 1996-2005 (NORA 1 period) â¢ After 2005 (NORA 2 period) Those timeframes are provided as general guidance; the period chosen for review will take into consideration suggestions from the NIOSH research pro- gram under review. It is recognized that many of the intermediate and end out- comes documented since 1996 are consequences of research outputs completed before 1996. III.B.2. Identifying Major Challenges (Figure 2, Circle) Early in the assessment process, the EC itself should identify the major work- place health and safety challenges for the research program under review. In arrivÂing at a list of challenges, the EC should rely on surveillance findings, includ- ing those of NIOSH investigations of sentinel events (through health-hazard or f Â atality-Âassessment programs), external advisory inputs, and its own expert judg- ment. The EC will then be able to compare its own assessment of workplace chal- lenges with the NIOSH program goals and objectives. The congruence between the two will be useful during the assessment of relevance.
A p p e n d i x A 159 III.B.3. Analysis of Research-Program Strategic Goals and Objectives (Figure 2, Box A) The research program goals and objectives should be evaluated with a focus on how each program goal is related to NIOSHâs agency wide strategic goals and to the program challenges identified in the step above (Section III.B.2.). The im- portance or relevance of an issue may differ from the influence of NIOSH-funded research in addressing it. The EC should recognize that NIOSH research priorities may be circumstantial (for example, congressionally funded) rather than based on NIOSHâs assessment of the state of knowledge. Questions to Guide the Evaluation Committee 1. Are the strategic goals and objectives of the program well defined and clearly described? 2. How well were program goals and objectives aligned with NORA 1 priorities during the last decade? 3. How are current program strategic goals and objectives related to cur- rent NIOSH strategy, including NORA 2? 4. Are the research program goals, objectives, and strategies relevant to the major challenges for the research program and likely to address emerging problems in the research program (as determined by the EC while addressing Section III.B.2.)? a. Did past program goals and objectives (research and dissemination and transfer activities) focus on the most relevant problems and anticipate the emerging problems in the research program? b. Do the current program goals and objectives target the most rel- evant problems? Assessment The EC should provide a qualitative assessment that discusses the relevance of the programâs goals, objectives, and strategies in relation to its major challenges. III.B.4. Review of Inputs (Figure 2, Box B) Planning inputs include input from stakeholders, surveillance and intervention data, and risk assessments. Production inputs include intramural and extramural funding, staffing, management structure, and physical facilities.
160 T r a u m a t ic I n j u r y R e s e a rc h at N I OSH The EC should examine existing intramural and extramural resources and, potentially, prior surveys or case studies that might have been developed specifically to assess progress in reducing workplace illnesses and injuries and to provide infor- mation relevant to the targeting of research to future needs. The NIOSH research program should provide the EC all relevant planning and production inputs (see below and Table 2 for examples). Planning Inputs Planning inputs can be qualitative or quantitative. Sources of qualitative inputs include the following: â¢ Federal advisory committees (such as the Board of Scientific CounÂ selors, the Mine Safety and Health Research Advisory Committee, and the National Advisory Committee on Occupational Safety and Health) â¢ NORA research partners, initial NORA stakeholder meetings, later NORA team efforts (especially strategic research plans), and the NORA Liaison Committee and federal liaison committee recommendations â¢ Industry, labor, academe, professional associations, industry asso- ciations, and the Council of State and Territorial Epidemiologists (CSTE) â¢ OSHA and MSHA strategic plans and other federal research agendas Attention should be given to how comprehensive the inputs have been and to what extent gaps in input have been identified and considered by NIOSH. Sources of quantitative inputs include the following: â¢ Intramural surveillance information, such as descriptive data on expo- sures and outcomes (appropriate data may be available from a number of NIOSH divisions and laboratories) â¢ HHEs â¢ Reports from the FACE program â¢ Extramural health-outcome and exposure-assessment data from OSHA, MSHA (both safety and health inspection data), the Bureau of Labor Statistics, the U.S. Department of Defense (DOD), and the U.S. Department of Agriculture (USDA) (fatality, injury, and illness surveil- lance data); state government partners, including NIOSH-funded state surveillance programs, such as Sentinel Event Notification System of Occupational Risks (SENSOR), Adult Blood Lead Epidemiology and
A p p e n d i x A 161 Surveillance (ABLES), and state-based FACE; and nongovernment organizations, such as the National Safety Council, the Association of Occupational and Environmental Clinics (AOEC), the American So- ciety of Safety Engineers, and the American College of Occupational and Environmental Medicine â¢ Appropriate data from investigator-initiated extramural research funded by NIOSH Production Inputs For the research program under review, NIOSH should identify portions of the NIOSH intramural budget, staff, facilities, and management that play major roles in the research program. Production inputs should be described primarily in terms of intramural research projects, relevant extramural projects (particularly cooperative agreements and contracts), HHEs, and related staff. Consideration should also be given to leveraged funds provided by such partners as the National Institutes of Health (NIH) and the Environmental Protection Agency (EPA) for joint requests for applications or program announcements; and to OSHA, MSHA, and U.S. Department of Defense (DOD) contracts with NIOSH. Assessment of inputs should include EC consideration of the degree to which allocation of funding and personnel was commensurate with the resources needed to conduct the research and the extent to which funding for the relevant intramural research activity has been limited by lack of discretionary spending beyond salaries (travel, supplies, external laboratory services, and so on). Thus, assessments should consider the adequacy of the qualitative and quantitative planning and production inputs, given the tasks at hand. Questions to Guide the Evaluation Committee 1. Do planning, production, and other input data promote program goals? 2. How well are major planning, production, and other program inputs used to support the major activities? 3. Is input obtained from stakeholders, including input representing vulnerable working populations and small businesses? 4. Are production inputs (intramural and extramural funding, staffing, management, and physical infrastructure resources) consistent with program goals and objectives?
162 T r a u m a t ic I n j u r y R e s e a rc h at N I OSH Assessment The EC should provide a qualitative assessment that discusses the quality, adequacy, and use of inputs. III.B.5. Review of Activities (Figure 2, Box C) Activities are defined as the efforts and work of a programâs staff, grantees, and contractors. For present purposes, activities of the NIOSH program under review are divided into research and transfer activities. Table 3 is intended to guide the EC and NIOSH as to the type and organization of information required to evalu- ate program activities. The table may be incomplete, and some types of research activity may not be applicable to a given NIOSH program. Research activities in- clude safety research, health-outcomes research, safety-design research, and safety- systems research. Transfer activities include information dissemination, training, technical assistance, and education designed to translate research outputs into content and formats that are designed for application in the workplace. Depend- ing on the scope of the program under review, activities may also be grouped by research-program objectives or subprograms. Conventional occupational safety and health research focuses appropriately on injury, illness, or death; on biomarkers of exposure; and on health effects of new technology, personal protective equipment, and regulations. A focus on surveillance research may be needed when available data inputs are inadequate. A focus on socioÂ economic and policy research and on diffusion research is also needed to effect change, because not all relevant intermediate outcomes occur in the workplace. NIOSH may be able to affect important outcomes farther out on the causal chain so as to influence health and safety in the workplace. Other research that might prove important in addressing NIOSHâs mission includes the following: â¢ Surveillance research to assess the degree of significant or systematic underreporting of relevant injuries, illnesses, and biomarkers â¢ Socioeconomic research on cost-shifting between worker compensa- tion and private insurance â¢ Research on methods to build health and safety capacity in community health centers that serve low-income or minority-group workers and to improve recognition and treatment of work-related conditions â¢ Transfer research to change health and safety knowledge of adolescents while they are in high school to improve the likelihood of reduced injuries as they enter the workforce
A p p e n d i x A 163 TABLE 3â Examples of NIOSH Program Research and Transfer Activities Surveillance (including hazard and injury, illness, and biomarkers of exposure or effect health surveillance and evaluation of surveillance systems) Health-effects research (illnesses, injuries, and biomarkers): Epidemiology Toxicology Physical and safety risk factors (laboratory-based) Development of clinical-screening methods and tools Exposure-assessment research: Chemical hazards Physical hazards Biologic hazards Ergonomic hazards Safety (traumatic injury) hazards Safer-design and safety-systems research Intervention research: Control technologies Engineering controls and alternatives Administrative controls Personal protective equipment Work organization Community participation Policy (such as alternative approaches to targeting inspections) Design for safety Emergency preparedness and disaster response Diffusion and dissemination research: Training effectiveness Information-dissemination effectiveness Diffusion of technology Health-services and other research: Access to occupational health care Infrastructureâdelivery of occupational-health services, including international health and safety Socioeconomic consequences of work-related injuries and illnesses Worker compensation Technology-transfer and other transfer activities: Information dissemination Training programs Technical assistance
164 T r a u m a t ic I n j u r y R e s e a rc h at N I OSH â¢ Community-based participatory research on differences between re- cently arrived immigrants and U.S.-born workers regarding percep- tions of acceptable health and safety risks so that programs can be targeted to meet the workforce training needs of immigrant workers Transfer activities should be reviewed to determine whether the NIOSH pro- gram appropriately targets its outputs in a manner that will have the greatest impact. Ideally, information dissemination should be proactive, and strategic dis- semination should be informed by research on the diffusion of new technologies, processes, and practices. Highly relevant information and technology transfer should include plans for appropriate transfer to all appropriate worker populations, including those considered vulnerable. Training should be incorporated into the strategic goals of all research fields where appropriate. The EC should review project-level research and transfer activities (including surveillance activities) that have been completed, are in progress, or planned by the program under review. The program under review should provide a list of activities and specify whether they are intramural or extramural. For each extra- mural project, the key organizations and principal investigatorsâ names should be requested, as should whether the project was in response to a request for proposal or a request for application. For each intramural project, the EC should ask NIOSH to provide a list of key collaborators (from another government agency, academe, industry, or unions). The EC should evaluate each of the research activities outlined in Table 3 if it forms an important element of the program research. In the case of a sector-based research program (for example, mining or construction) in which health-effects research is not being reviewed, the EC should determine what research inputs influence the programâs strategic goals and objective, and then assess the value of the inputs. Questions to Guide the Evaluation Committee in Assessing Research Activities 1. What are the major subprograms or groupings of activities within the program? 2. Are activities consistent with program goals and objectives? 3. Are research activities relevant to the major challenges of the research program? a. Do they address the most serious outcomes? b. Do they address the most common outcomes? c. Do they address the needs of both sexes, vulnerable working popu- lations, and small businesses?
A p p e n d i x A 165 4. Are research activities appropriately responsive to the input of stakeholders? 5. To what extent are partners involved in the research activities? 6. Are partners involved early in the research process so that they could participate in determining research objectives and research design? 7. Were original resource allocations appropriate for the research activi- ties, and do they remain appropriate? 8. To what extent does peer reviews (internal, external, and midcourse) affect the activities? 9. Is there adequate monitoring of quality-assurance procedures to Âensure credible research data, analyses, and conclusions? Questions to Guide the Evaluation Committee in Assessing Transfer Activities 1. Is there a coherent planned program of transfer activities? 2. Are the programâs information dissemination, training, education, technical assistance, or publications successful in reaching the work- place or relevant stakeholders in other settings? How widespread is the response? 3. To what degree have stakeholders responded to NIOSH information and training products? 4. Is there evidence that the formats for information products were s Â elected in response to stakeholder preferences? 5. To what extent do program personnel rely on assessment of stake- holder needs and reactions to prototype information and training projects (formative evaluation techniques)? 6. To what extent does the program build research and education capacity internally and among stakeholders? Assessment For this part of the assessment, the EC will provide a qualitative assessment that discusses relevance. This assessment should include consideration of the external factors identified in Section III.A. that constrain choices of research projects and the relevance and effectiveness of transfer activities. The EC should consider the appropriateness of resource allocations. A highly relevant program would address high-priority needs, produce high-quality results, be appropriately collaborative, be of value to stakeholders, and be substantially engaged in transfer activities. A program may be less relevant to the extent that those key elements are not up to the mark or are missing. The discussion should cover those aspects in sufficient detail
166 T r a u m a t ic I n j u r y R e s e a rc h at N I OSH to arrive at a qualitative assessment of the activities. Assessment of the transfer a Â ctivities must include considerations of program planning, coherence, and impact. The EC might also consider the incorporation of international research results into NIOSH knowledge-transfer activities for industry sectors in the United States. III.B.6. Review of Outputs (Figure 2, Box D) An output is a direct product of a NIOSH research program. Outputs may be designed for researchers, practitioners, intermediaries, and end-users, such as consumers. Outputs can be in the form of publications in peer-reviewed journals, recommendations, reports, Web-site content, workshops and presentations, data- bases, educational materials, scales and methods, new technologies, patents, techni- cal assistance, and so on. Outputs of the research programâs extramurally funded activities should also be considered. Table 4 lists examples of major outputs to be considered by the EC. The NIOSH research program should make every effort to include all pertinent data of the types listed in the table. Outputs may be tailored to the intended audience to communicate information most effectively and increase the likelihood of comprehension, knowledge, attitude formation, and behavioral intent. The extent of use of formative evaluation data (data gathered before communication for the purpose of improving the likelihood of the intended effects) and the extent of intended user feedback in the design of the output can be considered indicators of appropriate quality assessment. Some activities such as collaborations can also legitimately be conceptualized as outputs, because the collaboration itself is a result of NIOSH efforts. Cooperation, coordination, more intensive collaboration, and eventual formal partnering can be considered important outputs leading to desirable intermediate outcomes. Tech- nology and knowledge transfer is greatly facilitated through such relationships. The extent of collaboration with other organizations in the determination of research agendas, the conduct of research, the dissemination of research results, and interor- ganization involvement in the production of outputs can all be measures of output quality and quantity. The EC may consider coauthorship while trying to determine the importance of NIOSH research to the broader research community. The NIOSH program should provide information on all relevant outputs of the program under review produced during the chosen period. Questions to Guide the Evaluation Committee 1. What are the major outputs of the research program? 2. Are output levels consistent with resources allocated (were resources allocated and used efficiently to produce outputs)?
A p p e n d i x A 167 TABLE 4â Examples of Research-Program Outputs to Be Considered Peer-reviewed publications by NIOSH staff: Number of original research articles by NIOSH staff Number of review articles by NIOSH staff (including best-practices articles) Complete citation for each publication Complete copies of the âtop fiveâ articles Collaboration with other public- or private-sector researchers Publications in the field of interest with other support by investigators also funded by NIOSH (for example, ergonomic studies with other support by an investigator funded by NIOSH to do ergonomics work, in which case NIOSH should get some credit for seeding interest or drawing people into the field) Peer-reviewed publications by external researchers funded by NIOSH: Number of NIOSH-funded original research articles by external researchers Number of NIOSH-funded review articles by external researchers (including best-practices articles) Complete citation for each written report Complete copies of the âtop fiveâ articles Collaboration with other government or academic researchers NIOSH reports in the research program: Number of written reports Complete citation for each written report Complete copies of the âtop fiveâ reports Sponsored conferences and workshops: Number of sponsored conferences Number of sponsored workshops Description of conferences and workshops (title, date, sponsors, target audience, number of participants, and resulting products) NIOSHâs assessment of value or impact Databases: Number of major databases created by NIOSH staff Number of major databases created by external researchers funded by NIOSH grants Description of databases: Title, objective (in one to four sentences), and start and stop dates Partial vs. complete sponsorship (if partial, who were cosponsors?) Study or surveillance-system design, study population, and sample size Primary âproductsâ of the database (such as number of peer-reviewed articles and reports) Complete copies of the âtop twoâ publications or findings, to date, from each database continued
168 T r a u m a t ic I n j u r y R e s e a rc h at N I OSH TABLE 4â Continued Recommendations: Number of major recommendations Description of recommendations: Complete citation (article, report, or conference where recommendation was made) Summary in one to four sentences Percentage of target audience that has adopted recommendation 1, 5, and 10 years later Up to three examples of implementation in the field Identification of âtop fiveâ recommendations to date Tools, methods, and technologies (TMT): Number of major TMT (includes training and education materials) Descriptions of TMT Title and objective of TMT (in one to four sentences) Complete citation (if applicable) Percentage of target audience that has used TMT 1, 5, and 10 years later Up to three examples of implementation in the field Identification of âtop fiveâ TMT to date Patents: Total number of patents For each: Title and objective (in one to four sentences) Complete citation Percentage of target audience that has used product 1, 5, and 10 years later Up to three examples of implementation in the field Identification of âtop fiveâ patents to date Miscellaneous: Any other important program outputs 3. Does the research program produce outputs that address high-priority areas? 4. To what extent does the program generate important new knowledge or technology? 5. Are there widely cited peer-reviewed publications considered to report âbreakthroughâ results? 6. What, if any, internal or external capacity-building outputs are documented? 7. Are outputs relevant to both sexes, vulnerable populations, and do they address health disparities? 8. Are outputs relevant to health and safety problems of small businesses?
A p p e n d i x A 169 9. Are products user-friendly with respect to readability, simplicity, and design? 10. To what extent does the program help to build the internal or extra- mural institutional knowledge base? 11. Does the research produce effective cross-agency, cross-institute, or internal-external collaborations? 12. To what extent does the program build research and education capacity (internal or external)? Assessment The EC should provide a qualitative assessment discussing relevance and utility. The outputs of a highly ranked program will address needs in high-priority areas, contain new knowledge or technology that is effectively communicated, contribute to capacity-building inside and outside NIOSH, and be relevant to the pertinent populations. The discussion should cover those aspects in sufficient detail to sup- port the qualitative assessment of the outputs. III.B.7. Review of Intermediate Outcomes (Figure 2, Box E) Intermediate outcomes are important indicators of stakeholder response to NIOSH outputs. They reflect the impact of program activities and may lead to the desired end outcome of improved workplace safety and health. Intermediate out- comes include the production by those outside of NIOSH of guidelines or regula- tions based wholly or partly on NIOSH research (products adopted as national or international public policy or as policy or guidelines by private organizations or industry); contributions to training and education programs sponsored by other organizations; use of publications or other materials by workers, industry, and occupational safety and health professionals in the field; and citations of NIOSH research by industrial and academic scientists. Intermediate outcomes allow inference that a programâs outputs are associated with observed changes in the workplace. Thus, an intermediate outcome reflects an assessment of worth by NIOSH stakeholders (such as managers in indus- trial firms) about NIOSH research or its products (for example, NIOSH training workshops). Intermediate outcomes that are difficult to monitor but may be valid indicators of relevance or utility include self-report measures by users of NIOSH outputs. Such indicators include the extent to which key intermediaries find value in NIOSH products or databases for the repackaging of health and safety informa- tion, the Âextent to which NIOSH recommendations are in place and attended to in
170 T r a u m a t ic I n j u r y R e s e a rc h at N I OSH w Â orkplaces, and employee or employer knowledge of and adherence to NIOSH- recommended practices. Questions to Guide the Evaluation Committee 1. Do program outputs result in or contribute to stakeholder training or education activities used in the workplace or in school or apprentice programs? If so, how? 2. Do program activities and outputs result in regulations, public policy, or voluntary standards or guidelines that are transferred to or created by the workplace? 3. Has the program resulted in changes in employer or worker practices associated with the reduction of risk (for example, in the adoption of new feasible control or personal protective technologies or administra- tive control concepts)? 4. Does the program contribute to changes in health-care practices to improve recognition and management of occupational health conditions? 5. Does the program result in research partnerships with stakeholders that lead to changes in the workplace? 6. To what extent do the programâs stakeholders find value in NIOSH products (as shown by document requests, Web-site hits, conference attendance, and so on)? 7. Does the program or a subprogram provide unique staff or laboratory capability that is a necessary national resource? If so, is it adequate, or does it need to be enhanced or reduced? 8. Has the program resulted in interventions that protect both sexes, vulnerable workers, or address the needs of small businesses? 9. To what extent did the program contribute to increased capacity at worksites to identify or respond to safety and health threats? Assessment Only a qualitative assessment of product development, usefulness, and impact is required at this point in the EC report. Some thought should be given to the relative value of intermediate outcomes, and the FC recommends applying the well- accepted hierarchy-of-controls model. The discussion could include comments on how widely products have been used or programs implemented. The qualitative discussion should be specific as to the various products developed by the program and the extent of their use by specific entities (industry, labor, government, and
A p p e n d i x A 171 so on) for specific purposes. Whether the products have resulted in changes in the workplace or in the reduction of risk should be discussed. The recognition accorded to the program or the facilities by its peers (such as recognition as a âcenter of excellenceâ by national and international communities) should be considered in the assessment. To be highly ranked, a program should have high performance in most of the relevant questions in this section. An aspect of the evaluation can be whether the same changes in stakeholder activities and behaviors would probably have occurred without NIOSH efforts. III.B.8. Review of End Outcomes (Figure 2, Box F) It is necessary for the EC to assess, to the greatest extent possible, NIOSHâs contribution to end outcomesâimprovements in workplace health and safety (im- pact). For purposes of this evaluation, end outcomes are health-related changes that are a result of program activities, including decreases in injuries, illnesses, deaths and exposures or risk. Data on reductions in work-related injuries, illnesses, and hazardous exposures will be available for some programs, and in some cases they will be quantifiable. It is possible, however, to evaluate the impact of a NIOSH research program using either intermediate outcomes or end outcomes. If there is no direct evidence of improvements in health and safety, intermediate outcomes may be used as proxies for end outcomes in assessing impact as long as the EC qualifies its findings. The EC will describe the realized or potential benefits of the NIOSH program. Examples of realized intermediate outcomes are new regulations and widely accepted guidelines, work practices, and procedures, all of which may contribute measurably to enhancing health and safety in the workplace. The FC recognizes that assessing the causal relationship between NIOSH re- search and specific occupational health and safety outcomes is a major challenge because NIOSH does not have direct responsibility or authority for implement- ing its research findings in the workplace. Furthermore, the benefits of NIOSH research program outputs can be realized, potential, or limited to the knowledge gained. Studies that conclude with negative results may nevertheless have incor- porated excellent science and contribute to the knowledge base. The generation of important knowledge is a recognized form of outcome in the absence of measur- able impacts. The impact of an outcome depends on the existence of a âreceptorâ for research results, such as a regulatory agency, a professional organization, an employer, and an employee organization. The EC should consider questions related to the various stages that lead to outputs, such as these:
172 T r a u m a t ic I n j u r y R e s e a rc h at N I OSH 1. Did NIOSH research identify a gap in protection or a means of reduc- ing risk? 2. Did NIOSH convey that information to potential users in a usable form? 3. Were NIOSH research results (for example, recommendations, tech- nologies) applied? 4. Did the applied results lead to desired outcomes? Quantitative data are preferable to qualitative, but qualitative analysis may be necessary. Sources of quantitative data include the following: â¢ Bureau of Labor Statistics (BLS) data on fatal occupational injuries (the Census of Fatal Occupational Injuries) and nonfatal occupational injuries and illnesses (the annual Survey of Occupational Injury and Illnesses) â¢ NIOSH intramural surveillance systems, such as the National Elec- tronic Injury Surveillance System, the coal-worker x-ray surveillance program, and agricultural-worker surveys conducted by NIOSH in collaboration with USDA â¢ State-based surveillance systems, such as the NIOSH-funded ABLES, and the SENSOR programs (for asthma, pesticides, silicosis, noise- induced hearing loss, dermatitis, and burns) â¢ Selected state worker-compensation programs â¢ Exposure data collected in the OSHA Integrated Management Infor- mation System The FC is unaware of mechanisms for surveillance of many occupationally related chronic illnesses, such as cancers that arise from long exposure to chemicals and other stressors. The incidence and prevalence of many such outcomes are best evaluated by investigator-initiated research. Research that leads to new, effective surveillance concepts or programs warrants special recognition. The EC should recognize the strengths and weaknesses of outcome data sources. Quantitative accident, injury, illness, and employment data and databases are sub- ject to error and bias and should be used by the EC only for drawing inferences after critical evaluation and examination of available corroborating data. For example, it is widely recognized that occupational illnesses are poorly documented in the BLS Survey of Occupational Injuries and Illnesses, which captures only incident cases among active workers. It is difficult for health practitioners to diagnose work-relat- edness of most illnesses that may not be exclusively related to work; furthermore, few practitioners are adequately trained to make such an assessment. Many of those
A p p e n d i x A 173 illnesses have long latencies and do not appear until years after people have left the employment in question. Surveillance programs may systematically undercount some categories of workers, such as contingent workers. In addition to measures of illness and injury, measures of exposure to chemical and physical agents and to safety and ergonomic hazards can be useful. Exposure or probability of exposure can serve as an appropriate proxy for disease or injury when a well-described occupational exposure-health association exists. In such instances, a decrease in exposure can be accepted as evidence that the end outcome of reduced illness or injury is being achieved. That is necessary particularly when the latent period between exposure and disease outcome, as in the case of asbestos exposure and lung cancer, makes effective evaluation of the relevant end outcome infeasible. As an example of how an exposure level can serve as a proxy, reduction in the number of sites that exceed an OSHA PEL or an American Conference of Gov- ernmental Industrial Hygienists threshold limit value is a quantitative measure of improvement of occupational health awareness and reduction of risk. In addition to exposure level, the number of people exposed and the distribution of exposure levels are important. Those data are available from multiple databases and studies of exposure. Apart from air monitoring, such measures of exposure as biohazard controls, reduction in requirements for use of personal protective equipment, and reduction in ergonomic risks are important. Challenges posed by inadequate or inaccurate measurement systems should not drive programs out of difficult fields of study, and the EC will need to be aware of such a possibility. In particular, contingent and informal working arrangements that place workers at greatest risk are also those on which surveillance information is almost totally lacking, so novel methods for measuring impact may be required. The commitment of industry, labor, and government to health and safety are critical external factors. Several measures of that commitment can be useful for the EC: monetary commitments, attitude, staffing, and surveys of relative importance. To the extent that resources allocated to safety and health are limiting factors, the EC should explicitly assess NIOSH performance in the context of constraints. Questions to Guide the Evaluation Committee 1. What are the amounts and qualities of relevant end-outcomes data (such as injuries, illness, exposure, and productivity affected by health)? 2. What are the temporal trends in those data? 3. Is there objective evidence of improvement in occupational safety or health?
174 T r a u m a t ic I n j u r y R e s e a rc h at N I OSH 4. To what degree is the NIOSH program or subprogram responsible for improvement in occupational safety or health? 5. If there is no time trend in the data, how do findings compare with data from other comparable U.S. groups or the corresponding populations in other countries? 6. What is the evidence that external factors have affected outcomes or outcome measures? 7. Has the program been responsible for outcomes outside the United States that have not been described in another category? Assessment The EC should provide a qualitative assessment of the program and subpro- gram impact, discussing the evidence of reductions in injuries and illnesses or their appropriate proxies. III.B.9. Review of Potential Outcomes There may be health and safety impacts not yet appreciated and other benefi- cial social, economic, and environmental outcomes as a result of NIOSH activities. NIOSH study results may be influential outside the United States, and there may be evidence of implementation of NIOSH recommendations and training programs abroad. Questions to Guide the Evaluation Committee 1. Is the program likely to produce a favorable change that has not yet occurred or not been appreciated? 2. Has the program been responsible for social, economic, security, or environmental outcomes? 3. Has the programâs work had an impact on occupational health and safety in other countries? Assessment The EC may discuss other outcomes, including beneficial changes that have not yet occurred; social, economic, security, or environmental outcomes; and the impact that NIOSH has had on international occupational safety and health.
A p p e n d i x A 175 III.B.10. Summary Evaluation Ratings and Rationale The EC should use its expert judgment to rate the relevance and impact of the overall research program by first summarizing its assessments of the major subprograms and then appropriately weighting the subprograms to determine the overall program ratings. Table 5 provides some background context to aid the EC in reaching overall ratings for relevance and impact. The EC could consider the items in Table 5 for each subprogram then for the overall program and assess the relevance of the re- search subprograms and program by reviewing earlier responses to the questions in Sections III.B.2. through III.B.5. (reviews of program challenges, strategic goals and objectives, inputs, and activities). Items 1-4 in Table 5 are pertinent to assess- ing relevance. To assess overall impact, the EC first needs to consider the available evidence of changes in work-related risks and adverse effects and external factors related to the changes. The EC should review the responses to the questions in Sections III.B.6. through III.B.8. (reviews of outputs, intermediate outcomes, and end outcomes) and systematically assess the impact of the research program and its subprograms. Items 5-7 in Table 5 will be helpful. The EC should evaluate separately the impact of the research and the impact of transfer activities. Transfer activities occur in two contexts: NIOSH efforts to translate intellectual products into practice and stakeholder efforts to integrate NIOSH results into the workplace. High impact assessments require the ECâs judgment that the research program has contributed to outcomes; for example, outcomes have occurred earlier than they would have or are better than they would have been in the absence of the research program, TABLE 5â Background Context for Program Relevance and Impact Assess the following for each subprogram: 1. Relevance of current and recently completed research and transfer activities to objective improvements in workplace safety and health. 2. Contributions of NIOSH research and transfer activities to changes in work-related practices and reduction in workplace exposures, illnesses, or injuries. 3. Contributions of NIOSH research and transfer activities to improvements in work-related practices. 4. Contributions of NIOSH research to productivity, security, or environmental quality (beneficial side effects). 5. Evidence of reduction of risk in the workplace (intermediate outcome). 6. Evidence of reduction in workplace exposure, illness, or injuries (end outcome). 7. Evidence of external factors that prevented translation of NIOSH research results into intermediate or end outcomes.
176 T r a u m a t ic I n j u r y R e s e a rc h at N I OSH or outcomes would have occurred were it not for external factors beyond NIOSHâs control or ability to plan around. The EC must assign one overall integer score for the relevance of the research program to the improvement of occupational safety and health and one overall integer score for the impact of the program on the improvement of occupational safety and health. The EC will use its expert judgment, summary assessment of research-Âprogram elements, and any appropriate information to arrive at those two scores. In light of substantial differences among the types of research programs that will be reviewed and the challenge to arrive at a summative evaluation of both relevance and impact, the FC chose not to construct an algorithm to produce the two final ratings. Relevance and impact scores will be based on five-point categorical scales estabÂlished by the FC (see Boxes 2 and 3) in which 1 is the lowest and 5 the highest rating. The FC has made an effort to establish mutually exclusive rating categories in the scales. When the basis of a rating fits more than one category, the highest applicable score should be assigned. It is up to the EC to determine how individual subprograms should influence final scores. Single integer values should be assigned. Final program ratings will consist of integer scores for relevance and impact and prose justification of the scores. Box 2 includes the criteria for scoring the overall relevance of the NIOSH research program. As discussed in previous sections, numerous factors can be con- sidered in assessing relevance. The scoring criteria focus on two: the EC assessment BOX 2 Scoring Criteria for Relevance 5 = Research is in high-priority subject areas and NIOSH is significantly engaged in appro- priate transfer activities for completed research projects/reported research results. 4 = Research is in priority subject areas and NIOSH is engaged in appropriate transfer activities for completed research projects/reported research results. 3 = Research is in high priority or priority subject areas, but NIOSH is not engaged in aÂppropriate transfer activities; or research focuses on lesser priorities but NIOSH is engaged in appropriate transfer activities. 2 = Research program is focused on lesser priorities and NIOSH is not engaged in or plan- ning some appropriate transfer activities. 1 = Research program is not focused on priorities and NIOSH is not engaged in transfer activities.
A p p e n d i x A 177 BOX 3 Scoring Criteria for Impact 5 = Research program has made major contribution(s) to worker health and safety on the basis of end outcomes or well-accepted intermediate outcomes. 4 = Research program has made some contributions to end outcomes or well-accepted intermediate outcomes. 3 = Research program activities are ongoing and outputs are produced that are likely to result in improvements in worker health and safety (with explanation of why not rated higher). Well accepted outcomes have not been recorded. 2 = Research program activities are ongoing and outputs are produced that may result in new knowledge or technology, but only limited application is expected. Well accepted outcomes have not been recorded. 1 = Research activities and outputs do not result in or are NOT likely to have any application. of whether the program appropriately sets priorities among research needs and the EC assessment of how engaged the program is in appropriate transfer activities. Table 6 provides some guidance regarding how the EC may weight research priori- ties and transfer levels when determining relevance scores. The EC will consider both completed research and research that is in progress and related to likely future improvements in its assessment of relevance. The EC should keep in mind how well the program has considered the frequency and severity of the problems being addressed; whether appropriate attention has been directed to both sexes, vulnerable populations, or hard-to-reach workplaces; and whether the differ- ent needs of large and small businesses have been accounted for. It is up to the EC to determine how to consider external factors in assigning program scores. Box 3 includes the criteria established for the rating of impact. In general, the EC will consider completed research outputs during the assessment of impact. In assigning a score for impact, it is important to recognize that a âmajor contribu- tionâ (required for a score of 5) does not imply that the NIOSH program was solely responsible for observed improvements in worker health and safety. Many factors may be required to effect improvements. The EC could say that NIOSH made âmajor contributionsâ if the improvements would not have occurred when they did without NIOSH efforts.
178 T r a u m a t ic I n j u r y R e s e a rc h at N I OSH TABLE 6â Guidance for Weighting Research Priority and Engagement in Appropriate Transfer Activities in the Application of Relevance Score Assessment of Research Priority Engagement in Applicable Transfer Activities Applicable Score High priority Significantly engaged 5 High priority Engaged 4 High priority Not engaged 3 Priority Significantly engaged 4 Priority Engaged 4 Priority Not engaged 3 Lesser priority Significantly engaged 3 Lesser priority Engaged 3 Lesser priority Not engaged 2 Not focused on priorities Significantly engaged 2 Not focused on priorities Engaged 2 Not focused on priorities Not engaged 1 The FC has some concern that the imposed scoring criteria for impact might be considered a promotion of the conventional occupational-health research para- digm that focuses on health-effects and technology research without much empha- sis on the socioeconomic, policy, surveillance, and diffusion research (as opposed to diffusion activities) needed to effect change. The EC should remember that not all intermediate outcomes occur in the workplace. Important outcomes that NIOSH can effect also occur much farther out on the causal chain. NIOSH, for example, has an important role to play in generating knowledge that may contribute to changing norms in the insurance industry, in health-care practice, in public-health practice, and in the community at large. The EC may find that some of those issues need to be addressed and considered as external factors that facilitate or limit application of more traditional research findings. Given the rapidly changing nature of work and the workforce and some of the intractable problems in manufacturing, min- ing, and some other fields, the EC is encouraged to think beyond the traditional paradigm. III.C. Assessment of NIOSH Process for Targeting Priority Research Needs and Committee Assessment of Emerging Issues The second charge to the EC is the assessment of the research programâs effec- tiveness in targeting new research and identifying emerging issues in occupational safety and health most relevant to future improvements in workplace protection. The EC is also asked to provide a qualitative narrative assessment of the programâs
A p p e n d i x A 179 efforts and to make suggestions about emerging issues that the program should be prepared to address. Among the most challenging aspects of research in illness and injury prevention are the identification of new or emerging needs or trends and the formulation of a research response that appropriately uses scarce resources in anticipation of them. The EC should review the procedures that NIOSH and the research program have in place to identify needed research relevant to the NIOSH mission and should review the success that NIOSH has had in identifying and addressing research related to emerging issues. It should examine leading indicators from appropriate federal agencies, such as EPA, the Department of Labor, the National Institute of Standards and Technology, NIH, DOD, and the Department of Commerce. Those indicators should track new technologies, new products, new processes, and disease or injury trends. One source of information deserving particular attention is NIOSH HHE r Â eports. The HHE program offers a potential mechanism for identifying emerging research needs that could be incorporated as input into each of the programs evalu- ated. The EC should determine whether the program under review appropriately considers pertinent HHE investigation findings. Additional emerging issues may be revealed through consideration of NIOSH and the NIOSH-funded FACE reports, the AOEC reports, the U.S. Chemical Safety Board investigations, and SENSOR and other state-based surveillance programs. Appropriate federal advisory commit- tees and other stakeholder groups should also be consulted to provide qualitative information. The EC should systematically assess how the research program and its subÂprograms target new research by evaluating each subprogram for the items listed in Table 7. The EC will have to determine how best to weight subprogram contributions in the programâs targeting of new research. TABLE 7â Targeting of New Research and Identification of Emerging Issues Assess the following for each subprogram: 1. Past and present effectiveness in targeting most relevant research needs. 2. Effectiveness in targeting research in fields most relevant to future improvements in occupational safety and health. 3. Contribution of NIOSH research to enhancement of capacity in government or other research institutions.
180 T r a u m a t ic I n j u r y R e s e a rc h at N I OSH Questions to Guide the Evaluation Committee 1. What information does NIOSH review to identify emerging research needs? a. What is the process for review? b. How often does the process take place? c. How are NIOSH staff scientists and NIOSH leadership engaged? d. What is the process for moving from ideas to formal planning and resource allocation? 2. How are stakeholders involved? a. What advisory or stakeholder groups are asked to identify emerg- ing research targets? b. How often are such groups consulted, and how are suggestions followed up? 3. What new research targets have been identified for future development in the program under evaluation? a. How were they identified? b. Were lessons that could help to identify other emerging issues learned? c. Does the EC agree with the issues identified and selected as im- portant and with the NIOSH response, or were important issues overlooked? d. Is there evidence of unwise expenditure of resources on unimport- ant issues? The EC members should use their expert judgment both to evaluate the emerging research targets identified by NIOSH and to provide recommendations to NIOSH regarding additional research that NIOSH has not yet identified. Recom- mendations should include a brief statement of their rationale. IV. EVALUATION COMMITTEE REPORT TEMPLATE Consistency and comparability among EC report formats is desirable, but the FC recognizes that each NIOSH research program is different and that each EC is independent. The outline provided in Box 4 flows from the FCâs review of NIOSHâs generalized logic model (Figure 1), the evaluation flowchart (Figure 2), and the assessment model described earlier in this document. The EC should feel free to use or adapt this outline as necessary when organizing its final report. The FC encourages each EC to look at prior EC reports for organizational ideas.
A p p e n d i x A 181 BOX 4 Suggested Outline for Evaluation Committee Reports I. Introduction This section should be a brief descriptive summary of the history of the program (and subprograms) being evaluated with respect to pre-NORA, NORA 1, and current and future plans of the research program presented by NIOSH. It should present the context for the research on safety and health; goals, objectives, and resources; groupings of subprograms; and any other important pertinent information. (A list of the NIOSH materials reviewed should be provided in Appendix C.) II. Evaluation of Programs and Subprograms (Charge 1) A. Evaluation summary (should include a brief summary of the evaluation with respect to impact and relevance, scores for impact and relevance, and summary statements). B. Strategic goals and objectives: should describe assessment of the program and subÂ programs for relevance. C. Review of inputs: should describe adequacy of inputs to achieve goals. D. Review of activities: should describe assessment of the relevance of the activities. E. Review of research-program outputs: should describe assessment of relevance and potential usefulness of the research program. F. Review of intermediate outcomes and causal impact: should describe assessment of the intermediate outcomes and the attribution to NIOSH; should include the likely impacts and recent outcomes in the assessment. G. Review of end outcomes: should describe the end outcomes related to health and safety and provides an assessment of the type and degree of attribution to NIOSH. H. Review of other outcomes: should discuss health and safety impacts that have not yet occurred; beneficial social, economic, and environmental outcomes; and international dimensions and outcomes. I. Summary of ratings and rationale. III. NIOSH Targeting of New Research and Identification of Emerging Issues (Charge 2) The EC should assess the progress that the NIOSH program has made in targeting new research in occupational safety and health. The EC should assess whether the NIOSH pro- gram has identified important emerging issues that appear especially important in terms of relevance to the mission of NIOSH. The EC should respond to NIOSHâs perspective and add its own recommendations. IV. Recommendations for Program Improvement On the basis of the review and evaluation of the program, the EC may provide recommen- dations for improving the relevance of the NIOSH research program to health and safety conditions in the workplace and the impact of the research program on health and safety in the workplace. Appendix A â Framework Document Appendix B â Methods and Information-Gathering Appendix C â ist of NIOSH and Related Materials Collected in the Process of the L Evaluation