1
Introduction

Preventing tractor rollovers on farmers, protecting construction workers from falls, improving the health of miners in dusty environments, reducing back injuries in nursing aides, developing substitute materials to eliminate hazardous chemical exposures, and designing work conditions to reduce fatigue and stress are among the issues critical to improving worker safety and health. In 2007, 5,657 fatal work injuries occurred in the United States, along with an estimated 4 million nonfatal occupational injuries and illnesses among workers in private industry (BLS, 2008, 2009). In addition to the human suffering involved, these statistics are associated with high economic costs. One estimate puts the costs of occupational injury and illnesses for all industries for 2005 at more than $160 billion (Leigh, 2008). The continued attention to further improve occupational health and safety through research is not only fully warranted, but such research requires critical evaluation of its relevance and impact. The core mission of the National Institute for Occupational Safety and Health (NIOSH) is to conduct research to improve and protect the health and safety of workers.

In September 2004, NIOSH contracted with The National Academies to conduct a series of evaluations of individual NIOSH research programs.1 This set of independent evaluations focused on the relevance and impact of each of eight NIOSH programs on reducing work-related injuries, illnesses, and hazardous exposures.

1

A program is defined as a set of inputs and activities directed toward one or more common goals, typically under the direction of a manager or management team.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 11
1 Introduction P reventing tractor rollovers on farmers, protecting construction workers from falls, improving the health of miners in dusty environments, reducing back injuries in nursing aides, developing substitute materials to eliminate hazard- ous chemical exposures, and designing work conditions to reduce fatigue and stress are among the issues critical to improving worker safety and health. In 2007, 5,657 fatal work injuries occurred in the United States, along with an estimated 4 million nonfatal occupational injuries and illnesses among workers in private industry (BLS, 2008, 2009). In addition to the human suffering involved, these statistics are associated with high economic costs. One estimate puts the costs of occupational injury and illnesses for all industries for 2005 at more than $160 billion (Leigh, 2008). The continued attention to further improve occupational health and safety through research is not only fully warranted, but such research requires critical evaluation of its relevance and impact. The core mission of the National Institute for Occupational Safety and Health (NIOSH) is to conduct research to improve and protect the health and safety of workers. In September 2004, NIOSH contracted with The National Academies to conduct a series of evaluations of individual NIOSH research programs.1 This set of inde- pendent evaluations focused on the relevance and impact of each of eight NIOSH programs on reducing work-related injuries, illnesses, and hazardous exposures. 1A program is defined as a set ofinputs and activities directed toward one or more common goals, typically under the direction of a manager or management team. 

OCR for page 11
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s From the outset of the evaluations, NIOSH leadership established the primary goal as program improvement, but the context for the evaluations also included the PART (Program Assessment Rating Tool) federal agency evaluation process. The first step in this multiphase effort was to develop an evaluation framework that could be applied across the set of program evaluations to enhance cross-study consistency. An Institute of Medicine (IOM)/National Research Council (NRC) committee (the framework committee) was appointed to develop the evaluation framework. The resulting evaluation framework was then used by eight separately appointed ad hoc committees (evaluation committees) to assess NIOSH programs in hearing loss; mining; agriculture, forestry, and fishing; respiratory diseases; personal protective technology; traumatic injury; construction; and health hazard evaluation. Each evaluation committee produced an individual report (IOM and NRC, 2006, 2008, 2009; NRC and IOM, 2007, 2008a,b, 2009a,b). This report provides the evaluation framework developed, implemented, and refined over the course of four years and eight evaluations. The framework uses a standard tool in program management and evaluation—the logic model—and provides details on the types of information that are needed and questions to be considered in each phase of the evaluation. This report has two goals: (1) to sum- marize the evaluation process and lessons learned in the development and use of the framework and (2) to provide recommendations for future evaluation efforts. The evaluation framework may prove applicable in evaluating other federal agency research programs. SCOPE OF THE TASK The framework and evaluation committees followed the same basic statement of task (Box 1-1). Although the statement of task was modified to clarify specific issues or to accommodate programs that were not specifically research programs,2 the basic objectives for the program evaluations remained the same: • An assessment of the relevance and impact of the NIOSH program’s contribution to reducing work-related hazardous exposures, illnesses, and injuries based on integer scales of 1 to 5, with text to support the rating; • Assessment of the program’s effectiveness in targeting new research areas and identification of emerging issues that the program should be prepared to address; and • Recommendations for program improvement. 2 The charges to the committees to evaluate the NIOSH Health Hazard Evaluation and Personal Protective Technology programs were each slightly modified to accommodate the unique standards- setting and investigative aspects of these programs.

OCR for page 11
intRoduction  BOX 1-1 Review of NIOSH Research Programs Statement of Task   In  response  to  a  request  from  the  National  Institute  for  Occupational  Safety  and  Health (NIOSH), the Institute of Medicine and the Division of Earth and Life Studies of the  National Academies are conducting a series of evaluations of NIOSH research programs.  Each  evaluation  is  being  conducted  by  an  ad  hoc  committee,  using  a  methodology  and  framework developed by the Committee to Review NIOSH Research Programs (framework  committee).    Each evaluation committee will review the program’s impact, relevance, and future di- rections. The evaluation committee will evaluate not only what the NIOSH research program  is producing, but will also determine whether it is appropriate to credit NIOSH research with  changes  in  workplace  practices,  hazardous  exposures,  and/or  occupational  illnesses  and  injuries, or whether the changes are the result of other factors unrelated to NIOSH.    The program reviews should focus on evaluating the program’s impact and relevance  to health and safety issues in the workplace and make recommendations for improvement.  In conducting the review, the evaluation committee will address the following elements:   1.    ssessment of the program’s contribution through occupational safety and health  A research  to  reductions  in  workplace  hazardous  exposures,  illnesses,  or  injuries  through: a.    n assessment of the relevance of the program’s activities to the improvement  A of occupational safety and health; and b.    n  evaluation  of  the  impact  that  the  program’s  research  has  had  in  reducing  A work-related hazardous exposures, illnesses, and injuries.        he evaluation committee will rate the performance of the program for its relevance  T and impact using an integer score of 1 to 5. Impact may be assessed directly (e.g.,  reductions in illnesses or injuries) or, as necessary, using intermediate outcomes  to estimate impact. Qualitative narrative evaluations will be included to explain the  numerical ratings.    2.    ssessment  of  the  program’s  effectiveness  in  targeting  new  research  areas  and  A identifying emerging issues in occupational safety and health most relevant to future  improvements  in  workplace  protection.  The  committee  will  provide  a  qualitative  narrative  assessment  of  the  program’s  efforts  and  suggestions  about  emerging  issues that the program should be prepared to address.    In 2008, NIOSH requested that the Committee on the Review of NIOSH Research Pro- grams prepare a brief report to document the lessons learned in this evaluation effort and set  forth the process that it recommends be used for future evaluations of occupational health  and safety research programs. The report will build on the latest version of the framework  used  to  guide  the  work  of  the  eight  evaluation  committees  completed  to  date,  and  may  revise the framework as necessary based on lessons learned about the evaluation process  (continued)

OCR for page 11
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s BOX 1-1 Continued during these reviews. The committee will draw on the evaluation literature to provide ad- ditional context for the report. Recommendations for future program evaluations or more  generalizable guiding principles for these types of efforts may be provided. Input for this  report will involve discussions with NIOSH staff and with chairs and committee members  of the evaluation committees. OVERVIEW OF NIOSH Created in 1970 by the Occupational and Safety Health Act, NIOSH is charged with the responsibility to “conduct . . . research, experiments, and demonstrations relating to occupational safety and health” and to develop “innovative methods, techniques, and approaches for dealing with occupational safety and health prob- lems” (Public Law 91-596). The focus of NIOSH’s responsibilities is occupational health and safety research, along with professional education and training. NIOSH is also involved in surveillance and in providing advice to the Secretary of Labor regarding standards needed to protect workers as well as other efforts that support research and the transfer of research into the workplace. NIOSH does not have the authority to establish or enforce regulations on workplace safety and health.3 Regulatory and enforcement authority at the federal level rests largely with the Occupational Safety and Health Administration and the Mine Safety and Health Administration, both in the Department of Labor. NIOSH was established as an agency within the Department of Health and Human Services (HHS). Originally the director of the institute reported to the Secretary of HHS. In the 1980s, NIOSH was reorganized as one of seven compo- nents of the Centers for Disease Control and Prevention. NIOSH offices are located across the country: Washington, DC; Pittsburgh, PA; Spokane, WA; Cincinnati, OH; Atlanta, GA; and Morgantown, WV. Because NIOSH deals with issues spe- cific to particular work sectors (e.g., mining, construction, agriculture) as well as crosscutting issues that span multiple sectors (e.g., hearing loss, personal protective technologies, respiratory diseases), many programs now use a matrix-management approach. (For discussion of the pros and cons of matrix management, see IOM and NRC, 2006.) 3 NIOSH has responsibility for certifying most types of occupational respirators and for updating the federal respirator certification and testing regulations (42 C.F.R. Part 84).

OCR for page 11
intRoduction  The NIOSH mission is “to provide national and world leadership to prevent work-related illness, injury, disability, and death by gathering information, con- ducting scientific research, and translating the knowledge gained into products and services.” To fulfill its mission, NIOSH has established three strategic goals (NIOSH, 2009a): • Goal 1: Conduct research to reduce work-related illnesses and injuries. C Track work-related hazards, exposures, illnesses, and injuries for prevention. o Generate new knowledge through intramural and extramural research programs. o Develop innovative solutions for difficult-to-solve problems in high- risk industrial sectors. • Goal 2: Promote safe and healthy workplaces through interventions, rec- ommendations, and capacity building. o Enhance the relevance and utility of recommendations and guidance. o Transfer research findings, technologies, and information into practice. o Build capacity to address traditional and emerging hazards. • Goal 3: Enhance global workplace safety and health through international collaborations. o Take a leadership role in developing a global network of occupational health centers. o Investigate alternative approaches to workplace illness and injury re- duction and provide technical assistance to implement solutions. o Build global professional capacity to address workplace hazards through training, information sharing, and research experience. In 1994, NIOSH embarked on a national partnership effort to identify research priorities to guide occupational health and safety research for the next decade (NIOSH, 2009b). Participants included external stakeholders 4 from many areas, including universities, large and small businesses, professional societies, government agencies, and worker organizations. The result of these efforts was the National Occupational Research Agenda (NORA). Through the NORA effort, 21 priority research areas were identified and research agendas developed. NORA was intended not only for use by NIOSH, but also for the entire occupational health and safety community. In the second decade of NORA, NIOSH continues to work with its partners to update the research agenda using an approach based 4 Stakeholders are defined as the broad group of individuals or organizations with an interest in the mission of the program. Partners are the subset of stakeholders that contribute directly to program activities.

OCR for page 11
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s on industry sectors, with an increased emphasis on moving research results into practice in the workplace. The current NIOSH portfolio is organized into 8 sector programs and 24 crosscutting programs (Box 1-2). DEVELOPING AND IMPLEMENTING THE EVALUATION FRAMEWORK As noted above, before embarking on a series of program evaluations, the Na- tional Academies formed a committee to develop an evaluation framework and to continue over the course of the evaluations to monitor the use of the framework and adapt it as needed to make the evaluations more effective. Initial Framework Document The framework committee appointed by the National Academies consisted of 14 members with expertise in occupational medicine and health, industrial health and safety, industrial hygiene, epidemiology, civil and mining engineering, sociol- ogy, program evaluation, communication, toxicology, international occupational health, and industry and workforce issues. During the initial development of the evaluation criteria and process, the framework committee drew from the program evaluation literature and discussed evaluation strategies and precedents with NIOSH leadership and staff, industry leaders, RAND Corporation staff,5 occupational safety organizations, labor repre- sentatives, and staff from other federal agencies (see Appendix A). The framework committee decided to use the logic model as the basis for the evaluation framework. The logic model organizes the program and its efforts into inputs (e.g., budget, staffing, facilities), activities (e.g., research studies, surveillance, exposure measurement), outputs (e.g., reports, conferences, training, patents), and outcomes (e.g., collaborations, policy changes, reductions in injuries and hazard- ous exposures) (see Box 1-3 for definitions and additional examples). As will be further discussed in Chapter 2, logic models are widely used in program evaluation and planning. To provide guidance on evaluating and scoring the relevance and impact of each NIOSH program, the framework committee developed criteria and specific questions to be used in the assessment of each component of the logic model. Assessment of strategic goals and objectives, inputs, activities, and outputs largely 5 NIOSH contracted with RAND Corporation for the development of the logic models spe - cific to each program and assistance on identifying and assembling the materials needed for the evaluations.

OCR for page 11
intRoduction  BOX 1-2 NIOSH Program Portfolio   The National Institute for Occupational Safety and Health (NIOSH) Program Portfolio  has been organized into 8 National Occupational Research Agenda Sector Programs that  represent industrial sectors, and 24 cross-sector programs. Sector Programs: •  Agriculture, Forestry, and Fishing  •  Construction  •  Health Care and Social Assistance  •  Manufacturing  •  Mining; Oil and Gas Extraction Subsector  •  Services  •  Transportation, Warehousing, and Utilities  •  Wholesale and Retail Trade  Cross-Sector Programs: •  Authoritative Recommendations  •  Cancer, Reproductive and Cardiovascular Diseases  •  Communications and Information Dissemination  •  Economics  •  Emergency Preparedness and Response  •  Engineering Controls  •  Exposure Assessment  •  Global Collaborations  •  Health Hazard Evaluation •  Hearing Loss Prevention  •  Immune and Dermal Diseases  •  Musculoskeletal Disorders  •  Nanotechnology  •  Occupational Health Disparities  •  Personal Protective Technology  •  Prevention Through Design  •  Radiation Dose Reconstruction  •  Respiratory Diseases  •  Small Business Assistance and Outreach  •  Surveillance  •  Training Grants  •  Traumatic Injury  •  Work Organization and Stress-Related Disorders •  Worklife Initiative  SOURCE: NIOSH (2009c).

OCR for page 11
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s BOX 1-3 Logic Model Terms and Examples Planning Inputs:  Stakeholder  input,  surveillance  and  intervention  data,  and  risk  assess- ments (e.g., input from Federal Advisory Committee Act panels or the National Occupational  Research  Agenda  research  partners,  intramural  surveillance  information,  Health  Hazard  Evaluations [HHEs]). Production Inputs: Intramural and extramural funding, staffing, management structure, and  physical facilities.  Activities: Efforts and work of the program, staff, grantees, and contractors (e.g., surveil- lance, health effects research, intervention research, health services research, information  dissemination, training, technical assistance).  Outputs: Direct products of NIOSH programs that are logically related to the achievement  of desirable and intended outcomes (e.g., publications in peer-reviewed journals, recom- mendations, reports, website content, workshops and presentations, databases, educational  materials, new technologies, patents, technical assistance).  Intermediate Outcomes: Actions by stakeholders in response to NIOSH products or efforts  (e.g., policy changes; production of standards or regulations; adoption of NIOSH-developed  technologies; use of publications, technologies, methods, or recommendations by workers,  industry, and occupational safety and health professionals in the field; citations of NIOSH  research by industry and academic scientists).  End Outcomes: Improvements in safety and health in the workplace—specifically, reductions  in work-related injuries, illnesses, or deaths or reductions in hazardous exposures in the  workplace—that can be attributed to NIOSH efforts.  External Factors: Actions or forces beyond NIOSH’s control (e.g., by industry, labor, regula- tors,  and  other  entities)  with  important  bearing on  moving  research results into practice  in the workplace. SOURCE: Adapted from IOM and NRC (2006).  defined the relevance of the program; the committee examined the adequacy of the inputs and the scope and targeting of the activities and outputs in achieving the program’s goals. Assessment of the intermediate and end outcomes largely defined the program’s impact. Overlap necessarily occurred between the assessment of relevance and impact, particularly in the assessment of information transfer.

OCR for page 11
intRoduction  The framework committee met three times to develop the first version of the evaluation framework, which was released in December 2005. This version of the framework was then used by the evaluation committees examining the first four NIOSH programs to be considered: Hearing Loss Prevention; Mining; Agriculture, Forestry, and Fishing; and Respiratory Diseases. Issues in Developing and Implementing the Evaluation Framework The variations in the types of NIOSH programs undergoing evaluation raised several issues during the framework committee’s work and in the course of the initial set of program evaluations. These issues are further discussed in Chapter 4 and reflect the discussions among NIOSH staff and committee members at the November 2008 workshop held after the evaluations were completed. Primary issues included the following: • Differences in program mission, including whether a program is dedi- cated to multiple health and safety issues within a single industry sector (e.g., mining) or focuses on a set of health outcomes (e.g., hearing loss) in whichever industrial sector they occur (a cross-sector program); • Differences in program structure or management. Programs may use varying management styles and structures, with some programs using a matrix-management approach that coordinates across NIOSH divisions and facilities and others that are located within a single NIOSH division using a more traditional management structure; • Differences in program size; • Differences in program and subprogram maturity. An older program or subprogram may have better developed strategic goals and measurable outcomes; • Overlap among programs; • The weighting of different activities within and across study areas of a single program; • The influence of external factors on the implementation, translation, and impact of NIOSH’s efforts; • The definition and difference between intermediate and end outcomes; • Limitations in the availability of quantitative information (e.g., surveil- lance data), particularly related to priority setting and documentation of intermediate and end outcomes; and • The quantitative rating scales. The framework committee was concerned about developing scoring criteria that would maximize consistency across

OCR for page 11
0 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s the reports. Many evaluation committee members were initially concerned that the required task of specifying scores for relevance and for impact on 5-point integer scales was too quantitative for a process that, by nature, was largely qualitative. An additional concern discussed by the framework committee and the evalu- ation committees was the challenge of evaluating and scoring a federal program within the broader context of agency evaluations being conducted by the Office of Management and Budget using the Program Assessment Rating Tool (PART). Concerns about how the evaluations would be used and their potential impact on the programs’ budgets were voiced. NIOSH senior management staff acknowledged the PART evaluation requirements, but also emphasized that NIOSH’s reason for conducting thorough and independent external evaluations was focused on quality improvement. The evaluation and framework committees kept both goals in mind and focused on conducting a fair and balanced evaluation. Revisions to the Evaluation Framework To monitor and refine the evaluation framework over the course of the evalu- ations, the framework committee continued to meet once or twice annually from 2006 through 2009 (see Appendix A). In 2006 and 2007, the framework committee discussed the first set of ongoing and recently completed evaluations with evalua- tion committee chairs and members, NIOSH staff, National Academies’ staff, and stakeholder groups. They explored the applicability of the framework document, the usefulness of the initial sets of materials provided by NIOSH (the program evi- dence packages), and issues associated with the quantitative ratings and the ratings criteria. Issues associated with the evaluation of program quality in the evaluation framework were also discussed. In response, the framework committee released a revised framework in August 2007 to improve clarity, accommodate the revised statement of task as negotiated with NIOSH, and make the document more usable to the evaluation committees. Changes included reorganization of the document to more closely follow the re- vised statement of task and to improve readability. Slight modifications were made to the criteria for scoring impact and relevance that made the wording more precise. Additions were made in the form of guidance to the evaluation committees regard- ing scoring a program for relevance based on the committee’s observed levels of “research priority” and “engagement in appropriate transfer activities.” The four evaluation committees examining the NIOSH Personal Protective Tech- nology, Traumatic Injury, Construction, and Health Hazard Evaluation programs used the August 2007 version of the framework.

OCR for page 11
intRoduction  As described below, the framework committee continued to receive input from the individual evaluation committees and followed their progress. The framework committee met in 2008 and 2009 with the goals of (1) convening a workshop on lessons learned in the evaluation process, and (2) compiling this report, which includes a further revised framework (version 2009) and recommendations for future evaluations. Evaluation Committees As of the publication date of this report, the evaluation of eight NIOSH pro- grams has been completed (IOM and NRC, 2006, 2008, 2009; NRC and IOM, 2007, 2008a,b, 2009a,b). Table 1-1 lists the programs evaluated, and summarizes the activities of the evaluation committees, the version of the framework document applied, and the dates of report release. Framework committee members served as evaluation committee members, liaisons, and report reviewers. Each evaluation committee met three to four times. In addition to meetings, some committees conducted site visits of research program facilities. The NIOSH program provided each committee with an evidence package prior to its first meeting. Additional information was acquired through site visits; input from external stakeholders through open sessions at committee meetings, online questionnaires, and writ- ten responses; and oral and written responses to questions posed to the NIOSH programs. Throughout the evaluation process, evaluation committee members provided information to the framework committee on the successes and limitations of the framework in conducting their specific program evaluation. Process for Developing This Report Subsequent to the eight evaluations, the framework committee reconvened for two meetings to discuss the lessons learned during the process and to revise the evaluation framework accordingly. This report is the result of those discussions and revisions. The report draws on the experience of those who participated in develop- ing the framework and those who used the framework to conduct an evaluation. Additionally, the framework committee held a workshop on November 24, 2008, that focused on lessons learned in the evaluation efforts (see Appendix A). NIOSH staff members, evaluation committee members and chairs, framework committee members, and IOM and NRC staff members participated in the workshop and shared their perspectives on lessons learned and their ideas for future evaluations. Committee chairs and NIOSH staff members had an additional opportunity for input after the workshop.

OCR for page 11
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s TABLE 1-1 Evaluation Committee Timelines Evaluation Framework Meeting Datesa Program Name Used Report Release Hearing Loss 2005 January 5–6, 2006 August 2006 February 23–24, 2006 March 30–31, 2006 Mining Safety 2005 January 12–13, 2006 May 2007 and Health February 21–22, 2006 May 9–10, 2006 Agriculture, 2005 January 18–20, 2007 December 2007 Forestry, and March 28–29, 2007 Fishing May 30–31, 2007 Respiratory 2005 October 26–27, 2006 March 2008 Diseases December 5–6, 2006 March 22–23, 2007 2007b Personal September 27–28, 2007 June 2008 Protective December 17–18, 2007 Technology March 6–7, 2008 (PPT) Traumatic 2007 March 29–30, 2007 August 2008 Injury May 31–June 1, 2007 September 6–7, 2007 Construction 2007 July 17–18, 2007 October 2008 September 25–27, 2007 December 10–12, 2007 2007b Health Hazard October 18–19, 2007 October 2008 Evaluation December 10–11, 2007 (HHE) January 15–16, 2008 February 21–22, 2008 aThe list of meeting dates does not include site visits or committee conference calls. bThe PPT and HHE studies used slightly adapted statements of task because of the certification or investigative work of these programs. Therefore the committees had to slightly modify the application of the evaluation framework.

OCR for page 11
intRoduction  NIOSH Follow-Up on the Evaluation Reports During the time that the framework committee examined the evaluation pro- cess, NIOSH staff members have been responding to each of the evaluation reports. NIOSH program staff members were tasked with developing an action plan that addresses the evaluation report findings and recommendations. The action plans are presented to NIOSH senior management and then to the NIOSH Board of Scientific Counselors (BSC). The NIOSH BSC has committed to be actively in- volved in reviewing the action plans and to write a short assessment of each action plan. Additionally, the BSC is examining the recommendations from all eight of the reports to identify crosscutting issues (e.g., need for improved surveillance). Addressing the issues and strategies identified through this process might provide the greatest efficiency and effectiveness in continuing to improve the relevance and impact of the NIOSH programs. OVERVIEW OF THIS REPORT Chapter 2 of this report sets the National Academies’ approach to evaluating the NIOSH programs in the larger context of program evaluation. Chapter 3 pro- vides the revised evaluation framework. Because the framework has been integrated into this report (as opposed to previous versions that were stand-alone documents and appendixes to the evaluation reports), a number of the changes involved plac- ing introductory material in Chapter 1. Other changes were more substantive and were the result of the framework committee’s careful considerations of the lessons learned, which are detailed in Chapter 4. The concluding chapter provides the committee’s recommendations for NIOSH to consider in planning future evalua- tions. The committee believes the evaluation framework is sufficiently robust and the lessons learned are sufficiently generalizable to offer insights to other federal agencies as they consider program evaluation. REFERENCES BLS (Bureau of Labor Statistics). 2008. Workplace illnesses and injuries in 00. http://www.bls.gov/ iif/oshwc/osh/os/osnr0030.pdf (accessed April 10, 2009). BLS. 2009. Fatal occupational injuries by industry and event or exposure. http://www.bls.gov/iif/oshwc/ cfoi/cftb0223.pdf (accessed April 10, 2009). IOM and NRC (Institute of Medicine and National Research Council). 2006. Hearing loss research at NIOSH. Committee to Review the NIOSH Hearing Loss Research Program. Rpt. No. 1, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press.

OCR for page 11
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s IOM and NRC. 2008. The personal protective technology program at NIOSH. Committee to Review the NIOSH Personal Protective Technology Program. Rpt. No. 5, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. IOM and NRC. 2009. Traumatic injury research at NIOSH. Committee to Review the NIOSH Trau- matic Injury Research Program. Rpt. No. 6, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. Leigh, P. S. 2008. Cost of occupational injury and illness combining all industries. Presentation at the November 2008 Seminar for Western Center for Agriculture Health and Safety. http://agcenter. uc davis.edu/seminar/flyer/2009/Leigh_Nov3_2008.ppt#256 (accessed March 26, 2009). NIOSH (National Institute for Occupational Safety and Health). 2009a. NIOSH strategic plan outline, 00–00. http://www.cdc.gov/niosh /docs/strategic (accessed March 19, 2009). NIOSH. 2009b. NORA. http://www.cdc.gov/niosh/nora/ (accessed March 17, 2009). NIOSH. 2009c. NIOSH program portfolio. http://www.cdc.gov/niosh/programs/ (accessed March 17, 2009). NRC and IOM. 2007. Mining safety and health research at NIOSH. Committee to Review the NIOSH Mining Safety and Health Research Program. Rpt. No. 2, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Acad- emies Press. NRC and IOM. 2008a. Agriculture, forestry, and fishing research at NIOSH. Committee to Review the NIOSH Agriculture, Forestry, and Fishing Research Program. Rpt. No. 3, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. NRC and IOM. 2008b. Respiratory diseases research at NIOSH. Committee to Review the NIOSH Re- spiratory Diseases Research Program. Rpt. No. 4, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. NRC and IOM. 2009a. The health hazard evaluation program at NIOSH. Committee to Review the NIOSH Health Hazard Evaluation Program. Rpt. No. 7, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Acad- emies Press. NRC and IOM. 2009b. Construction research at NIOSH. Committee to Review the NIOSH Construc- tion Research Program. Rpt. No. 8, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press.