|
||||||||||||||||
|
||||||||||||||||
The following HTML text is provided to enhance online readability. Many aspects of typography translate only awkwardly to HTML. Please use the page image as the authoritative form to ensure accuracy. 1
|
||||||||||||||||
 |
 |
Page 11 |
 |
 |
 |
|
| ||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 11
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
1
Introduction
Preventing tractor rollovers on farmers, protecting construction workers from falls, improving the health of miners in dusty environments, reducing back injuries in nursing aides, developing substitute materials to eliminate hazardous chemical exposures, and designing work conditions to reduce fatigue and stress are among the issues critical to improving worker safety and health. In 2007, 5,657 fatal work injuries occurred in the United States, along with an estimated 4 million nonfatal occupational injuries and illnesses among workers in private industry (BLS, 2008, 2009). In addition to the human suffering involved, these statistics are associated with high economic costs. One estimate puts the costs of occupational injury and illnesses for all industries for 2005 at more than $160 billion (Leigh, 2008). The continued attention to further improve occupational health and safety through research is not only fully warranted, but such research requires critical evaluation of its relevance and impact. The core mission of the National Institute for Occupational Safety and Health (NIOSH) is to conduct research to improve and protect the health and safety of workers.
In September 2004, NIOSH contracted with The National Academies to conduct a series of evaluations of individual NIOSH research programs.1 This set of independent evaluations focused on the relevance and impact of each of eight NIOSH programs on reducing work-related injuries, illnesses, and hazardous exposures.
1
A program is defined as a set of inputs and activities directed toward one or more common goals, typically under the direction of a manager or management team.
OCR for page 12
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
From the outset of the evaluations, NIOSH leadership established the primary goal as program improvement, but the context for the evaluations also included the PART (Program Assessment Rating Tool) federal agency evaluation process.
The first step in this multiphase effort was to develop an evaluation framework that could be applied across the set of program evaluations to enhance cross-study consistency. An Institute of Medicine (IOM)/National Research Council (NRC) committee (the framework committee) was appointed to develop the evaluation framework. The resulting evaluation framework was then used by eight separately appointed ad hoc committees (evaluation committees) to assess NIOSH programs in hearing loss; mining; agriculture, forestry, and fishing; respiratory diseases; personal protective technology; traumatic injury; construction; and health hazard evaluation. Each evaluation committee produced an individual report (IOM and NRC, 2006, 2008, 2009; NRC and IOM, 2007, 2008a,b, 2009a,b).
This report provides the evaluation framework developed, implemented, and refined over the course of four years and eight evaluations. The framework uses a standard tool in program management and evaluation—the logic model—and provides details on the types of information that are needed and questions to be considered in each phase of the evaluation. This report has two goals: (1) to summarize the evaluation process and lessons learned in the development and use of the framework and (2) to provide recommendations for future evaluation efforts. The evaluation framework may prove applicable in evaluating other federal agency research programs.
SCOPE OF THE TASK
The framework and evaluation committees followed the same basic statement of task (Box 1-1). Although the statement of task was modified to clarify specific issues or to accommodate programs that were not specifically research programs,2 the basic objectives for the program evaluations remained the same:
An assessment of the relevance and impact of the NIOSH program’s contribution to reducing work-related hazardous exposures, illnesses, and injuries based on integer scales of 1 to 5, with text to support the rating;
Assessment of the program’s effectiveness in targeting new research areas and identification of emerging issues that the program should be prepared to address; and
Recommendations for program improvement.
2
The charges to the committees to evaluate the NIOSH Health Hazard Evaluation and Personal Protective Technology programs were each slightly modified to accommodate the unique standards-setting and investigative aspects of these programs.
OCR for page 13
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
BOX 1-1
Review of NIOSH Research Programs Statement of Task
In response to a request from the National Institute for Occupational Safety and Health (NIOSH), the Institute of Medicine and the Division of Earth and Life Studies of the National Academies are conducting a series of evaluations of NIOSH research programs. Each evaluation is being conducted by an ad hoc committee, using a methodology and framework developed by the Committee to Review NIOSH Research Programs (framework committee).
Each evaluation committee will review the program’s impact, relevance, and future directions. The evaluation committee will evaluate not only what the NIOSH research program is producing, but will also determine whether it is appropriate to credit NIOSH research with changes in workplace practices, hazardous exposures, and/or occupational illnesses and injuries, or whether the changes are the result of other factors unrelated to NIOSH.
The program reviews should focus on evaluating the program’s impact and relevance to health and safety issues in the workplace and make recommendations for improvement. In conducting the review, the evaluation committee will address the following elements:
Assessment of the program’s contribution through occupational safety and health research to reductions in workplace hazardous exposures, illnesses, or injuries through:
An assessment of the relevance of the program’s activities to the improvement of occupational safety and health; and
An evaluation of the impact that the program’s research has had in reducing work-related hazardous exposures, illnesses, and injuries.
The evaluation committee will rate the performance of the program for its relevance and impact using an integer score of 1 to 5. Impact may be assessed directly (e.g., reductions in illnesses or injuries) or, as necessary, using intermediate outcomes to estimate impact. Qualitative narrative evaluations will be included to explain the numerical ratings.
Assessment of the program’s effectiveness in targeting new research areas and identifying emerging issues in occupational safety and health most relevant to future improvements in workplace protection. The committee will provide a qualitative narrative assessment of the program’s efforts and suggestions about emerging issues that the program should be prepared to address.
In 2008, NIOSH requested that the Committee on the Review of NIOSH Research Programs prepare a brief report to document the lessons learned in this evaluation effort and set forth the process that it recommends be used for future evaluations of occupational health and safety research programs. The report will build on the latest version of the framework used to guide the work of the eight evaluation committees completed to date, and may revise the framework as necessary based on lessons learned about the evaluation process
OCR for page 14
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
during these reviews. The committee will draw on the evaluation literature to provide additional context for the report. Recommendations for future program evaluations or more generalizable guiding principles for these types of efforts may be provided. Input for this report will involve discussions with NIOSH staff and with chairs and committee members of the evaluation committees.
OVERVIEW OF NIOSH
Created in 1970 by the Occupational and Safety Health Act, NIOSH is charged with the responsibility to “conduct … research, experiments, and demonstrations relating to occupational safety and health” and to develop “innovative methods, techniques, and approaches for dealing with occupational safety and health problems” (Public Law 91-596). The focus of NIOSH’s responsibilities is occupational health and safety research, along with professional education and training. NIOSH is also involved in surveillance and in providing advice to the Secretary of Labor regarding standards needed to protect workers as well as other efforts that support research and the transfer of research into the workplace. NIOSH does not have the authority to establish or enforce regulations on workplace safety and health.3 Regulatory and enforcement authority at the federal level rests largely with the Occupational Safety and Health Administration and the Mine Safety and Health Administration, both in the Department of Labor.
NIOSH was established as an agency within the Department of Health and Human Services (HHS). Originally the director of the institute reported to the Secretary of HHS. In the 1980s, NIOSH was reorganized as one of seven components of the Centers for Disease Control and Prevention. NIOSH offices are located across the country: Washington, DC; Pittsburgh, PA; Spokane, WA; Cincinnati, OH; Atlanta, GA; and Morgantown, WV. Because NIOSH deals with issues specific to particular work sectors (e.g., mining, construction, agriculture) as well as crosscutting issues that span multiple sectors (e.g., hearing loss, personal protective technologies, respiratory diseases), many programs now use a matrix-management approach. (For discussion of the pros and cons of matrix management, see IOM and NRC, 2006.)
3
NIOSH has responsibility for certifying most types of occupational respirators and for updating the federal respirator certification and testing regulations (42 C.F.R. Part 84).
OCR for page 15
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
The NIOSH mission is “to provide national and world leadership to prevent work-related illness, injury, disability, and death by gathering information, conducting scientific research, and translating the knowledge gained into products and services.” To fulfill its mission, NIOSH has established three strategic goals (NIOSH, 2009a):
Goal 1: Conduct research to reduce work-related illnesses and injuries.
Track work-related hazards, exposures, illnesses, and injuries for prevention.
Generate new knowledge through intramural and extramural research programs.
Develop innovative solutions for difficult-to-solve problems in high-risk industrial sectors.
Goal 2: Promote safe and healthy workplaces through interventions, recommendations, and capacity building.
Enhance the relevance and utility of recommendations and guidance.
Transfer research findings, technologies, and information into practice.
Build capacity to address traditional and emerging hazards.
Goal 3: Enhance global workplace safety and health through international collaborations.
Take a leadership role in developing a global network of occupational health centers.
Investigate alternative approaches to workplace illness and injury reduction and provide technical assistance to implement solutions.
Build global professional capacity to address workplace hazards through training, information sharing, and research experience.
In 1994, NIOSH embarked on a national partnership effort to identify research priorities to guide occupational health and safety research for the next decade (NIOSH, 2009b). Participants included external stakeholders4 from many areas, including universities, large and small businesses, professional societies, government agencies, and worker organizations. The result of these efforts was the National Occupational Research Agenda (NORA). Through the NORA effort, 21 priority research areas were identified and research agendas developed. NORA was intended not only for use by NIOSH, but also for the entire occupational health and safety community. In the second decade of NORA, NIOSH continues to work with its partners to update the research agenda using an approach based
4
Stakeholders are defined as the broad group of individuals or organizations with an interest in the mission of the program. Partners are the subset of stakeholders that contribute directly to program activities.
OCR for page 16
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
on industry sectors, with an increased emphasis on moving research results into practice in the workplace. The current NIOSH portfolio is organized into 8 sector programs and 24 crosscutting programs (Box 1-2).
DEVELOPING AND IMPLEMENTING THE EVALUATION FRAMEWORK
As noted above, before embarking on a series of program evaluations, the National Academies formed a committee to develop an evaluation framework and to continue over the course of the evaluations to monitor the use of the framework and adapt it as needed to make the evaluations more effective.
Initial Framework Document
The framework committee appointed by the National Academies consisted of 14 members with expertise in occupational medicine and health, industrial health and safety, industrial hygiene, epidemiology, civil and mining engineering, sociology, program evaluation, communication, toxicology, international occupational health, and industry and workforce issues.
During the initial development of the evaluation criteria and process, the framework committee drew from the program evaluation literature and discussed evaluation strategies and precedents with NIOSH leadership and staff, industry leaders, RAND Corporation staff,5 occupational safety organizations, labor representatives, and staff from other federal agencies (see Appendix A).
The framework committee decided to use the logic model as the basis for the evaluation framework. The logic model organizes the program and its efforts into inputs (e.g., budget, staffing, facilities), activities (e.g., research studies, surveillance, exposure measurement), outputs (e.g., reports, conferences, training, patents), and outcomes (e.g., collaborations, policy changes, reductions in injuries and hazardous exposures) (see Box 1-3 for definitions and additional examples). As will be further discussed in Chapter 2, logic models are widely used in program evaluation and planning.
To provide guidance on evaluating and scoring the relevance and impact of each NIOSH program, the framework committee developed criteria and specific questions to be used in the assessment of each component of the logic model. Assessment of strategic goals and objectives, inputs, activities, and outputs largely
5
NIOSH contracted with RAND Corporation for the development of the logic models specific to each program and assistance on identifying and assembling the materials needed for the evaluations.
OCR for page 17
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
BOX 1-2
NIOSH Program Portfolio
The National Institute for Occupational Safety and Health (NIOSH) Program Portfolio has been organized into 8 National Occupational Research Agenda Sector Programs that represent industrial sectors, and 24 cross-sector programs.
Sector Programs:
Agriculture, Forestry, and Fishing
Construction
Health Care and Social Assistance
Manufacturing
Mining; Oil and Gas Extraction Subsector
Services
Transportation, Warehousing, and Utilities
Wholesale and Retail Trade
Cross-Sector Programs:
Authoritative Recommendations
Cancer, Reproductive and Cardiovascular Diseases
Communications and Information Dissemination
Economics
Emergency Preparedness and Response
Engineering Controls
Exposure Assessment
Global Collaborations
Health Hazard Evaluation
Hearing Loss Prevention
Immune and Dermal Diseases
Musculoskeletal Disorders
Nanotechnology
Occupational Health Disparities
Personal Protective Technology
Prevention Through Design
Radiation Dose Reconstruction
Respiratory Diseases
Small Business Assistance and Outreach
Surveillance
Training Grants
Traumatic Injury
Work Organization and Stress-Related Disorders
Worklife Initiative
SOURCE: NIOSH (2009c).
OCR for page 18
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
BOX 1-3
Logic Model Terms and Examples
Planning Inputs: Stakeholder input, surveillance and intervention data, and risk assessments (e.g., input from Federal Advisory Committee Act panels or the National Occupational Research Agenda research partners, intramural surveillance information, Health Hazard Evaluations [HHEs]).
Production Inputs: Intramural and extramural funding, staffing, management structure, and physical facilities.
Activities: Efforts and work of the program, staff, grantees, and contractors (e.g., surveillance, health effects research, intervention research, health services research, information dissemination, training, technical assistance).
Outputs: Direct products of NIOSH programs that are logically related to the achievement of desirable and intended outcomes (e.g., publications in peer-reviewed journals, recommendations, reports, website content, workshops and presentations, databases, educational materials, new technologies, patents, technical assistance).
Intermediate Outcomes: Actions by stakeholders in response to NIOSH products or efforts (e.g., policy changes; production of standards or regulations; adoption of NIOSH-developed technologies; use of publications, technologies, methods, or recommendations by workers, industry, and occupational safety and health professionals in the field; citations of NIOSH research by industry and academic scientists).
End Outcomes: Improvements in safety and health in the workplace—specifically, reductions in work-related injuries, illnesses, or deaths or reductions in hazardous exposures in the workplace—that can be attributed to NIOSH efforts.
External Factors: Actions or forces beyond NIOSH’s control (e.g., by industry, labor, regulators, and other entities) with important bearing on moving research results into practice in the workplace.
SOURCE: Adapted from IOM and NRC (2006).
defined the relevance of the program; the committee examined the adequacy of the inputs and the scope and targeting of the activities and outputs in achieving the program’s goals. Assessment of the intermediate and end outcomes largely defined the program’s impact. Overlap necessarily occurred between the assessment of relevance and impact, particularly in the assessment of information transfer.
OCR for page 19
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
The framework committee met three times to develop the first version of the evaluation framework, which was released in December 2005. This version of the framework was then used by the evaluation committees examining the first four NIOSH programs to be considered: Hearing Loss Prevention; Mining; Agriculture, Forestry, and Fishing; and Respiratory Diseases.
Issues in Developing and Implementing the Evaluation Framework
The variations in the types of NIOSH programs undergoing evaluation raised several issues during the framework committee’s work and in the course of the initial set of program evaluations. These issues are further discussed in Chapter 4 and reflect the discussions among NIOSH staff and committee members at the November 2008 workshop held after the evaluations were completed. Primary issues included the following:
Differences in program mission, including whether a program is dedicated to multiple health and safety issues within a single industry sector (e.g., mining) or focuses on a set of health outcomes (e.g., hearing loss) in whichever industrial sector they occur (a cross-sector program);
Differences in program structure or management. Programs may use varying management styles and structures, with some programs using a matrix-management approach that coordinates across NIOSH divisions and facilities and others that are located within a single NIOSH division using a more traditional management structure;
Differences in program size;
Differences in program and subprogram maturity. An older program or subprogram may have better developed strategic goals and measurable outcomes;
Overlap among programs;
The weighting of different activities within and across study areas of a single program;
The influence of external factors on the implementation, translation, and impact of NIOSH’s efforts;
The definition and difference between intermediate and end outcomes;
Limitations in the availability of quantitative information (e.g., surveillance data), particularly related to priority setting and documentation of intermediate and end outcomes; and
The quantitative rating scales. The framework committee was concerned about developing scoring criteria that would maximize consistency across
OCR for page 20
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
the reports. Many evaluation committee members were initially concerned that the required task of specifying scores for relevance and for impact on 5-point integer scales was too quantitative for a process that, by nature, was largely qualitative.
An additional concern discussed by the framework committee and the evaluation committees was the challenge of evaluating and scoring a federal program within the broader context of agency evaluations being conducted by the Office of Management and Budget using the Program Assessment Rating Tool (PART). Concerns about how the evaluations would be used and their potential impact on the programs’ budgets were voiced. NIOSH senior management staff acknowledged the PART evaluation requirements, but also emphasized that NIOSH’s reason for conducting thorough and independent external evaluations was focused on quality improvement. The evaluation and framework committees kept both goals in mind and focused on conducting a fair and balanced evaluation.
Revisions to the Evaluation Framework
To monitor and refine the evaluation framework over the course of the evaluations, the framework committee continued to meet once or twice annually from 2006 through 2009 (see Appendix A). In 2006 and 2007, the framework committee discussed the first set of ongoing and recently completed evaluations with evaluation committee chairs and members, NIOSH staff, National Academies’ staff, and stakeholder groups. They explored the applicability of the framework document, the usefulness of the initial sets of materials provided by NIOSH (the program evidence packages), and issues associated with the quantitative ratings and the ratings criteria. Issues associated with the evaluation of program quality in the evaluation framework were also discussed.
In response, the framework committee released a revised framework in August 2007 to improve clarity, accommodate the revised statement of task as negotiated with NIOSH, and make the document more usable to the evaluation committees. Changes included reorganization of the document to more closely follow the revised statement of task and to improve readability. Slight modifications were made to the criteria for scoring impact and relevance that made the wording more precise. Additions were made in the form of guidance to the evaluation committees regarding scoring a program for relevance based on the committee’s observed levels of “research priority” and “engagement in appropriate transfer activities.”
The four evaluation committees examining the NIOSH Personal Protective Technology, Traumatic Injury, Construction, and Health Hazard Evaluation programs used the August 2007 version of the framework.
OCR for page 21
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
As described below, the framework committee continued to receive input from the individual evaluation committees and followed their progress. The framework committee met in 2008 and 2009 with the goals of (1) convening a workshop on lessons learned in the evaluation process, and (2) compiling this report, which includes a further revised framework (version 2009) and recommendations for future evaluations.
Evaluation Committees
As of the publication date of this report, the evaluation of eight NIOSH programs has been completed (IOM and NRC, 2006, 2008, 2009; NRC and IOM, 2007, 2008a,b, 2009a,b). Table 1-1 lists the programs evaluated, and summarizes the activities of the evaluation committees, the version of the framework document applied, and the dates of report release. Framework committee members served as evaluation committee members, liaisons, and report reviewers. Each evaluation committee met three to four times. In addition to meetings, some committees conducted site visits of research program facilities. The NIOSH program provided each committee with an evidence package prior to its first meeting. Additional information was acquired through site visits; input from external stakeholders through open sessions at committee meetings, online questionnaires, and written responses; and oral and written responses to questions posed to the NIOSH programs. Throughout the evaluation process, evaluation committee members provided information to the framework committee on the successes and limitations of the framework in conducting their specific program evaluation.
Process for Developing This Report
Subsequent to the eight evaluations, the framework committee reconvened for two meetings to discuss the lessons learned during the process and to revise the evaluation framework accordingly. This report is the result of those discussions and revisions. The report draws on the experience of those who participated in developing the framework and those who used the framework to conduct an evaluation. Additionally, the framework committee held a workshop on November 24, 2008, that focused on lessons learned in the evaluation efforts (see Appendix A). NIOSH staff members, evaluation committee members and chairs, framework committee members, and IOM and NRC staff members participated in the workshop and shared their perspectives on lessons learned and their ideas for future evaluations. Committee chairs and NIOSH staff members had an additional opportunity for input after the workshop.
OCR for page 22
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
TABLE 1-1 Evaluation Committee Timelines
Program Name
Evaluation Framework Used
Meeting Datesa
Report Release
Hearing Loss
2005
January 5–6, 2006
February 23–24, 2006
March 30–31, 2006
August 2006
Mining Safety and Health
2005
January 12–13, 2006
February 21–22, 2006
May 9–10, 2006
May 2007
Agriculture, Forestry, and Fishing
2005
January 18–20, 2007
March 28–29, 2007
May 30–31, 2007
December 2007
Respiratory Diseases
2005
October 26–27, 2006
December 5–6, 2006
March 22–23, 2007
March 2008
Personal Protective Technology (PPT)
2007b
September 27–28, 2007
December 17–18, 2007
March 6–7, 2008
June 2008
Traumatic Injury
2007
March 29–30, 2007
May 31–June 1, 2007
September 6–7, 2007
August 2008
Construction
2007
July 17–18, 2007
September 25–27, 2007
December 10–12, 2007
October 2008
Health Hazard Evaluation (HHE)
2007b
October 18–19, 2007
December 10–11, 2007
January 15–16, 2008
February 21–22, 2008
October 2008
aThe list of meeting dates does not include site visits or committee conference calls.
bThe PPT and HHE studies used slightly adapted statements of task because of the certification or investigative work of these programs. Therefore the committees had to slightly modify the application of the evaluation framework.
OCR for page 23
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
NIOSH Follow-Up on the Evaluation Reports
During the time that the framework committee examined the evaluation process, NIOSH staff members have been responding to each of the evaluation reports. NIOSH program staff members were tasked with developing an action plan that addresses the evaluation report findings and recommendations. The action plans are presented to NIOSH senior management and then to the NIOSH Board of Scientific Counselors (BSC). The NIOSH BSC has committed to be actively involved in reviewing the action plans and to write a short assessment of each action plan. Additionally, the BSC is examining the recommendations from all eight of the reports to identify crosscutting issues (e.g., need for improved surveillance). Addressing the issues and strategies identified through this process might provide the greatest efficiency and effectiveness in continuing to improve the relevance and impact of the NIOSH programs.
OVERVIEW OF THIS REPORT
Chapter 2 of this report sets the National Academies’ approach to evaluating the NIOSH programs in the larger context of program evaluation. Chapter 3 provides the revised evaluation framework. Because the framework has been integrated into this report (as opposed to previous versions that were stand-alone documents and appendixes to the evaluation reports), a number of the changes involved placing introductory material in Chapter 1. Other changes were more substantive and were the result of the framework committee’s careful considerations of the lessons learned, which are detailed in Chapter 4. The concluding chapter provides the committee’s recommendations for NIOSH to consider in planning future evaluations. The committee believes the evaluation framework is sufficiently robust and the lessons learned are sufficiently generalizable to offer insights to other federal agencies as they consider program evaluation.
REFERENCES
BLS (Bureau of Labor Statistics). 2008. Workplace illnesses and injuries in 2007. http://www.bls.gov/iif/oshwc/osh/os/osnr0030.pdf (accessed April 10, 2009).
BLS. 2009. Fatal occupational injuries by industry and event or exposure. http://www.bls.gov/iif/oshwc/cfoi/cftb0223.pdf (accessed April 10, 2009).
IOM and NRC (Institute of Medicine and National Research Council). 2006. Hearing loss research at NIOSH. Committee to Review the NIOSH Hearing Loss Research Program. Rpt. No. 1, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press.
OCR for page 24
Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps
IOM and NRC. 2008. The personal protective technology program at NIOSH. Committee to Review the NIOSH Personal Protective Technology Program. Rpt. No. 5, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press.
IOM and NRC. 2009. Traumatic injury research at NIOSH. Committee to Review the NIOSH Traumatic Injury Research Program. Rpt. No. 6, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press.
Leigh, P. S. 2008. Cost of occupational injury and illness combining all industries. Presentation at the November 2008 Seminar for Western Center for Agriculture Health and Safety. http://agcenter.uc davis.edu/seminar/flyer/2009/Leigh_Nov3_2008.ppt#256 (accessed March 26, 2009).
NIOSH (National Institute for Occupational Safety and Health). 2009a. NIOSH strategic plan outline, 2004–2009. http://www.cdc.gov/niosh /docs/strategic (accessed March 19, 2009).
NIOSH. 2009b. NORA. http://www.cdc.gov/niosh/nora/ (accessed March 17, 2009).
NIOSH. 2009c. NIOSH program portfolio. http://www.cdc.gov/niosh/programs/ (accessed March 17, 2009).
NRC and IOM. 2007. Mining safety and health research at NIOSH. Committee to Review the NIOSH Mining Safety and Health Research Program. Rpt. No. 2, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press.
NRC and IOM. 2008a. Agriculture, forestry, and fishing research at NIOSH. Committee to Review the NIOSH Agriculture, Forestry, and Fishing Research Program. Rpt. No. 3, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press.
NRC and IOM. 2008b. Respiratory diseases research at NIOSH. Committee to Review the NIOSH Respiratory Diseases Research Program. Rpt. No. 4, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press.
NRC and IOM. 2009a. The health hazard evaluation program at NIOSH. Committee to Review the NIOSH Health Hazard Evaluation Program. Rpt. No. 7, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press.
NRC and IOM. 2009b. Construction research at NIOSH. Committee to Review the NIOSH Construction Research Program. Rpt. No. 8, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press.