Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Evaluating Occupational Health and Safety Research Programs FRAMEWORK AND NEXT STEPS Committee for the Review of NIOSH Research Programs
THE NATIONAL ACADEMIES PRESS â 500 Fifth Street, N.W. â Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This study was requested by the National Institute for Occupational Safety and Health of the Centers for Disease Control and Prevention and supported by Award No. 211-2006-19152 T.O. #1 between the National Academy of Sciences and the Centers for Disease Control and Prevention. Any opinions, findings, conclusions, or recommendations expressed in the publication are those of the author(s) and do not necessarily reflect the view of the organizations or agencies that provided support for this project. International Standard Book Number-13: 978-0-309-13795-9 International Standard Book Number-10: 0-309-13795-0 Additional copies of this report are available from the National Academies Press, 500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu. For more information about the Institute of Medicine, visit the IOM home page at: www.iom.edu. Copyright 2009 by the National Academy of Sciences. All rights reserved. Printed in the United States of America Suggested citation: IOM (Institute of Medicine) and National Research Council. 2009. Evaluating occupational health and safety research programs: Framework and next steps. Washington, DC: The National Academies Press.
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academyâs purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council. www.national-academies.org
COMMITTEE FOR THE REVIEW OF NIOSH RESEARCH PROGRAMS* DAVID H. WEGMAN (Chair), University of MassachusettsâLowell WILLIAM B. BUNN III, International Truck and Engine Corporation, Warrenville, IL CARLOS A. CAMARGO, JR., Massachusetts General Hospital, Bostonï¿½ SUSAN E. COZZENS, Georgia Institute of Technology, Atlanta LETITIA K. DAVIS, Massachusetts Department of Public Health, Boston JAMES W. DEARING, Kaiser Permanente, Denver, CO FRED A. METTLER, JR., New Mexico VA Healthcare System, Albuquerque FRANKLIN E. MIRER, Hunter School of Health Sciences, New York JACQUELINE NOWELL, United Food and Commercial Workers International Union, Washington, DC ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ RAJA V. RAMANI, Pennsylvania State UniversityâUniversity Park JORMA RANTANEN, Finnish Institute of Occupational Health, Helsinki, Finland RICHARD L. TUCKER, The University of TexasâAustin JAMES J. ZUICHES, North Carolina State University, Raleigh Study Staff CATHY LIVERMAN, Project Director (since January 2008) EVAN B. DOUPLE, Staff Officer (until January 2008) SAMMANTHA MAGSINO, Program Officer ANDREW M. POPE, Board Director GREGORY H. SYMMES, Deputy Executive Director JUDY ESTEP, Program Associate *Maxine Hayes, University of WashingtonâOlympia, resigned August 2005. Donald Henderson, State University of New York, resigned June 2005. Rosemary K. Solas, University of Illinois-Chicago, resigned January 2009. Joseph. S. Wholey, University of Southern California, resigned March 2006.
Reviewers This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures ap- proved by the National Research Councilâs Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: Alfred Franzblau, University of Michigan School of Public Health Patricia A. Gleason, Safety Equipment Institute Paul D. Gunderson, Dakota Center for Technology-Optimized Agriculture, Lake Region State College Brian M. Kleiner, Virginia Polytechnic Institute and State University Frances P. Lawrenz, University of Minnesota Laura C. Leviton, The Robert Wood Johnson Foundation Julia Melkers, Georgia Institute of Technology Jonathan G. Price, Nevada Bureau of Mines and Geology, University of Nevada Stanley C. Suboleski, Federal Mine Safety and Health Review Commission (retired), Midlothian, Virginia vii
viii Reviewers Although the reviewers listed above have provided many constructive com- ments and suggestions, they were not asked to endorse the conclusions or recom- mendations, nor did they see the final draft of the report before its release. The review of this report was overseen by David G. Hoel, Medical University of South Carolina and John C. Bailar III, The University of Chicago, (emeritus). Appointed by the National Research Council and Institute of Medicine, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.
Preface O wad some power the giftie gie us to see ourselâs as ithers see us.1 âââââââ âRobert Burns I n our modern society, we observe evaluation or assessment efforts in all walks of life. Among the many systematic approaches to evaluation are those seen in outcomes assessment for clinical practice, in standard exams for student per- formance in primary and secondary education, in accreditation reviews for higher education programs and institutions as well as hospitals, and in the Government Performance and Results Act and Performance Assessment Rating Tool approaches to assess effectiveness of the great variety of federal programs. Each of these settings has promoters and detractors of the objectives to be achieved by assessment as well as of the specific assessment mechanisms used for the evaluation. â1RobertBurns, To a Louse. On seeing one on a ladyâs bonnet at church. Original: O wad some power the giftie gie us To see ourselâs as ithers see us! It wad frae monie a blunder free us, Anâ foolish notion: What airs in dress anâ gait wad leaâe us, Anâ evân devotion! Standard English translation: O would some Power the gift to give us To see ourselves as others see us! It would from many a blunder free us, And foolish notion: What airs in dress and gait would leave us, And even devotion! ix
Preface Hardly anyone anticipates the prospect of being evaluated with pleasure. Most, however, agree that the often-burdensome process of preparing for, as well as un- dergoing, an evaluation ends up as instructive and promotes improved individual or program performance. Facing the challenges of any assessment by an external organization entails assembling and organizing large quantities of data and pro- gram details, frequently accompanied by the task of preparing a self-assessment. When the National Institute for Occupational Safety and Health (NIOSH) determined that a formal external program evaluation would contribute impor- tantly to achieving its mission, the Institute took the bold step of asking that the evaluation be not only qualitative, but also quantitative. Furthermore, despite the predominant research mission of the Institute, the leadership determined that each programâs activities and outputs should be assessed on a standard that judged (1) whether they were contributing to new scientific knowledge, and (2) to what extent the program was making a significant and important contribution to improving worker safety and health. The Institute, thus, wished to acknowledge fully the ap- plied nature of its research mission. As the framework committee began this task in 2004, there were many un- knowns about what the challenges would be in providing consistent evaluations across a set of widely diverse NIOSH programs. The evaluation framework devel- oped by the committee has now been used by eight separate evaluation committees and has been found to be a useful and thorough approach for program evaluation. The resulting evaluations have provided a consistent assessment and have identified a number of recommendations to improve the NIOSH programs. In general the separate evaluations found the programs to be productive, to represent substantial quality and applicability, and to be efficient in using very limited resources. When the eight program evaluations were examined together, some common needs emerged that would benefit future evaluations and NIOSH as a whole. Most of the evaluation committees noted the lack of adequate surveillance data on occupational exposures, illnesses, and injuries and made recommendations to bolster surveillance systems. An increased focus on strategic planning was another recurring theme, as was the need to strengthen efforts to move research findings to the worksite. Improvements in integrating extramural and intramural research at NIOSH were also recommended. This report provides an opportunity to step back and look at the broader pic- ture of the evaluation process developed for the NIOSH research programs. The framework committee has heard throughout the process about the strengths and challenges of this endeavor. In November 2008 the framework committee held a workshop in which that committeeâs members, members of the eight evaluation committees, NIOSH staff, and National Academiesâ staff met to discuss the lessons
Preface xi learned. Thus, this report draws on those insights as well as others gained by the framework committee over the course of the past four years. The committee and the individual evaluation committees greatly benefited from the thorough briefings and informative discussions with NIOSH staff mem- bers. On behalf of the committee, I want to especially thank Ray Sinclair and Lewis Wade, who provided excellent guidance from the inception of the project and who held fast to the goals of examining the relevance and impact of NIOSHâs work on the end outcomes of worker safety and health, an evaluation process envisioned by John Howard. This committee and each of the evaluation committees appreciate the dedicated efforts by NIOSH staff in compiling information and responding to numerous inquiries. It is heartening to see that NIOSH staff have found that the evaluations address their needs and are devoting similar efforts to the development of implementation plans in response to the evaluations. Chairing this National Academies committee and having the opportunity to interact with the eight evaluation committees was a privilege and a pleasure. The framework committee members sustained their energy, interest, and dedication to this task over the course of four years. In addition to serving on this committee, many members also served as liaisons or members of the evaluation committees, and I thank them for the time and high level of engagement they gave to this evaluation process. NIOSH has a large taskâconducting research to improve occupational safety and healthâbut limited resources. This evaluation process has shown the great relevance and impact of NIOSHâs work. Our hope is that all of these efforts will contribute to further improvements in the safety and health of workers. David H. Wegman, Chair Committee for the Review of NIOSH Research Programs
Contents ABBREVIATIONS AND ACRONYMS xv SUMMARY 1 1 INTRODUCTION 11 Scope of the Task, 12 Overview of NIOSH, 14 Developing and Implementing the Evaluation Framework, 16 Overview of This Report, 23 References, 23 2 THE PROGRAM EVALUATION CONTEXT 25 Program Evaluation, 25 Logic Models, 27 Role of Stakeholders, 28 Methods of Evaluation, 29 Summary, 33 References, 34 3 EVALUATION FRAMEWORK 37 Overview of the Evaluation Framework, 38 xiii
xiv Contents Evaluation Committees, 42 Steps in the Evaluation Process, 43 References, 72 4 IMPROVING THE EVALUATION PROCESS 75 Evaluation Framework, 76 Composition of the Evaluation Committees, 80 Agency Inputs to the Evaluation, 81 Stakeholder Input, 86 Timelines for Evaluations, 87 Evaluation Committee Reports and Recommendations, 88 Summary, 90 References, 91 5 RECOMMENDATIONS FOR MOVING FORWARD 93 Ongoing Evaluation, 93 Continue to Bolster Research Translation Efforts, 95 Enhance Occupational Health and Safety Surveillance, 97 Integrate Evaluations of Extramural and Intramural Research, 98 On the Horizon, 99 References, 100 APPENDIXES A Meeting Agendas: Open Sessions 103 B Biographical Sketches of Committee Members 109
Abbreviations and Acronyms AFF Agriculture, Forestry, and Fishing AOEC Association of Occupational and Environmental Clinics BLS Bureau of Labor Statistics BSC Board of Scientific Counselors CDC Centers for Disease Control and Prevention CSTE Council of State and Territorial Epidemiologists DoD U.S. Department of Defense EPA Environmental Protection Agency FACE Fatality Assessment and Control Evaluation Program GPRA Government Performance and Results Act HHE Health Hazard Evaluation HHS U.S. Department of Health and Human Services IOM Institute of Medicine xv
xvi A b b r e v i at i o n s a n d A c r o n y m s MSHA Mine Safety and Health Administration NIH National Institutes of Health NIOSH National Institute for Occupational Safety and Health NORA National Occupational Research Agenda NRC National Research Council OMB Office of Management and Budget OSHA Occupational Safety and Health Administration PART Performance Assessment Rating Tool PEL permissible exposure limit PPT Personal Protective Technology RFA Request for Application USDA U.S. Department of Agriculture