4
Improving the Evaluation Process

The evaluation of eight National Institute for Occupational Safety and Health (NIOSH) programs presented challenges in providing a consistent approach to the assessment of a set of diverse research programs, each involving numerous activities. The goal of the series of evaluations was the improvement of the programs and ultimately—given the mission of NIOSH—the improvement of worker safety and health. This chapter considers how the experience gained during this multiphase evaluation effort could be applied to improving future evaluation efforts.

The following sections are based on the framework committee’s detailed discussions with those who used the framework (members of the evaluation committees), the sponsor’s experience with the evaluation reports, and input from external stakeholders.1 The resulting revised framework (Chapter 3), and suggestions for its use, are intended as a guide for future evaluations of NIOSH programs and may prove useful for evaluations of research programs in other settings. The chapter is not a summary of the recommendations of the eight program evaluations. Those recommendations and the accompanying detailed evaluations of each program are

1

The chapter draws on the November 24, 2008, workshop organized by the framework committee and held at the National Academies, “Evaluating NIOSH Programs: Lessons Learned and Next Steps.” Workshop participants included NIOSH program and senior staff, members of the NIOSH Board on Scientific Counselors, evaluation committee members, framework committee members, and National Academies’ staff. Findings also reflect the experiences relayed throughout the eight studies by evaluation committee members and NIOSH staff.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 75
4 Improving the Evaluation Process T he evaluation of eight National Institute for Occupational Safety and Health (NIOSH) programs presented challenges in providing a consistent approach to the assessment of a set of diverse research programs, each involving numerous activities. The goal of the series of evaluations was the improvement of the programs and ultimately—given the mission of NIOSH—the improvement of worker safety and health. This chapter considers how the experience gained during this multiphase evaluation effort could be applied to improving future evaluation efforts. The following sections are based on the framework committee’s detailed dis- cussions with those who used the framework (members of the evaluation commit- tees), the sponsor’s experience with the evaluation reports, and input from external stakeholders.1 The resulting revised framework (Chapter 3), and suggestions for its use, are intended as a guide for future evaluations of NIOSH programs and may prove useful for evaluations of research programs in other settings. The chapter is not a summary of the recommendations of the eight program evaluations. Those recommendations and the accompanying detailed evaluations of each program are 1 The chapter draws on the November 24, 2008, workshop organized by the framework committee and held at the National Academies, “Evaluating NIOSH Programs: Lessons Learned and Next Steps.” Workshop participants included NIOSH program and senior staff, members of the NIOSH Board on Scientific Counselors, evaluation committee members, framework committee members, and National Academies’ staff. Findings also reflect the experiences relayed throughout the eight studies by evalu - ation committee members and NIOSH staff. 

OCR for page 75
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s available in the individual committee reports (IOM and NRC, 2006, 2008, 2009; NRC and IOM, 2007, 2008a,b, 2009a,b). EVALUATION FRAMEWORK Applying the Logic Model and Assessing Outcomes As discussed in Chapter 2, the logic model is used increasingly by program evaluations to delineate what a program does and accomplishes into the catego - ries of inputs, activities, outputs, and outcomes (see Figure 2-1). The focus of this series of NIOSH program evaluations was on assessing the relevance and impact of the NIOSH program on reducing work-related injuries, illnesses, and hazardous exposures (end outcomes). Initially a number of framework com - mittee members had concerns about developing evaluation criteria that hold a research agency accountable for end outcomes when so many of the factors that affect end outcomes (e.g., workplace conditions, regulations) are outside of the agency’s authority or control. Many members of the evaluation committees shared similar concerns. Several aspects of the framework that were ultimately adopted made the focus on end outcomes more acceptable. The first was the fact that the framework calls for the identification of external factors and provides a fairly extensive list of examples. This provides an opportunity for evaluators to identify obstacles to or promoters of research impact and underscores the changes necessary outside the research agency to improve worker safety and health. A more conventional approach to research evaluation that focuses on knowledge generation would not typically allow for making explicit statements about these obstacles. The second aspect was the clear direction that intermediate outcomes could be used as measures of success in the absence of end outcomes. This was particularly appropriate for exposure-illness relationships that have years of latency before diagnosis. Intermediate outcomes are observable results that could plausibly lead from outputs to end outcomes. Intermediate outcomes (e.g., policy change, adoption of NIOSH-developed tech- nologies in the workplace) may serve as the short- and medium-term proxies for expected end outcomes. Setting the metric for program success at demonstrating an impact on end outcomes is laudable. Evaluation and framework committee members give NIOSH a great deal of credit for holding their research programs accountable for real out- comes that affect life and health. Evaluations of research supported by other federal agencies often focus on output productivity and intermediate outcomes and do not hold the agency accountable for real-world impacts. This may be appropriate for the goals of some research programs, but NIOSH’s focus is on applied research

OCR for page 75
i m P R o v i n g t H e e va l u at i o n P R o c e s s  and thus the end outcomes are appropriately focused on improving worker safety and health. The challenge for the evaluation committees was to search for data on end outcomes while also thoroughly examining the multiple pathways leading to in- termediate outcomes and from there to end outcomes. Particularly in the absence of good end outcome data, a frequent finding, attention was often focused on the most observable pathways between program activities and end outcomes. Addi- tional consideration in future evaluations could be given to important, but often less obvious, intermediate outcomes that include: • Reframing of problems and issues in workplace safety and health in ways that are absorbed into general discussions; • Contributing indirect knowledge—NIOSH research results that build a body of knowledge that as a whole influences regulation or employer action; • Accumulating expertise, among NIOSH personnel and in the broader com- munity of NIOSH partners, to allow quick response as needs arise (i.e., building human capital); and • Building relationships among institutions that need to work together to improve end outcomes (i.e., building social capital). Measures of these types of contributions are seldom readily available as they are often difficult to quantify and assess. The framework committee views reports from external stakeholders as the most accessible source of this information. In future evaluations, the questions posed by evaluation committees to relevant stakeholders could include this larger set of intermediate outcomes. A specific issue discussed during the workshop was whether information and technology transfer-activities should be considered in the evaluation and scoring of the program’s relevance (as in the initial evaluation framework) or of the program’s impact. Research transfer or translation efforts encompass NIOSH program activities conducted to increase the likelihood that research results will impact worker safety and health as well as research aimed at identifying effective translation approaches. External stakeholders frequently use the results of NIOSH research and training to develop programs, processes, materials, or technologies that incorporate research findings in the workplace. Thus, transfer activities are important to the evaluation of both relevance and impact. Because the evaluation framework was designed to assess the efforts of the NIOSH research programs, the framework committee decided to continue to include the evaluation of the extent to which NIOSH programs engage in transfer activities as part of the relevance scoring criteria (Chapter 3).

OCR for page 75
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s The framework committee also noted that the logic model approach has some limitations because of its linearity including challenges in handling the multiple inputs, outputs, and external factors and the iterative actions that may contribute toward an outcome being achieved. Relevance and Impact Ratings As requested by NIOSH from the outset of this evaluation process, the state- ment of task included the stipulation that numerical rating systems from 1 to 5 be used to assess the relevance and the impact that the NIOSH program has had on improving worker safety and health. The initial statement of task did not ex- plicitly state that an integer score was needed, but a subsequent revision made this explicit. Evaluation committee chairs and members indicated that the task of scoring posed significant challenges to the evaluation committees. Initially, many commit- tees struggled with the utility of scoring and were concerned that the restriction to integer scores of 1 to 5 would not convey the nuanced and detailed evaluations they were conducting. Furthermore, some committee members were concerned that a low rating might result in budget reductions or terminated programs. Other com- mittee members thought a high score would not encourage an excellent program to further improve. In several instances scoring took a great deal of the committee’s time both at meetings and subsequently through conference calls and emails. Although this was considered a drawback, some members of the evaluation com- mittees indicated that the rating process promoted a detailed prose commentary and in-depth review. NIOSH staff recognized the challenges that the evaluation committees faced in determining integer-only scores for relevance and impact using a 1–5 scale. How- ever, during the November workshop, NIOSH staff reported that the prospect of receiving impact and relevance scores had a motivational effect, in general, on the NIOSH program staff who were assembling the evidence packages and responding to additional requests for information from the evaluation committees. Throughout the evolution of the framework, the scoring criteria have been clarified. The wording for each of the scores for relevance discusses the extent to which the NIOSH program addresses priority research areas and is engaged in transfer activities; the wording for each of the impact scores discusses the contribu- tion that the NIOSH program has made to worker safety and health. Furthermore, the framework emphasizes that the scoring systems serve as starting points for descriptive text that defines what the scores meant and the program’s strengths and limitations that led to the scores. Although there was initial concern that the scores would be the only endpoint noted from the evaluation reports, the committee is

OCR for page 75
i m P R o v i n g t H e e va l u at i o n P R o c e s s  pleased to see the detailed strategic plans and action plans developed since publica- tion of the evaluation reports in response to the recommendations. Integer Scores Versus Decimal Scores At the outset of the process when developing the scoring criteria, the frame- work committee had extensive discussions about the nature of the scoring scales and the advantages and disadvantages of using decimals. Although a bigger range of integers or the use of decimals could add finer granularity to the ratings, these options present increased challenges in maintaining consistency across the work of a series of evaluation committees. It was also pointed out that use of decimals implies more accuracy than was possible. In revisiting the scoring issues at the end of the eight evaluations, the framework committee decided to stay with the integer approach using a 5-point scale because it offers greater opportunity for consistency. Rating Quality During the workshop, participants discussed the question of whether a specific numerical rating for quality should be added to the task of future NIOSH program evaluations. The issue of developing a process for rating research quality had been discussed early on in the development of the framework, and it was decided that the evaluation of quality was implicit in the evaluation of relevance and impact. Similar thoughts were expressed by workshop participants who stated that the quality of efforts will be seen in the extent and nature of the outcomes. If there is poor qual- ity in the process, a significant intermediate or long-term outcome would not be as likely. Low ratings for program relevance or impact would likely cause program managers to examine research quality as well as other issues such as budget, staffing, or physical plant. Participants noted that the evaluation framework already includes many issues regarding quality. Furthermore, the framework provides opportunities for evaluation committees to include examples of best practices as well to describe the components of an ideal program. Human Capital Outputs The critical role of various NIOSH research programs in the training and de- velopment of scientists and other professionals was discussed during the workshop and by the framework committee. The framework committee did not see the need to numerically rate this parameter because of lack of data (particularly related to outcomes). The evaluation framework addresses many of the relevant human

OCR for page 75
0 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s capital issues through questions about research partners and about training and education as part of the discussion regarding transfer activities. Use of the Framework Document The general consensus of workshop participants and evaluation committee members was that the evaluation framework was useful but that the extent of close adherence to the details of the framework varied among the evaluation committees. Some evaluation committees adhered rather closely to the framework guidelines and structure, and others used it more as a starting point and reference document. Some evaluation committees thought the structure and format presented by the framework for evaluation was rather rigid, while others found that the framework provided committee members with the latitude needed to individualize the evalu- ation to fit a specific NIOSH program. In forming each evaluation committee, one or two members of the framework committee were asked to serve as a liaison or as an evaluation committee member. Additionally, the chair of the framework committee provided an introductory pre- sentation on the framework for each of the evaluation committees. Getting evalu- ation committee members “buy-in” to follow the framework in conducting the evaluation was often a challenge, and further efforts throughout the process may have improved its use and application. Evaluation committee members differed in the extent to which they used the framework when constructing the narrative. One concern was the limited amount of time allotted by many of the evaluation committees in their initial meetings for discussion of the framework, the evaluation process, and the purpose of the evaluation. Furthermore, the framework commit- tee now recognizes that familiarity and understanding of the framework could be enhanced if the chairs of the evaluation and framework committees discussed the framework and its implementation in depth prior to the first meeting of the evaluation committee. The two evaluation committees that evaluated programs with major non- research components (the Health Hazard Evaluation Program and the Personal Protective Technology Program) found that the framework worked well and with only minor adaptations. These studies had slightly revised statements of task and expanded the rating criteria to include the nonresearch components applicable to each study. COMPOSITION OF THE EVALUATION COMMITTEES Producing high-quality and fair assessments begins with the composition and balance of the membership of the evaluation committees. The framework com-

OCR for page 75
i m P R o v i n g t H e e va l u at i o n P R o c e s s  mittee and workshop participants believed the independence and credibility of the external organization conducting the evaluations was critical in convincing stakeholders and other interested parties to participate and provide input to the evaluation committees. In following the processes of the National Academies, each evaluation committee member was selected for his or her professional expertise within the breadth of disciplines relevant to the specific program to be evaluated including expertise in program evaluation. Each committee conducted a thorough bias and conflict of interest discussion at its first meeting. AGENCY INPUTS TO THE EVALUATION Agency Presentations At the first meeting of each evaluation committee, NIOSH program and man- agement staff provided briefings on the goals of the evaluation process from the agency’s perspective and an overview of the program being evaluated. These pre- sentations were professional, thorough, and detailed. Many evaluation committee members indicated that additional time on the first meeting agenda for discussion with NIOSH staff would have been helpful. Additionally, this initial overview would have benefitted from including more information on how NIOSH sees the program as fitting into the broader context of other research efforts on this topic (see section below on Scope of the Field). Framework and evaluation committee members expressed interest in hearing more from the NIOSH program staff on what they view as the program’s challenges and its opportunities for improvement. Greater emphasis on self-assessment prior to external evaluation could provide valuable information and facilitate a greater understanding of each program. Evidence Packages The term evidence package refers to the collection of information provided by NIOSH staff to each evaluation committee. Each evidence package consisted of program descriptions, staffing levels, program goals and objectives, and details on the program and its accomplishments. The evidence packages were structured to follow the logic model developed by NIOSH with consultation from RAND Corporation (Williams et al., 2009). Although the evidence package was the core input to each evaluation, it was one of many sources of information that the evalu- ation committees assessed. Other sources included committee requests to NIOSH for additional information, stakeholder presentations and other input, scientific workshop presentations, and in some cases, site visits.

OCR for page 75
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s Although some evidence packages were quite long (more than 900 pages) and detailed, the NIOSH program staff learned throughout the course of the evalua- tion what evidence was key and what was superfluous to the committee’s work. As a result, later evidence packages were more focused. As on any committee, some members wanted more detail and others preferred the big picture. For many com- mittee members, the amount of information was initially overwhelming, but once the organization of information was understood, each member could identify the sections relevant to his or her contributions to the report. An overview or executive summary of the evidence package is suggested to orient the evaluation committee to program goals, budget, staffing, and congres- sional mandates as well as to provide highlights of the program’s successes and challenges. Several NIOSH evidence packages included such a summary that was helpful to the committee. Because most evaluation committees divided into working groups that focused on specific subgoals, organizing the evidence package in a similar manner would be particularly useful. Providing an online-searchable version that can be categorized and linked in multiple ways might improve committee use of the package. Com- mittee members noted that the ample references provided in the evidence packages for documentation were quite helpful in assessing the scope and relevance of the program’s work. Evaluation committee members found that the majority of the information was substantive and detailed, but some materials tended to present information from a more positive than critical perspective and the quality of writing in some packages was inconsistent. Some committee members expressed concern that projects that had not been as successful in producing intermediate outcomes were not detailed. Although highlighting success is understandable, it is also important to demon- strate the program’s ability to learn from projects that are not successful. The evidence package was particularly useful in outlining data available on intermediate outcomes but evaluation committee challenges included the lack of surveillance data and the lack of information on end outcomes as related to specific program efforts. Pertinent surveillance data on time trends of the occupa- tional health outcomes relevant to the program’s activities should be included in the evidence packages with information, where applicable, about time trends of nonoccupational but relevant health endpoints. For example, assessing the con- tribution of NIOSH to the decline in work-related homicide is difficult without an assessment of changes in the overall homicide rate over time. Limitations of surveillance data should also be discussed. Because many of the programs rely on the same data sources, such as the Survey of Occupational Injuries and Illnesses, it may be efficient in preparing for future evaluations to develop a generic discussion of the strengths and limitations of surveillance data sources that could be adapted as needed by the various programs.

OCR for page 75
i m P R o v i n g t H e e va l u at i o n P R o c e s s  NIOSH staff noted at the November workshop that the goal in assembling the evidence package was to provide a manageable set of evidence for the evaluation that reflected all aspects of the NIOSH program without overburdening NIOSH staff and budget. NIOSH staff indicated that the process of compiling the evidence packages, although time consuming and intense, was a valuable effort because it focused each program on its objectives and provided an opportunity to gather in- ternal knowledge and experience to reflect on the program’s goals and objectives. Assembling the evidence packages offered NIOSH staff the opportunity for long- term retrospective assessment. Now that the packages have been assembled for the eight NIOSH programs, these documents provide the programs with a baseline and benchmark for future evaluations and a valuable compilation for responding to a wide range of requests for program information. Quantity and Quality of the Materials As noted above, the evaluation committees found a great deal of useful informa- tion in the evidence packages. During the November workshop, breakout session participants discussed the need for NIOSH staff to be more consistent in categoriz- ing a program’s efforts according to the components of the logic model, particularly in differentiating activities and intermediate outcomes. Operational definitions and examples that could improve the categorization of components consistently across programs might be helpful for NIOSH staff in preparing the evidence packages. An online resource that provides this information could be a useful tool and potentially could be incorporated as an integral part of NIOSH’s workflow, hopefully reducing the time and staffing burden posed by assembling an evidence package. Budget and Staffing Information Several evaluation committee members stated that greater clarification was needed on budgetary and staffing processes (e.g., recruitment of new staff). This information was often the topic of follow-up questions by the evaluation com- mittees to NIOSH. These issues were particularly important for evaluating the programs that function through a matrix management approach, such as the Hearing Loss Research Program, where budgetary authority and line items were in multiple NIOSH divisions. Scope of the Field and Intramural/Extramural Research The evaluation committees spent considerable time in grappling with how the NIOSH program fit into the larger scope of research efforts funded by the National Institutes of Health (NIH) and other federal agencies, private-sector companies,

OCR for page 75
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s foundations, and other organizations. For example, in the respiratory diseases area, NIOSH-funded efforts examining the mechanisms of respiratory cancers were questioned in light of extensive research at the NIH. More information would be helpful in the evidence packages on how NIOSH sees its role and niche in context with other research efforts in that area. The evidence packages varied in the extent to which information was provided on relevant extramural research, particularly with respect to the types of extramural projects, the levels of extramural funding, and the extent to which the intramu- ral program had input on topics for extramural funding. Many of the evidence packages provided few data on extramural projects, and the committees therefore varied in the extent to which they addressed extramural research. Additionally, the information provided in the evidence packages on the program’s strategic plan- ning efforts did not include much attention to the balance or interaction between the extramural and intramural research portfolios related to the program. Under- standably the NIOSH research programs must appropriately attend to relevant congressional mandates. Knowledge about such mandates would help evaluation committees gain a better appreciation of how the extramural community’s efforts are accounted for in arriving at the intramural research strategy. The evaluation committees also need details on how the intramural research program is engaged in providing input into the development of program announcements or requests for applications for extramural research. Three of the eight evaluation reports specifically recommended greater collaboration and communication between the intramural and extramural aspects of the NIOSH program (IOM and NRC, 2006, 2008; NRC and IOM, 2008b). Research Prioritizing and Planning Process Evaluation committee members expressed the need for evidence packages to include more complete discussions of the prioritization of research agendas and the planning processes used by the NIOSH program to establish priorities. Evalu- ation committees were interested in additional information on how new research programs are initiated and on NIOSH processes for ending efforts that are found to be unproductive. This includes more information on how the NIOSH program identifies emerging research issues. Furthermore, the evaluation committees would have appreciated more information in the evidence packages on processes and balancing of investigator-initiated research versus strategically planned research. Overlap Between Programs Another challenge for NIOSH and the evaluation committees was how to deal with the overlap in areas of research between programs. For example, research

OCR for page 75
i m P R o v i n g t H e e va l u at i o n P R o c e s s  on hearing protective devices is relevant to the NIOSH programs on hearing loss; agriculture, forestry, and fishing; personal protective technology; mining; and construction. However, because of the overlap among programs this research area was not thoroughly addressed by any of the committees, and, therefore, may have been incompletely evaluated. The NIOSH respiratory diseases and mining programs also have overlapping research areas, as is also seen in the traumatic injury and construction areas. Evaluation committees generally relied on the extent to which the issues were discussed by NIOSH staff and presented in the evidence packages. In addressing the issue of how to present research that overlaps multiple pro- grams, it may be helpful for NIOSH to use a consistently defined approach, such as presenting the information in the evidence package for the program that has the lead on the topic, rather than in all areas where the issue relates. Another possibility is that these overlap areas could be described once, then used by different groups preparing evidence packages for different program areas and adapted as needed. This strategy would increase consistency and avoid duplication of staff effort. Evidence Packages: Conclusions and Next Steps After reviewing and considering all the comments regarding the evidence packages, the framework committee believes there has been a positive progression and evolution of the evidence packages over the course of the eight studies. Les- sons learned in assembling and using the evidence packages for the initial program evaluations were passed along (through comments by NIOSH staff, evaluation committee members, and National Academies staff), and adaptations in the content and organization of the packages were made along the way. This was particularly helpful for the latter evaluations as NIOSH staff became more aware of the critical materials and the best ways of organizing the data. As evaluation becomes an integral part of the ongoing work process at NIOSH, the framework committee hopes that systematic data collection and perhaps online tools will be available to collect and analyze data for future evaluations and their evidence packages. Additional points to consider in further efforts to produce evidence packages include: • Staff training on the components of the logic model; • Continued sharing of ideas on improving the evidence packages; and • An executive summary or overview of the evidence package that outlines the basic information for each program (budget and staffing information, list of major goals, highlights of program successes and challenges). A summary that serves as the introductory section of the evidence package could also provide a guide to the materials and a map of the contents.

OCR for page 75
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s The framework committee acknowledges the huge effort made by NIOSH staff in assembling the evidence packages and hopes that the institutional commitment that NIOSH made to this evaluation effort is reflected in NIOSH staff evaluations that value the time and effort that was spent in this endeavor. Site Visits Evaluation committees that had the opportunity to make site visits to NIOSH facilities found these visits quite informative to the evaluation process. Site visits provided greater context for the evaluation and allowed committee members an opportunity to see the research and program facilities and interact directly with more program staff. Committee members noted that seeing the day-to-day work- ing structure of the program was worth the additional investment of energy and resources. However, given the distributed nature of some programs with many locations and no one central facility, it was not practical for some evaluation com- mittees to conduct a site visit. The site visits highlighted the importance to the evaluation committees of in- formal interactions with NIOSH staff members including opportunities to observe program staff in their work environments, ask questions, and discuss issues outside of a formal presentation setting. STAKEHOLDER INPUT Input from external stakeholders was vital in the evaluation process, particu- larly given the lack of surveillance and other end outcome data related to deter- mining the program’s impact on reducing hazardous exposures or reducing worker injuries or illnesses. Stakeholders may include individuals from labor unions and other worker groups; employers; manufacturers; health and safety and other pro- fessional organizations; federal, state, and local regulatory and public health agen- cies; certifying organizations; the medical community; academic institutions and groups; agricultural, marine, and other nontraditional industry groups; and the international counterparts to NIOSH and the aforementioned groups. In accordance with the policies of the National Academies for the formation of balanced committees, all of the evaluation committees included individuals from labor organizations, industry, and other stakeholder groups among its subject- matter experts. The insights of such members during committee deliberations were valuable, but cannot be considered to represent all stakeholders effectively. Therefore, the evaluation framework included instructions to the evaluation com- mittees to seek additional stakeholder input. A significant issue the committees faced, however, was how best to use the limited available time and resources to obtain effective and meaningful stakeholder input.

OCR for page 75
i m P R o v i n g t H e e va l u at i o n P R o c e s s  Although the evaluation committees lacked the time and resources to con- duct a rigorous scientific survey of external stakeholders, well-balanced input was sought through active mechanisms, such as presentations at committee meetings, or through passive mechanisms, such as requests for written input disseminated via the website or e-mail. Evaluation committees identified stakeholders through lists provided by the NIOSH programs, through recommendations from com- mittee members, and through open calls for input. Some evaluation committees provided the opportunity for stakeholders to submit input anonymously through online questionnaires. Discussions on stakeholder input during the November 2008 workshop indi- cated that although all agreed that meeting time devoted to stakeholder input was quite valuable, workshop participants had a mixed reaction regarding the value of seeking comments through a website or online mailing. This approach offered another avenue for receiving comments, but some committees received few mean- ingful responses in this way. Explicitly clarifying the nature of information being sought from stakeholders and widely disseminating the availability of opportunities for comment may improve the utility of seeking input online. The adequacy of these approaches must be decided on independently by each evaluation committee. Flexibility and creativity in obtaining stakeholder input are necessary because no single, prescribed method will be suitable for all evaluations. In addition to the methods described above, future committees may consider obtaining input through the National Occupational Research Agenda (NORA) Liaison Committee, which has worked since 1995 to seek regular input from differ- ent types of stakeholder groups. Focus groups of external stakeholders also could provide important insights. Evaluation committees identified several issues in determining how best to use stakeholder input. Although stakeholders often have an inherent biases, their suggestions regarding current and needed NIOSH research efforts can bring the realities of the workplace into focus. Furthermore, stakeholders can relay accounts of responses to NIOSH program activities—the necessary intermediate outcomes that could ultimately lead to end outcomes. TIMELINES FOR EVALUATIONS The contract between NIOSH and the National Academies specified that each evaluation report was to be delivered to NIOSH 9 months after the first evaluation committee meeting. Two of the eight committees were able to meet that deadline, another two reports were released within 12 months, and the remaining four re- ports took 15 to 17 months (Table 1-1). The discussion during one of the breakout sessions of the November workshop focused on the realities of the 9-month timeline and options for future studies. Par-

OCR for page 75
 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s ticipants suggested that nine months and three meetings did not provide enough time, particularly for the full evaluation of large-scale programs. Reviewing the evidence package and receiving stakeholder input took considerable amounts of time at the first two evaluation committee meetings. The framework committee proposes that a 12- to 14-month time frame with four meetings would be preferable so that the evaluation committee could have more time to gather information, care- fully consider the body of evidence as a whole, determine the ratings, and develop its recommendations. A longer time line would allow time for an additional meet- ing. Consideration could be given to holding one of the meetings, where feasible, in proximity to the site where the majority of the research program resides, allow- ing for a better understanding of the context and a more relaxed discussion with research staff about their priority concerns for success. Smaller programs could potentially be evaluated in the 9-month time frame. The framework committee understands and appreciates the problems that NIOSH encountered as a result of the delays in completing some of the reports, such as the impact on the budget planning cycle. Given the number of variables (including time to assemble the committee, scheduling committee meetings and site visits, delivery of the evidence packages, and the timeline for the peer-review process) and the complexity and depth of analysis required, experience has shown that allowing more time for the conduct of future evaluations would be prudent. EVALUATION COMMITTEE REPORTS AND RECOMMENDATIONS Report Format NIOSH staff members expressed an interest in a more consistent format for the evaluation reports. The framework committee noted that the evaluation framework provides a suggested table of contents for evaluation reports, but leaves decisions on the organization and format to the discretion of the individual evaluation commit- tees. The framework committee believes any efforts to further prescribe the report format might jeopardize the creativity and focus of the evaluation committees. Part of the advance training for evaluation committee chairs might include attention to the report format and the recommended structure. Evaluation Committee Recommendations All of the evaluation committee reports included recommendations to NIOSH pertinent to improving the relevance and impact of the program. The recommen- dations addressed a broad array of issues, including enhanced surveillance and monitoring; strategic planning and goal setting; improvements in NIOSH internal

OCR for page 75
i m P R o v i n g t H e e va l u at i o n P R o c e s s  and external collaborations; increased integration of extramural and intramural research priority setting; and the need for further emphasis on research-to-practice efforts. Although most of the recommendations were of a general nature, some were highly specific to the individual program. At the November 2008 workshop, NIOSH staff expressed the challenges they face in addressing recommendations that emphasize increased research efforts without accompanying suggestions on what activities should be curtailed or stopped. The “do-more” recommendations far outnumbered the “do-less” recom- mendations. Although NIOSH staff appreciated the confidence in the programs implied by these recommendations, they also expressed interest in receiving more guidance from the evaluation committees on where and how to set priorities, particularly if faced with static or declining resources. A suggestion was made that future evaluation committees provide recommendations from both perspectives (increases in resources and flat or declining resources). Alternatively, ranking or providing a priority order of the recommendations was also suggested to assist program managers. After consideration of these issues, the framework committee suggests that if only a few recommendations are made (e.g., fewer than five) that is, in itself, a de facto prioritization. By choosing a few specific recommendations, as opposed to offering numerous recommendations, it is likely that all of the evaluation committee’s recommendations are of major importance. If the report contains a large number of recommendations (more than 10), then the evaluation committee should make an effort to discuss priorities. The discussion in the supporting text should emphasize and fully describe the committee’s thoughts on what needs to occur first. In some cases, recommendations are related and time ordered, and so prioritization must reflect these issues. Like most organizations, NIOSH has limited resources. Whenever possible, the evaluation committees should try to consider the optimal allocation of those finite resources within a specific program when making recommendations. Although funding and cost must always be considered, the evaluation committees should not allow funding constraints to be so limiting that they restrict the committee’s freedom and the breadth of its collective thinking in addressing its task. By logical extension, such restrictions could curtail the usefulness of its final recommenda- tions. In providing constructive criticism of the agency and vision for its progress, the evaluation committee should not be constrained to treat funding limitations as if they are immutable. Similarly, the framework committee thought that decisions about prioritizing and categorizing the recommendations are best determined by the individual evalu- ation committees as the recommendations may be organized differently depending on the program’s specific organizational structure and programmatic needs.

OCR for page 75
0 e va l u at i n g o c c u Pat i o n a l H e a lt H a n d s a f e t y R e s e a R c H P R o g R a m s The framework committee also emphasizes the constraints of evaluating a set of individual programs. Pursuing the “top” priority recommendation for each individual program might not be the optimal approach to achieve overall NIOSH objectives. The prioritization of recommendations for the entire organization re- quires additional information and further analysis not available to or appropriate for the individual program evaluations. SUMMARY The evaluation of eight NIOSH programs within four years was made as con- sistent as possible by the initial establishment of an evaluation framework. Having a common statement of task and evaluation approach in the use of the logic model provided a starting point for each evaluation committee. Furthermore, knowing that the endpoint involved the application of defined scoring criteria for rating the program’s relevance and impact on reducing work-related injuries, illnesses, or hazardous exposures provided a common definitive goal for the evaluation committees. As with any multiphase effort involving numerous agency staff and multiple committees, there were variations in how the task was completed. NIOSH is to be commended on its focus on end outcomes and for holding its research and program staff accountable for progress toward those outcomes. Increased opportunities, particularly informal opportunities, for NIOSH staff to discuss issues with evaluation committees would be helpful, as would increased attention by evaluation committees to some of the more indirect measures of intermediate outcomes. This chapter has addressed both the framework and its application. The evalu- ation committee’s experiences with the use of the framework document were vari- able. A number of areas in which the evaluation process could be improved were highlighted, including the provision of more information in the evidence packages on priority setting efforts and on budget and staffing; further input on extramural research and the connections between the intramural and extramural program objectives; ensuring plenty of opportunities for input by external stakeholders and agency staff; considerations regarding the timeline for the evaluations; and options for other types of recommendations or ways of categorizing recommendations in the evaluation reports. The framework committee has made some changes to the evaluation framework (Chapter 3) based on this assessment of lessons learned, and as indicated in the following chapter has thoughts on future evaluation efforts.

OCR for page 75
i m P R o v i n g t H e e va l u at i o n P R o c e s s  REFERENCES IOM and NRC (Institute of Medicine and National Research Council). 2006. Hearing loss research at NIOSH. Committee to Review the NIOSH Hearing Loss Research Program. Rpt. No. 1, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. IOM and NRC. 2008. The personal protective technology program at NIOSH. Committee to Review the NIOSH Personal Protective Technology Program. Rpt. No. 5, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. IOM and NRC. 2009. Traumatic injury research at NIOSH. Committee to Review the NIOSH Trau- matic Injury Research Program. Rpt. No. 6, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. NRC and IOM. 2007. Mining safety and health research at NIOSH. Committee to Review the NIOSH Mining Safety and Health Research Program. Rpt. No. 2, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Acad- emies Press. NRC and IOM. 2008a. Agriculture, forestry, and fishing research at NIOSH. Committee to Review the NIOSH Agriculture, Forestry, and Fishing Research Program. Rpt. No. 3, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. NRC and IOM. 2008b. Respiratory diseases research at NIOSH. Committee to Review the NIOSH Re- spiratory Diseases Research Program. Rpt. No. 4, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. NRC and IOM. 2009a. The health hazard evaluation program at NIOSH. Committee to Review the NIOSH Health Hazard Evaluation Program. Rpt. No. 7, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Acad- emies Press. NRC and IOM. 2009b. Construction research at NIOSH. Committee to Review the NIOSH Construc- tion Research Program. Rpt. No. 8, Reviews of Research Programs of the National Institute for Occupational Safety and Health. Washington, DC: The National Academies Press. Williams, V. L., E. Eiseman, E. Landree, and D. M. Adamson. 2009. Demonstrating and communicating research impact: Preparing NIOSH programs for external review. Santa Monica, CA: RAND.

OCR for page 75