6

Evaluation in Action: Examples and Resources

Many higher education institutions are actively working to evaluate their efforts to address sexual harassment, but those evaluations may or may not be consistent with what research has identified as best practices. To this end, the workshop included three examples of organizations that have had success with aspects of evaluating their interventions. Additionally, this chapter offers resources for applying implementation science to the evaluation of sexual harassment prevention efforts.

EXAMPLES OF EFFORTS TO EVALUATE SEXUAL HARASSMENT INTERVENTIONS

Michael Leiter, Acadia University and Deakin University, provided the first of three examples of evidence-based efforts to evaluate sexual harassment interventions. He discussed his research on an initiative designed to promote and evaluate workplace civility, including how employees show respect and disrespect at work, called strengthening a culture of respect and engagement (SCORE). This initiative, which has been implemented in various countries, including in Australia and North America, is held in five sessions, each 90 minutes in length, over 2 to 4 weeks apart; it assesses civility, incivility, and levels of institutional intimidation among employees. SCORE also includes a facilitated group process to strengthen social climate in the workplace. It has been used in a variety of workplaces and is based on the components of the CREW (civility, respect, and engagement

at work) initiative, which has been deployed in veterans hospitals across the United States.

Social engagement scales were created to assess interactions with ratings from civil to uncivil. The Maslach Burnout Inventory, measured on three dimensions—exhaustion, cynicism, and inefficacy—assesses employees on scales ranging from engaged to burnout. The evaluation design also included treatment and control groups. Over time, based on results from various workplaces, the program resulted in higher sources of civility among coworkers and individuals, while sources of incivility also decreased among supervisors (see Figure 6-1).

To evaluate impact, Leiter noted that he and his colleagues relied on diverse sources of information, including surveys, institutional data, interviews, and observations, including validated measures to assess experience at work and social dynamics. The evaluation design included before and after assessment and comparison groups.

Following Leiter’s presentation, Jane Stapleton, Soteria Solutions, discussed the Workplace Violence Prevention and Response Program Collaboration of the National Oceanic and Atmospheric Administration (NOAA), designed to identify sexual assault and sexual harassment (SASH) protective and risk factors for different NOAA workplace environments. The effort was designed to determine social norms, recognition, bystander behaviors, and prevalence of SASH and other inappropriate behaviors. It was also developed to support NOAA-specific strategies that are knowledge- and skills-based and tailored to NOAA workplace environments. The current evaluation uses public health approaches to prevention, based on data to define the problem, inform solutions, make course corrections, and measure outcomes. The evaluation begins with the start of the project, involves multiple methods, actively engages the target audience, and uses a variety of indicators to measure outcomes. The evaluation cycle begins with a needs assessment, followed by a workplace culture survey and a formative evaluation, leading to piloting the prevention strategy. This work is followed by course correction and refinement strategies, a program evaluation, and, finally, outcome evaluation.

To develop the needs assessment, Stapleton explained, focus groups and interviews with NOAA employees were conducted, along with examinations of existing data on SASH complaints and investigations. A review of current training was also conducted. The needs assessment yielded several learning outcomes, including how variable the work environments, cultures, and readiness levels are across the agency. Stapleton added that

the assessment reinforced the need for consistent messaging throughout the organization and core training, as well as the importance of leveraging social norms to change behaviors.

In terms of program evaluation, Stapleton noted that the key interest is around whether the program is meeting its objectives. To determine if the program is being implemented as designed, the investigators are planning to track participation and assess how the program implementation is functioning from administrative, organizational, and personnel perspectives.

In terms of outcome evaluation, the focus will be on assessing the:

• increase in recognition of SASH and other inappropriate behaviors;
• increase in bystander, de-escalation, and strategic resistance behaviors;
• increase in positive social norms that support safe and respectful workplace environments;
• increase in reporting of SASH and other inappropriate behaviors; and
• decrease in SASH and other inappropriate behaviors at NOAA.

Qualitative methods have been important to this work, stated Stapleton. As one participant noted, qualitative methods can delve into intersectional experiences, process, and power dynamics, and highlight what is going on in ways that are sometimes hard to capture with surveys.

Melissa Kwon, University of California, Berkeley, discussed the university’s PATH to Care Center program, which includes sexual harassment prevention initiatives, advocacy, training, and healing services. The cornerstone of the program is its population-specific prevention managers who work directly with undergraduate students, graduate students, staff, and faculty.

Kwon discussed the program’s evaluation process, which includes determining short-, intermediate-, and long-term outcomes of prevention programs and survivor support programs; developing goals of training and other programs; examining existing evaluation questions; mapping existing evaluation questions onto outcomes; identifying indicators for each outcome; developing new survey questions to fill gaps; designing new interview and focus group questions; and creating a “menu” of evaluation questions. The evaluation “menu” allows for individual program managers to pull from a list of questions to support their evaluation (see Figure 6-2). The program also captures information through a service tracking form.

Kwon noted that the staff are in the process of refining their data collection systems, continuing their routine data collection, demonstrating to the team how data are being used, determining how leadership and the team want to use the evaluation data, and assessing what data will be included in a shared dashboard of information.

APPLYING IMPLEMENTATION SCIENCE TO UNDERSTAND CONTEXTUAL FACTORS THAT FACILITATE OR IMPEDE SUCCESSFUL EFFORTS

In addition to the efforts to evaluate sexual harassment initiatives described above, there are also several examples of evidence-based frameworks that can be applied to the evaluation of sexual harassment prevention efforts. The commissioned paper by Raechel Soicher, Oregon State University,¹ provides several such examples, including resources that outline how to apply models of implementation science to the evaluation of sexual harassment prevention efforts. In addition, Appendix E includes resources and worksheets based on these models.

As described by Soicher and Kathryn Becker-Blease, Oregon State University, the first step in conducting an implementation study is selecting a model, which can help at all phases of a research study, from planning to execution to evaluation. As described in Chapter 3, one resource that can help facilitate this process is the use of a logic model. Figure E-1 in Appendix E offers a logic model of an implementation science project that could be adapted to a sexual harassment prevention program. The model begins with an assessment of starting conditions, in which one outlines the problem or gap; continues with an analysis of inputs and proximal outcomes; and concludes with the identification of distal outcomes, including the long-term outcomes of the project. Figure E-2 includes a logic model worksheet to support this analysis, offering options for a user to identify what factors are relevant to the project, particularly as related to the problem; the program, intervention, or policy and implementation strategies; and outcomes.

The Consolidated Framework for Implementation Research (CFIR), as described in Chapter 3, synthesizes constructs from across multiple implementation theories and provides a consistent taxonomy for building a knowledge base. The framework is commonly cited in the literature, as

___________________

¹ Available: https://www.nap.edu/catalog/26279.

presented by Becker-Blease, and can be effectively applied to the evaluation of sexual harassment prevention efforts. Table E-1 (in Appendix E) includes a worksheet to support the application of the CFIR to sexual harassment prevention, including offering guiding questions for a user to assess intervention characteristics; outer-setting characteristics (e.g., stakeholders’ needs and resources, external policies and incentives); inner-setting characteristics (e.g., structural characteristics, culture, implementation climate, readiness for implementation); characteristics of individuals; and process issues (e.g., planning and engaging).

Also as described in Chapter 3, the Practical, Robust Implementation and Sustainability Model (PRISM) can improve implementation efforts, as well as helping to develop potential intervention and implementation questions; it can be applied to the evaluation of sexual harassment prevention efforts. Table E-2 (in Appendix E) includes a worksheet to guide the application of this model to sexual harassment prevention efforts. The worksheet includes guiding questions to help a user describe and assess the program intervention, environment, implementation and sustainability infrastructure, and program recipients at the organizational and consumer levels.

These resources are select examples of implementation science frameworks and models that can be used as a guide for higher education institutions in applying implementation science to evaluation of sexual harassment prevention efforts. Additional information about these and other models can be found in the commissioned paper by Soicher.²

___________________

² Available: https://www.nap.edu/catalog/26279.

This page intentionally left blank.