National Academies Press: OpenBook
« Previous: Evaluation in Action:Examples and Resources
Suggested Citation:"Reflections by Eden Kingon Themes and Next Steps." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×

7

Reflections by Eden King on Themes and Next Steps

The analysis in this chapter was commissioned by the planning committee for this workshop. Eden King, Rice University, is an expert in the field and was tasked to synthesize key ideas, connections, and themes shared at the meeting, as well as to suggest actionable next steps to “move the needle” on the evaluation of sexual harassment prevention efforts. The opinions expressed in this chapter reflect the views of Eden King and are written in first person to retain her voice. This shift to a first-person narrative distinguishes her paper from the third-person summaries of workshop discussions in the previous chapters.

THE UNIQUE CHALLENGE

Measurement and evaluation scholars and practitioners deal with several challenges (see American Educational Research Association [2014] for a review of some of these issues). What is unique about measurement and evaluation in the context of efforts to address sexual harassment is that all of the challenges apply. In other words, everything that makes good measurement and evaluation difficult is amplified in the context of sexual harassment.

The challenge of conducting high-quality evaluation of sexual harassment prevention efforts includes such issues as selection bias, social desirability concerns, construct ambiguity, event history, consequential validity, emotional reactivity, confidentiality, legal considerations, introspective

Suggested Citation:"Reflections by Eden Kingon Themes and Next Steps." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×

biases, retrospective biases, and more. At the same time, evaluation plays an essential role in the extent to which any program can be implemented effectively and sustained consistently over time. This tension between the challenges and the essential nature of high-quality evaluation in harassment prevention served as the impetus for a workshop. This paper briefly synthesizes my perspective on some of the major themes that emerged in the presentations and discussions during the workshop, as well as my own personal reflections and suggestions for moving forward.

THEMATIC SYNTHESIS

Theme 1: Evaluation as a systematic, strategic process. Workshop presenters described a variety of best practices in training and program evaluation that might be applied to the specific case of sexual harassment prevention efforts. These descriptions included frameworks developed in intervention1 and implementation science,2 as well as more frameworks that are typically applied to organizational training evaluation (Salas et al., 2012). Each of these disciplinary lenses offers unique emphases, but they also converge on a critical, core common principle: Evaluation can and should be a systematic and strategic process. The specific steps or generative questions vary from one framework to another (or from one disciplinary lens to another), but the overarching commonality is that evaluation will be most defendable and informative to the extent that it is based on a systematic process.

Theme 2: This process must begin at the end. Workshop presenters and attendees also converged in articulating that the systematic process of evaluation of sexual harassment prevention efforts must begin with a focus on the intended outcomes of the efforts. In other words, what are the goals of the effort itself? These might be individual-level goals (determined via an individual needs analysis or a change readiness assessment) or unit-level goals (determined via an organizational needs analysis or assessment; see Salas et al., 2012), or both. Clearly thinking through and specifying in writing what programs are intended to achieve is a critical first step in any systematic program evaluation.

Theme 3: Metrics must align with goals. It is fairly straightforward to communicate that the content of program evaluation should match the

___________________

1 Available: https://www.nap.edu/catalog/26279.

2 Available: https://www.nap.edu/catalog/26279.

Suggested Citation:"Reflections by Eden Kingon Themes and Next Steps." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×

goals of the program—in other words, that we should measure what we set out to change. What is less obvious, but nonetheless equally important, is that the measurement method should also align with the goals of the program. If the goal of a particular intervention is to increase knowledge about sexual harassment policies, for example, evaluators should assess precisely that knowledge (i.e., the goal content). The method of the assessment also needs to align with the goal and content—knowledge can be assessed in a direct manner through a declarative knowledge test. As another example, if the goal of an intervention is to improve the everyday lived experiences of women in a cell biology department, a test would not be an effective methodological approach. Instead, interviews or focus groups may be a method that would allow for deeper analysis of those everyday experiences of a small number of women. Metrics should match goals with regard to content and method.

Theme 4: Outcomes exist at many levels and in many forms. One of the features of sexual harassment prevention efforts is that they are often designed with multiple explicit purposes—they set out to affect not only short-term, proximal, or immediate goals (like increasing knowledge and awareness), but also longer-term, distal, or systemic goals (like improving the recruitment and retention of female faculty in male-dominated fields). These outcomes might relate to changes in cognitions, emotions, behaviors, policies, practices, or demographic patterns. Intentional and unintentional outcomes might also vary with regard to the expected level of influence—individual, unit, or institutional. This variety of possibilities reinforces the need to clearly specify and measure the goals and expected outcomes of a given program or set of programs.

Theme 5: There is no one, single best method or measure. Workshop attendees discussed a variety of existing tools to assess institutional climates (e.g., masculinity contest culture; Berdahl et al., 2018) and individual experiences (e.g., psychological safety; Edmonson, 1999). Case studies included examples of both qualitative and quantitative approaches. An implicit assumption that needs to be explicit is that, unlike other fields of science or medicine, there is no one, single “gold standard” assessment tool for sexual harassment prevention efforts. And importantly, this lack of a gold standard tool is not because tools haven’t been designed, but because anti-harassment programs serve so many different goals (at so many different levels, see above) that no single measure or metric could possibly be sufficient. This is a profoundly important conclusion that is often misunderstood.

Suggested Citation:"Reflections by Eden Kingon Themes and Next Steps." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×

Theme 6: Equity cannot be ignored. Substantial discussion emerged in the workshop (in both the verbal discussions and the synchronous text chats) at the prompting of attendees regarding the intersections between harassment (and harassment prevention efforts) and equity (and diversity, equity, and inclusion efforts). One key aspect of this discussion is acknowledging and addressing the notion that our methods might be constrained by the nature of privilege and the ways it perpetuates the status quo. For example, as one workshop participant queried, “When the culture and climate is harassment-promoting, how do we get buy-in, institutional support, etc., to be able to implement and evaluate programs over time?” Who does and who does not get asked what questions and when and how and by whom has been inextricably intertwined with identity, status, and power. Evaluation conclusions must be confronted in light of the reality of inequity in our institutions.

REFLECTIONS FOR MOVING FORWARD

Context matters. Nearly every presenter in the workshop mentioned the importance of context, and indeed, it is a critical component of every model that could be applied to program evaluation. In the case of harassment prevention, Sexual Harassment of Women: Climate, Culture, and Consequences in Academic Sciences, Engineering, and Medicine (NASEM, 2018; hereafter, National Academies), identified some specific aspects of STEM (science, technology, engineering, and mathematics) and higher education contexts that may exacerbate harassment and increase the complexity of its prevention. These STEM/higher education–specific elements of context should not be overlooked when it comes to evaluation. For example, evaluators must consider how the embedded hierarchies and dependencies of academic life (e.g., the precariousness of academic life pre- or without tenure, principal investigators’ funding of students) influence the ways that we can and should measure outcomes. As another example, the unique role of faculty governance may play into the evaluation process. It would also be helpful to clarify the ways that a nonprofit orientation influences leadership behaviors or motivations. The role of the scientific process itself, for example, in when and how work gets done (e.g., in the field or in the lab) can also influence who, how, when, and where evaluation is needed in a manner that is unique to the context of STEM in higher education.

Leverage varied methods. The tried-and-true (traditional) methodological approaches that are brought to bear on typical program evaluation

Suggested Citation:"Reflections by Eden Kingon Themes and Next Steps." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×

processes (such as attitude or climate surveys, knowledge tests, focus groups, and interviews) have great value in the evaluation process for harassment prevention efforts. Indeed, triangulating across these methods (i.e., looking for consistent themes from multiple methodological lenses) is a particularly compelling way to draw robust conclusions. In addition, I think there is opportunity to utilize a more diverse set of methodological tools. New and underutilized approaches, such as text mining, experience sampling or pulse surveys, and unobtrusive observation, may provide helpful information and overcome some of the methodological challenges that are common in other approaches. These techniques could be useful complements to more traditional approaches.

Harassment prevention and diversity, equity, inclusion (DEI) are inextricably intertwined. Though different individuals, offices, and efforts are often directed toward harassment prevention and what has been labeled DEI, these topics are inherently interrelated. Efforts to prevent harassment, for example, should attend to the groups of people who are most likely to perpetrate or be targets of harassment. As another example, inclusion and equity may be key elements of organizational contexts that reduce harassment. The treatment of these efforts as distinct has set up a false separation that could ultimately make it harder to achieve the goals of each set of programs. It is likely that many of the goals of these programs are overlapping (e.g., increasing the representation of white women and women of color in STEM). From an evaluation standpoint, then, measurement may be more efficient if redundancies are explicitly addressed.

NEXT STEPS

The observations noted above and in the workshop highlight the challenging variety of requirements for high-quality evaluation of harassment prevention efforts. In my view, it would not be possible to specify one single strategy or approach to this process that could adequately inform effective evaluation across the wide range of contexts and goals. The National Academies report (NASEM, 2018) on sexual harassment and the ongoing work of the National Academies Action Collaborative on Preventing Sexual Harassment in Higher Education (see Chapter 1) yielded a helpful rubric [see Appendix D] that outlines some goals for these efforts that might serve as a starting place. Re-envisioning this checklist, I think it would be helpful to develop and communicate guiding questions or principles to evaluate harassment prevention efforts.

Suggested Citation:"Reflections by Eden Kingon Themes and Next Steps." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×

That is, rather than specifying exactly what measures or metrics to apply and how in every case, it would be helpful to explicitly enumerate a set of core elements of high-quality evaluations of harassment prevention programs. Some workshop attendees commented that it might be motivating to create a “scorecard” or ranking systems through which institutions could strive for external recognition. While there may be value in this kind of model, I think it might be especially valuable to identify the key components of evaluation approaches that can be customized to the needs of a particular institution with particular goals. As a potential example, a guiding principle might be that evaluations should be a systematic, strategic process. As another example, it could be suggested that sexual harassment prevention effort evaluation processes should center the perspectives of marginalized members of the community. Describing such conceptual principles, guiding questions, or considerations that would clarify who should be assessed, when, in what way, and under what conditions could provide a more balanced and potentially instructive step for moving forward.

In addition to providing this guidance, I think an exciting next step for the National Academies’ community would be to ideate, encourage, implement, and evaluate efforts in research and practice that explore the integration of DEI and harassment. I understand that a National Academies workshop on DEI is being organized as I type this paper; I would strongly encourage the explicit consideration of sexual harassment prevention (and the typical elements of sexual harassment prevention practice and research) in this conversation. These research programs and practices have been driven by different disciplines and interest groups, thereby artificially perpetuating their separation and limiting their development. Indeed, I think sexual harassment and DEI scholars and practitioners have the opportunity to find and build new synergies across intervention implementation and evaluation.

In sum, the workshop reflected a thoughtful conversation with great insights about the challenges of effective evaluation of sexual harassment prevention efforts and strategies through which these challenges might be met. I hope that these reflections are helpful in propelling this critical work forward.

Suggested Citation:"Reflections by Eden Kingon Themes and Next Steps." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×
Page 47
Suggested Citation:"Reflections by Eden Kingon Themes and Next Steps." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×
Page 48
Suggested Citation:"Reflections by Eden Kingon Themes and Next Steps." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×
Page 49
Suggested Citation:"Reflections by Eden Kingon Themes and Next Steps." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×
Page 50
Suggested Citation:"Reflections by Eden Kingon Themes and Next Steps." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×
Page 51
Suggested Citation:"Reflections by Eden Kingon Themes and Next Steps." National Academies of Sciences, Engineering, and Medicine. 2021. Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/26279.
×
Page 52
Next: References »
Evaluating the Effectiveness of Interventions to Prevent and Address Sexual Harassment: Proceedings of a Workshop Get This Book
×
Buy Paperback | $50.00 Buy Ebook | $40.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Rising awareness of and increased attention to sexual harassment has resulted in momentum to implement sexual harassment prevention efforts in higher education institutions. Work on preventing sexual harassment is an area that has recently garnered a lot of attention, especially around education and programs that go beyond the standard anti-sexual harassment trainings often used to comply with legal requirements.

On April 20-21, 2021, the National Academies of Sciences, Engineering, and Medicine hosted the workshop Developing Evaluation Metrics for Sexual Harassment Prevention Efforts. The workshop explored approaches and strategies for evaluating and measuring the effectiveness of sexual harassment interventions being implemented at higher education institutions and research and training sites, in order to assist institutions in transforming promising ideas into evidence-based best practices. Workshop participants also addressed methods, metrics, and measures that could be used to evaluate sexual harassment prevention efforts that lead to change in the organizational climate and culture and/or a change in behavior among community members. This publication summarizes the presentations and discussion of the workshop.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!