National Academies Press: OpenBook

The Science of Effective Mentorship in STEMM (2019)

Chapter: 6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?

« Previous: 5 Mentorship Behaviors and Education: How Can Effective Mentorship Develop?
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 127
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 128
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 129
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 130
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 131
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 132
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 133
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 134
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 135
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 136
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 137
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 138
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 139
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 140
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 141
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 142
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 143
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 144
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 145
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 146
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 147
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 148
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 149
Suggested Citation:"6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How?." National Academies of Sciences, Engineering, and Medicine. 2019. The Science of Effective Mentorship in STEMM. Washington, DC: The National Academies Press. doi: 10.17226/25568.
×
Page 150

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

6 Assessment and Evaluation: What Can Be Measured in Mentorship, and How? Assessments and evaluations enable institutions and individuals to determine if mentor- ship is achieving the desired goals and outcomes. However, there is a folly in hoping for a specific outcome if measures used to evaluate what is happening focus on something else entirely (Kerr, 1995). To fully understand mentorship, evaluation measures would ideally address both mentorship processes and mentorship outcomes and the system factors that can profoundly shape it.1 Measurement and evaluation play a critical role in assessing interventions, determining organizational priorities, and developing and testing theory—three key elements that underlie understanding the effectiveness of mentorship. In addition, initiatives and their outcomes that are assessed consistently are better positioned to provide insights for improvement and long-term outcomes. Therefore, intentionality is needed when selecting measures to assess mentorship and outcomes of mentorship.2 This chapter draws on theories and frameworks from Chapter 2 to highlight how to evaluate mentorship in its various forms and contexts. Box 6-1 highlights how theory may inform the concepts that are discussed, like the process-oriented model shown in Figure 6-1. Evaluating the effectiveness of mentorship depends on both quantitative and qualitative measures and tools. Ideally, such measures identify how mentorship or spe- cific mentorship factors contribute to desired outcomes and provide specific insights into how interventions work. Integrating theories is important because they make explicit the 1    s A articulated in the discussion of theories applicable to mentorship in Chapter 2. 2    ntentionality I is defined as a calculated and coordinated method of engagement to effectively meet the needs of a designated person or population within a given context. 127 PREPUBLICATION COPY—Uncorrected Proofs

128 Th e S c i e n c e o f E f f e c t i v e Me n to r sh i p i n ST E M M BOX 6-1 Theory and the Process-Oriented Model of Mentorship Concepts from and aligned with ecological systems theory are evident in the process-oriented model of mentorship used to review existing measures in this chapter. Core premises from this theory draw attention to assessment and evaluation of person-level and environmental-level factors in exam- ining the processes and outcomes of mentoring experiences as well as the contexts in which those experiences occur. Ecological systems theory, along with others such as social exchange theory, draws attention to both the technical aspects (e.g., research skill development) and the relational aspects (e.g., motivation) of mentorship. Figure 6-1. Simplified process-oriented model of mentorship FIGURE 6-1  Simplified process-oriented model of mentorship. SOURCE: Adapted from Eby et al., 2013. mechanisms by which mentorship is expected to operate and therefore the appropriate measures to use to assess mentorship activities and programs. This chapter focuses on quantitative measures, although the importance and value of qualitative assessment is acknowledged. PREPUBLICATION COPY—Uncorrected Proofs

A s s e s s m e n t a n d E va l u a t i o n 129 CONSIDERATIONS IN ASSESSMENT OF MENTORSHIP As mentioned in Chapter 4, it is challenging to determine how to assess effective mentorship at the program, institutional, individual, and relationship levels across sci- ence, technology, engineering, mathematics, and medicine (STEMM) disciplines and career stages. In selecting appropriate measures, there are at least three important ques- tions to consider: 1. How can we quantify “quality” mentoring relationships and programs—and at what time and from whose perspective? Similarly, what are the indicators that prevailing evidence suggests constitute quality in mentoring relationships? 2. What measures assess effective mentoring relationships in STEMM fields that allow for multiple mentoring relationships at one time? 3. What outcome measures are useful in assessing the most successful characteristics of mentoring relationships and programs? Measures must be theoretically grounded, psychometrically sound, and reliable across demographic groups. This includes careful consideration of factors such as selection bias. Ideally, measures also provide information that can be used by mentors and mentees to adapt their behaviors to maximize positive outcomes, and by programs and institutions to help them improve their mentorship activities. This chapter discusses the work that has been done on developing and using such measures, some of the challenges in doing so, and potential areas of research to better assess the effectiveness of mentorship educa- tion and initiatives. It focuses on identifying validated quantitative measures for use in assessment efforts for mentorship improvement and summarizing qualitative work on outcomes, antecedents, and correlates of mentorship.3 THEORETICAL APPROACH Existing research on mentorship tends to examine the relationships between mentor- ing functions, intervening processes, and individual-level outcomes, such as satisfaction, career progression, STEMM persistence, and retention. Still, there is opportunity for future work to augment our understanding of the intervening psychological, cognitive, 3    e measures highlighted in this chapter have been studied and are supported by some validity and Th reliability evidence. They have also been used in practice. However, it was beyond the scope of this report to determine how widely these instruments are used. Table 6-1 provides a list of measures that could be con- sidered over measures that lack such reliability and validity evidence, along with context for those measures. In addition, Chapter 1 discussed what qualifies as evidence and reminded the reader that the committee endorses using both qualitative and quantitative methods. PREPUBLICATION COPY—Uncorrected Proofs

130 Th e S c i e n c e o f E f f e c t i v e Me n to r sh i p i n ST E M M affective, and behavioral processes that link the quality of the mentorship a mentee receives and outcomes in STEMM contexts.4 Assessing mentorship’s relational processes involves moving beyond cross-sectional studies and interpersonal analysis and including intrapersonal research methods such as experience sampling assessment or ecological momentary assessment designs (­Shiffman et al., 2008).5 For example, experience sampling assessments can involve, but are not limited to, the use of cell phone and computer-based applications.6 Using an app, indi- viduals could be prompted to record mentorship behaviors they experienced that day. Such methods allow for analysis of daily variations in mentoring functions as predictors of relationship development over time or relate mentor or mentee behaviors over time to other factors, such as institutional support, and outcomes. Effective tracking allows users and researchers to examine which factors are related to and predictive of happiness. Such approaches will facilitate the study of how relational experiences over time culminate to predict outcomes, which could provide important insights for understanding both immediate and cumulative effects of mentorship. To highlight the available valid measures and the strength of evidence supporting them, the committee drew on a process-based model of mentorship that suggests key individual and relational characteristics and processes for mentees (Figure 6-1) (Eby et al., 2013). This model focuses on the individual level in the ecological systems model discussed in Chapter 2. As noted throughout this report, contexts are important for mentorship. However, the committee failed to find any valid measures for assessing a culture of mentorship for STEMM undergraduate students and graduate students at the level of the department, college, or institution, or professional associations or societies.7 4    rganizational O scholarship has relevant information that may be useful to consider factors, such as resilience, that mediate mentorship (Kao et al., 2014). 5    xperience sampling asks individuals to “provide systematic self-reports at random occasions during the E waking life of a normal week. Sets of these self-reports from a sample of individuals create an archival file of daily experience” (Larson and Csikszentmihalyi, 2014). Ecological momentary assessment (EMA) “involves repeated sampling of subjects’ current behaviors and experiences in real time, in subjects’ natural environments. EMA aims to minimize recall bias, maximize ecological validity, and allow study of microprocesses that influence behavior in real-world contexts. EMA studies assess particular events in subjects’ lives or assess subjects at periodic intervals, often by random time sampling, using technologies ranging from written diaries and telephones to electronic diaries and physiological sensors” (Shiffman et al., 2008b). 6    ne example is the Track Your Happiness app. More information is available at https://www.­ O trackyourhappiness.org/; accessed April 24, 2019. 7   n organizational behavior, culture and climate assessments are oftentimes aggregates of individual I assess­ ents: if a large number of people feel their organization is safety focused, a strong safety culture m exists. However, it is unclear that a full parallel can be made to the mentorship culture for STEMM under- graduate and graduate students. PREPUBLICATION COPY—Uncorrected Proofs

A s s e s s m e n t a n d E va l u a t i o n 131 MEASURES OF MENTORING RELATIONSHIP PROCESSES IN STEMM CONTEXTS Efforts to assess mentorship at any level are ideally a part of a larger evaluation effort of expected outcomes. Scholars have provided guidance for evaluating formal mentor programs (Lunsford, 2016), and such assessments may be formative—used to change mentorship behaviors or practice and to inform decision-making about programs—or summative—used to demonstrate the effectiveness and significance of practices, behav- iors, or programs. As noted in Chapter 4, mentorship occurs formally and informally, but in all cases it is expected to result in an improved outcome for participants. Meta-analyses from a mentee perspective8 indicate four categories of outcomes for mentorship: attitudinal, behavior, career, and health-related (Eby et al., 2013). Attitudinal outcomes change the fastest and include attitudes such as sense of belonging in and satisfaction with an aca- demic major, department, discipline, or program. Behavioral outcomes refer to behaviors such as remaining in a major or a graduate program. Career outcomes refer to career prospects, such as gaining admission to graduate school or to a job. Health-related out- comes refer to strain or stress and self-efficacy, which are related to psychological health. Process-Oriented Model In the ideal case, measurement models would map onto theoretical models to test research questions and hypotheses of mentorship processes and outcomes. In the com- mittee’s review of assessment methods, recent theoretical and empirical evidence sup- ports a process-oriented model of mentorship (Figure 6-1) (Eby et al., 2013) that can be mapped onto assessment methodologies. This model holds that personal, contextual, and relational inputs shape the characteristics of the mentoring relationship processes, and these relationship processes influence cognitive, emotional, and behavioral out- puts. Outputs from mentorship in STEMM contexts vary widely across the literature, with examples including psychological processes such as self-efficacy, learning or skill, scholarly achievement, and enhanced career aspirations and advancement including persistence in STEMM pathways (Crisp and Cruz, 2009; Eby et al., 2013; Gershenfeld, 2014; Ghosh, 2014; Ghosh and Reio, 2013; Jacobi, 1991; Pfund et al., 2016; Sadler et al., 2010; Syed et al., 2011). According to this process model, mentorship includes active functions such as career support or instrumental support (i.e., sponsorship, coaching, exposure and vis- ibility, protection, and challenging work assignments), and psychosocial support (i.e., acceptance, counseling, and friendship) that were discussed in Chapter 2. Additional 8   A meta-analysis involves quantitatively combining and analyzing data from multiple studies to deter- mine aggregate effect sizes for relationships between variables across multiple quantitative studies. PREPUBLICATION COPY—Uncorrected Proofs

132 Th e S c i e n c e o f E f f e c t i v e Me n to r sh i p i n ST E M M roles include passive functions, such as role modeling,9 in which a mentor serves as an inspirational example of the norms, attitudes, and behaviors necessary to achieve success (Lockwood and Kunda, 1997). Mentorship also includes negative experiences, including mismatch within the mentorship dyad, distancing behavior, manipulative behavior, lack of expertise, and general dysfunctionality, as discussed in Chapter 5 (Eby et al., 2004; Eby, Durley et al., 2008; Kram, 1985b). Benevolent mentorship support f ­unctions—career support, psychosocial support, and role modeling, for example—­ promote relationship quality, which includes overall relationship satisfaction, trust, reciprocity, and effectiveness (Kram, 1985b), whereas negative experiences diminish relationship quality. Relationship quality, in turn, reciprocally influences future levels of provided and received mentor support functions (Eby et al., 2013). For research men- toring in STEMM, performance outputs encompass an array of research skills, as well as critical research products such as publications. A landscape review conducted for this report identified 35 assessments of mentor- ing relationship processes in postsecondary educational STEMM contexts from the perspectives of mentees, mentors, or programs/institutions, many of which contain components that map onto process models (Hernandez, 2018). Most of these assess- ments have focused on measuring characteristics of the mentoring relationship from the mentee’s perspective, and the majority of those assessments focused on under­ graduate and graduate students, with fewer looking at postdoctoral researchers. Of the few assessments focused on the mentor’s perspective of the mentoring relationship, most examined university faculty, graduate student, and postdoctoral researcher perceptions of the mentoring relationship they had with undergraduate mentees (Hernandez, 2018). Assessments of mentoring relationships from the program or institutional perspective drew on the perceptions of institutional staff members who run mentorship programs or faculty mentors involved in those programs. The quantity and quality of validity evidence varies substantially across mentee, men- tor, program, and institutional evaluation perspectives and within specific assessments from each perspective. Figure 6-2 summarizes the strength of the validity evidence based on assessment content, internal structure, and relationships among processes within the process-oriented model of mentorship (Eby et al., 2013). Table 6-1 lists the instruments that have moderate levels of validity evidence supporting their use (Hernandez, 2018). Assessments from the mentee perspective examined types of career and psychosocial support mentees received as well as overall mentor relationship quality. Items in these assessments ranged from general support functions that apply across contexts, such as goal setting, to support functions that are specific to STEMM contexts, such as research collaboration. Assessments from the mentor perspective examine a variety of behaviors categorized as provision of career support and psychosocial support. Assessments at the 9    ole modeling, as a support function of mentorship, is sometimes broken out and sometimes subsumed R in the psychosocial support functions (Crisp and Cruz, 2009). PREPUBLICATION COPY—Uncorrected Proofs

PREPUBLICATION COPY—Uncorrected Proofs Figure 6-2. Synthesis of mentoring relationship processes validation evidence in postsecondary STEMM contexts FIGURE 6-2  Synthesis of mentoring relationship processes validation evidence in postsecondary STEMM contexts. NOTES: Solid lines separate institutional perspective from mentor/mentee perspective, as there is little to no evidence connecting these perspectives. For simplicity, double-headed arrows were omitted where no evidence of a correlation exists in STEMM contexts. The term “Instrumental Support” is used instead of career support. SOURCE: Hernandez, 2018. 133

134 Th e S c i e n c e o f E f f e c t i v e Me n to r sh i p i n ST E M M TABLE 6-1  Assessments by Career Stage with Moderate Levels of Validity Evidence Scale Name Subscales For Career Stage Discipline [No. of Items] (Author, Year) Mentorship Functions Scale Career Support Mentees Doctoral “Hard” [MFS, 29] Psychosocial Support Sciences (Noe, 1988) (from use in original text) Mentor Role Instrument Career Support Mentees Graduate Academic [MRI, 33] Psychosocial Support Medical Center, (Ragins and McFarlin, 1990) clinical and translational science trainees Global Measure of Mentorship One factor encompassing career Undergraduate STEMM Practices and psychosocial support and Graduate [GMMP, 18] networking (Dreher and Ash, 1990) Mentor Satisfaction scale Satisfaction [3] (Ensher and Murphy, 1997) Need Satisfaction Scale Three factors: Mentees UR Medical Center [9] Autonomy Undergraduate, (La Guardia et al., 2000) Competence Postdoc, Relatedness Faculty Survey on Doctoral Education Six factors: Mentees Doctoral STEMM – Mentorship Subscale Affective [23] Instrumental (Golde and Dore, 2001; Noy Intellectual and Ray, 2012) Exploitive Available Respectful Working Alliance in Advisor- Three factors: Mentees Doctoral STEMM Advisee Relationships Rapport Undergraduate [AWAI, 29] Apprenticeship in summer (Schlosser and Gelso, 2001) Identification-Individuation research Mentorship Effectiveness N/A Mentees Undergraduate N/A Scale in summer [12] research (Berk et al., 2005) College Student Mentorship Two dimensions of psychosocial Mentees Undergraduate N/A Scale support: [CSMS, 25] Psychological and emotional (Crisp and Cruz, 2009); Crisp Role model and Cruz, 2010) Two dimensions of career support: Goal setting and career paths Academic subject knowledge PREPUBLICATION COPY—Uncorrected Proofs

A s s e s s m e n t a n d E va l u a t i o n 135 TABLE 6-1  Continued Scale Name Subscales For Career Stage Discipline [No. of Items] (Author, Year) Role Model Identification Role model Mentees Undergraduate STEM [4] in summer (Hoyt et al., 2012) research Mentoring Competency Maintaining effective Mentees Undergraduate STEMM Assessment communication [MCA, 26] Aligning expectations (Fleming et al., 2013; Pfund et Assessing understanding al., 2013; Pfund et al., 2014) Addressing diversity Fostering independence Promoting professional development Mentor Effectiveness Scale Effectiveness Mentees Undergraduate N/A [26] in summer (Byars-Winston et al., 2015) research Mentorship Structure, Mentor network structure Mentees Master’s N/A Motivation, and Effectiveness Motivations to be mentor in clinical [32] characteristics research (McGinn et al., 2015) Effectiveness Mentorship Experience in Challenge Mentees Undergraduate STEMM College Authenticity [24] Commitment (Gullan et al., 2016) Community Mentorship Strategies and Instrumental support Mentees Undergraduate Science Approaches Socioemotional support [14] Culturally responsive support (Haeger and Fresquez, 2016) Deaf Mentorship Survey Being a scientist Mentees Undergraduate Scientific [DMS, 15] Deaf community capital disciplines (Braun et al., 2017) Asking for accommodations Communication access Evaluation of Mentoring Global measure of similarity, Mentees Undergraduate Engineering Relationship support, and satisfaction [9] (Dennehy and Dasgupta, 2017) Mentoring Competency Six factors: Mentors Undergraduate STEMM Assessment Maintaining effective faculty [MCA, 26] communication (Fleming et al., 2013; Pfund et Aligning expectations al., 2013; Pfund et al., 2014) Assessing understanding Addressing diversity Fostering independence Promoting professional development SOURCE: Hernandez, 2018. PREPUBLICATION COPY—Uncorrected Proofs

136 Th e S c i e n c e o f E f f e c t i v e Me n to r sh i p i n ST E M M program or institution level included items that ranged from general support functions to items that are specific to STEMM contexts, such as fostering research independence. USING EXISTING MEASURES AND TAILORING ASSESSMENTS TO STEMM CONTEXTS There are several pathways for developing and selecting measures to evaluate men- toring relationships. First, a large body of research on mentorship measures in the organi- zational behavior literature delineates and differentiates between psychosocial and career support mentorship functions and sometimes role modeling functions. These measures can be adapted through minimal wording changes to STEMM contexts—by changing contextual components of items from “workplace” to “university” or “research group,” for example—and some of them have been used in assessments of academic mentorship (Eby, Allen et al., 2008; Pfund et al., 2016). Second, significant development and valida- tion work on STEMM-specific measures can supplement broad mentorship measures with STEMM context-specific behaviors, competencies, and outcomes. Two examples illustrate the benefits of adapting assessments or developing them for postsecondary STEMM contexts. The Global Measure of Mentorship Practices (GMMP) (Dreher and Ash, 1990) was developed as a comprehensive assessment of mentorship support received, and it was adapted for use in postsecondary STEMM contexts by omit- ting two questions that were irrelevant to graduate students and adding four additional questions that related to disseminating research and exploring career options (Tenen- baum et al., 2001). The resulting adapted GMMP instrument measures 10 behaviors of career and psychosocial support that are generally specified to mentee experiences in postsecondary STEMM (see Box 6-2). The adaptation of the GMMP was efficient and relatively low in cost, but without a more complete attempt to establish validity with the population of interest, it is possible that the modified instrument misses important career support behaviors unique to STEMM. In contrast, the Mentoring Competency Assessment (MCA) (Fleming et al., 2013) is an example of an instrument developed specifically for postsecondary STEMM research contexts.10 The content validation process for this measure involved (1) an extensive review of the mentorship assessments, (2) cognitive interviews with mentors and mentees in postsecondary STEMM research contexts, and (3) aligning assessment content to a framework and learning objectives for an Entering Mentoring–based mentor education program (Fleming et al., 2013; Handelsman et al., 2005; Pfund et al., 2013; Pfund et al., 2006). The resulting 26-item MCA measures six mentor competencies that are specific to postsecondary STEMM research contexts, with one version for mentors and one for mentees. The MCA includes sets of items, or subscales, that could be useful 10    xamples, designed for self-reflection, are available at https://ictr.wisc.edu/mentoring/mentor-­ E evaluation-form-examples/; accessed May 23, 2019. PREPUBLICATION COPY—Uncorrected Proofs

A s s e s s m e n t a n d E va l u a t i o n 137 BOX 6-2 The Global Measure of Mentorship Practices Adapted for Use in Postsecondary STEMM Contexts The Global Measure of Mentorship Practices (GMMP) has been adapted to postsecondary STEMM contexts by removing two items and adding four other, context-specific ones. Each item is prefaced with the phrase “to what extent has a mentor...”. The 15 items retained from the original GMMP are as follows: • Gone out of his/her way to promote your academic interests? • Conveyed feelings of respect for you as an individual? • Conveyed empathy for the concerns and feelings you have discussed with him/her? • Encouraged you to talk openly about anxiety and fears that detract from your work? • Shared personal experiences as an alternative perspective to your problems? • Discussed your questions or concerns regarding feelings of competence, commitment to advancement, relationships with peers and supervisors, or work/family conflicts? • Shared history of his/her career with you? • Encouraged you to prepare for the next steps? • Served as a role model? • Displayed attitudes and values similar to your own? • Helped you finish assignments/tasks or meet deadlines that otherwise would have been dif- ficult to complete? • Protected you from working with other faculty, lecturers, or staff before you knew about their likes/dislikes, opinions on controversial topics, and the nature of the political environment? • Given you challenging assignments that present opportunities to learn new skills? • Helped you meet other people in your field at the University? • Helped you meet other people in your field elsewhere? The two omitted items are as follows: • Given or recommended you for assignments that increased your contact with higher level managers? • Kept you informed about what is going on at higher levels in the company or how external conditions are influencing the company? The four additional items are as follows: • Given you authorship on publications? • Helped you improve your writing skills? • Helped you with a presentation (either within your department, or at a conference)? • Explored career options with you? SOURCE: Tenenbaum et al., 2001. PREPUBLICATION COPY—Uncorrected Proofs

138 Th e S c i e n c e o f E f f e c t i v e Me n to r sh i p i n ST E M M for measuring elements of mentorship outside of STEMM or research contexts, such as active listening. Other subscales are specific to STEMM research, such as accurately estimating a mentees’ ability to conduct research. The decision to adapt or develop an assessment—and in particular, the content of an assessment—for postsecondary STEMM is not trivial, particularly given limited empirical evidence supporting the assertion that context-specific measures necessarily result in enhanced predictive and construct validity (AERA, 2014). GAPS IN STEMM MENTORSHIP ASSESSMENT Similar to the broader literature of the science of mentorship in postsecondary settings (Crisp and Cruz, 2009; Jacobi, 1991), a review of the mentorship assessment literature reveals there is little consensus on how to determine either the most essential specific forms of mentorship support or the programmatic or institutional structures that could enhance, incentivize, or reward mentorship support. This ambiguity is often related to a lack of valid measures at various levels or from various perspectives. Program- and institution-level evaluations have attempted to evaluate mentorship support in a variety of ways, ranging from perceived costs and benefits to opportunities for professional development. However, to date there is a lack of theoretical or empirical work linking the content or aspects of institutional support structures for mentorship to dyadic mentorship processes, such as the perceptions of mentorship provided by a mentor to a mentee. As a result, the current assessments of mentorship from program and institutional perspectives do not align well with theoretical models of mentoring relationship processes such as career support, psychosocial support, role modeling, and negative experiences. There are several measures of relationship quality in STEMM contexts from the m ­ entee perspective (Byars-Winston et al., 2015; Dennehy and Dasgupta, 2017; Ensher et al., 2001; Hernandez et al., 2016), but a dearth of measures of relationship quality from mentors’ perspective. For example, negative mentoring experiences have been documented,11 and there are robust assessments of negative mentorship experiences outside of STEMM contexts (Eby et al., 2004; Eby, Durley et al., 2008). These could be adapted and leveraged for use in STEMM contexts for both mentee and mentors. In addi- tion, numerous measures are available for documenting mentee outcomes of mentoring relationships (Hernandez, 2018), but measures of mentor outcomes are scarce. Finally, there is a shortage of assessments for STEMM mentorship at the department, college, university, and professional association level. Development of these assessments could contribute to an enhanced understanding of contextual factors conducive or prohibitive to mentorship, such as departmental, institutional, or disciplinary culture. Preliminary evidence for what constitutes a mentorship-supportive culture is avail- 11    egative N mentoring experiences are discussed further in Chapter 5. PREPUBLICATION COPY—Uncorrected Proofs

A s s e s s m e n t a n d E va l u a t i o n 139 able, and it has the potential to inform the development of assessments in this domain (­Zachary, 2011). MENTORSHIP OUTCOMES Support for mentorship within STEMM contexts is more likely if comprehensive evidence shows how and why mentorship and specific mentorship processes are linked to desirable outcomes for mentees, mentors, and the research enterprise. One potential component of a greater assessment of a mentorship practice or program could be an evaluation of programs or campaigns to demonstrate how and why mentorship can ben- efit mentees. Therefore, in addition to gaining an in-depth understanding of mentorship experiences from both the mentors’ and mentees’ perspective, it is important to review different outcomes of mentorship for mentees, mentors, and their broader contexts. This section discusses outcomes of mentorship, with an emphasis on assessment and measurement practices. A major purpose of STEMM mentorship is to improve outcomes for mentees, including improved academic and professional performance, increased persistence in pursuing a degree and career, greater self-efficacy, and a stronger sense of science identity and belonging, among others. Successful mentoring relationships can be measured by mentees’ successes in reaching individual milestones along their educational or career trajectory. In addition, successful mentoring relationships yield mentees with the ability to define their career goals, identify the skills they need to achieve those goals, and take the necessary steps to make progress toward those goals. In that way, a successful mentor will be one with the skills and knowledge to support mentees’ development by helping them gain the competencies, knowledge, and confidence they will need to reach their educational and career goals. Achieving success involves mentors understanding each mentee’s unique needs and desires, as well as being flexible and humble enough to adapt their mentoring behaviors to best meet the mentee’s needs and desires (Pfund, 2016). One example illustrating the link between mentor effectiveness and mentee efficiency in achieving academic milestones comes from Vanderbilt University, which is currently assessing the value and impact of mentorship on almost 1,000 basic biomedical sciences Ph.D. students (see Box 6-3). A substantial body of research compiled over the past 30 years has examined the effect of the mentoring relationships individuals engage in during their careers. This research, conducted across a broad range of professional domains, indicates mentorship has a net positive effect on academic achievement, retention, and degree attainment (Campbell and Campbell, 1997; Crisp and Cruz, 2009; Nagda et al., 1998; Terenzini et al., 1996), as well as career success, career satisfaction, and career commitment (Cox, 1997; Schlosser et al., 2003). PREPUBLICATION COPY—Uncorrected Proofs

140 Th e S c i e n c e o f E f f e c t i v e Me n to r sh i p i n ST E M M BOX 6-3 The Relationship Between Mentoring and Graduate Student Outcomes in Basic Biomedical Sciences at Vanderbilt University The basic biomedical sciences at Vanderbilt University have been collecting and anonymizing information in two areas from graduating Ph.D. students for nearly 20 years: students’ performance, such as time to degree and number of first-author (and other position) papers published, and an assess- ment of students’ performance at the time of graduation by the faculty who have mentored and advised them; and students’ assessment of the mentorship received during their tenure. Mentorship is assessed in 13 categories:   1.   Provide scientific training and advice   2.   Provide constructive feedback on oral and written communication skills   3.   Set reasonable goals and expectations   4.   Communicate reasonable goals and expectations   5.   Set aside time to meet with you   6.   Encourage creativity and independence   7.   Treat you with dignity and respect   8.   Provide opportunities to present data   9.   Help navigate graduate school program requirements 10.   Encourage a healthy work-life balance 11.   Help you complete your thesis project in a reasonable length of time 12.   Support your professional development activities 13.   Support your career goals Recently, an effort is being made to correlate the results of the students’ perception of the mentor- ship and the students’ performance or outcomes. While causality cannot clearly be attributed, there appears to be correlation between mentorship assessment and time to degree (see Table 6-3-1 for results of Ph.D. students between 2007 and 2017), number of papers published within 3 years of graduating (the lowest-ranked quartile of faculty had 11 students who ended up with eight or more publications compared with the highest-ranked quartile, which had 30 students with eight or more publications), and faculty assessment of student performance (the lowest-ranked third of mentors had nearly 6 times as many lower-performing students as the highest-ranked third of mentors). Outcomes of Mentorship in STEMM for Mentees For undergraduates in STEMM, participating in mentored research experiences has been linked to self-reported gains in research skills, productivity, and retention in STEMM (Laursen et al., 2010; Linn et al., 2015; Sadler and McKinney, 2010). Studies have also shown that research experiences combined with quality mentorship that includes providing psychosocial and career support and networking opportunities contributes to students feeling integrated into STEMM fields (Estrada et al., 2018). Effective mentor- ing relationships have been shown to influence undergraduate mentees’ confidence in PREPUBLICATION COPY—Uncorrected Proofs

A s s e s s m e n t a n d E va l u a t i o n 141 TABLE 6-3-1  Student Time to Defense and Rating of Thesis Mentor Students’ Rating of Their Thesis Mentors Top Second Third Bottom 25% 25% 25% 25% Years to Ph.D. thesis defense 5.26 ± 0.98 5.60 ± 0.86* 6.01 ± 1.03** 6.01 ± 1.00*** (Avg. ± Std. Dev.) Number of students who rated 103 213 158 174 their mentors Number of mentors 63 63 63 64 NOTES: Graduating Vanderbilt biomedical sciences Ph.D. students (2007–2017) rated their mentors on a scale of 1 (highest) to 4 (lowest) in 13 categories. Students in this analysis were admitted through the Inter­ isciplinary d G ­ raduate Program (IGP), Quantitative and Chemical Biology (QCB), and Chemical and Physical Biology (CPB) u ­ mbrella entry programs. Based on the average rating from all their students, mentors were grouped into quartiles from top 25 percent to bottom 25 percent, and the average time to defense for their students was analyzed. Students in the top quartile had a significantly shorter time to defense compared with students in the second (*p = 0.02), third (**p < 0.0001), and bottom (***p < 0.0001) quartiles (one-way analysis of variance test followed by Tukey post hoc test). Mentor total n = 253; Student total n = 648; Overall average time to defense = 5.75 years. (Institutional Review Board approval number: 190162) SOURCE: Brown et al., 2019. their research skills, a key predictor of persistence in STEMM (Byars-Winston et al., 2015). One investigator described STEMM environments as ideal for the development of undergraduate mentor-mentee relationships because there is often a focus on work- ing in laboratories (DeAngelo, 2016), which places the faculty member and student in a one-on-one situation conducive to mentorship. Still, students and faculty have to initiate this pairing on their own. As noted in a 2017 National Academies report on undergraduate research experi- ences in STEM (NASEM, 2017b), mentees perceive mentors who model ethical behav- iors, kindness, and competence as exhibiting outstanding mentor qualities (Johnson, PREPUBLICATION COPY—Uncorrected Proofs

142 Th e S c i e n c e o f E f f e c t i v e Me n to r sh i p i n ST E M M 2002; Mullen et al., 2000; Rice and Brown, 1990). In addition, research has shown that perceived mentor effectiveness indirectly predicts enrollment in science-related doctoral or medical degree programs (Byars-Winston et al., 2015). Graduate students who have good mentoring relationships are more likely to per- sist in their academic decisions (McGee and Keller, 2007; Williams et al., 2016), with positive mentorship cited as the most important factor in completing a STEM degree (Ashtiani and Feliciano, 2012; Solorzano and Yosso, 2000). Quality mentorship focusing on graduate students’ psychosocial needs appears to increase how mentees perceive the quality of the mentoring relationship and how satisfied they are with that relationship, which in turn enables them to see themselves as more competent STEMM researchers (Tenenbaum et al., 2001; Waldeck et al., 1997). Mentored graduate students and medical trainees are also more likely to publish their research than those who are not mentored (Steiner et al., 2004; Steiner et al., 2002; Wingard et al., 2004). The association between quality mentoring relationships and achievement among mentees from groups who are underrepresented (UR) in STEMM12 is even stronger NASEM, 2017a).13 Evidence suggests that positive mentor-mentee relationships and qual- ity mentorship are particularly important for integrating women and UR students into the STEMM academic community (Anderson and Kim, 2006; Byars-Winston et al., 2015; Estrada et al., 2018; Felder, 2010; Good et al., 2000; Griffith, 2010; Huang et al., 2000; Lewis et al., 2016; Lisberg and Woods, 2018). Studies have also shown that quality mentorship increases recruitment of UR mentees into graduate school and research-related career paths (Hathaway et al., 2002; Junge et al., 2010; Nagda et al., 1998; Thiry and Laursen, 2011). Outcomes of Mentorship in Higher Education Outside of STEMM for Mentees Researchers have conducted a wide range of qualitative and quantitative studies on mentorship outcomes in higher education outside of STEMM. In qualitative studies, for example, investigators used case study methods and interviews to study recommended characteristics of mentorship, how students and mentors experience the mentoring relationship, and what both students and mentors expect from mentoring relationships and what their roles are in that relationship (Baker and Griffin, 2010; Bell and Treleaven, 2011; Griffin, 2013). For the most part, quantitative research has examined college adjust- ment (Apprey et al., 2014), career and personal development (Haddock et al., 2013; Kinkel, 2011; Sams et al., 2015), and measures of academic progress and success (Fox et al., 2010; Hu and Ma, 2010; Zell, 2011). 12    is report refers to UR groups as including women of all racial/ethnic groups and individuals specifi- Th cally identifying as Black, Latinx, and American Indian/Alaska Native. Where possible, the report specifies if the UR groups to which the text refers to Black, Latinx, or of American Indian/Alaska Native heritage. 13    is topic is explored in more depth in Chapter 3. Th PREPUBLICATION COPY—Uncorrected Proofs

A s s e s s m e n t a n d E va l u a t i o n 143 Most of these studies in higher education outside of STEMM did not distinguish between mentorship and other forms of supportive relationships, including those with advisors, institutional agents, developers, and coaches (Baker and Griffin, 2010; Bettinger ­ and Baker, 2011; Museus and Neville, 2012; Tovar, 2015). Nonetheless, there are lessons from these studies that suggest what outcomes STEMM mentees might experience. This research suggests, for example, that informal mentorships are more likely to be successful for mentees and result in outcomes superior than with formal mentorship, which is when relationships are based on assigning students to mentors (Davidson and Foster-Johnson, 2001; Gandara and Maxwell-Jolly, 1999).14 This research also shows that career and psychosocial support in mentorship often contribute in different ways to different types of outcomes for mentees, and that career support typically results in better career outcomes, such as greater publication output for graduate students (Haeger and Fresquez, 2016; Tenenbaum et al., 2001). Psychosocial support results in outcomes that are crucial for student well-being and other criteria necessary for promotion and productivity, such as greater satisfaction with the mentoring relationship and commit- ment to one’s own academic program (Phinney et al., 2011). Other positive outcomes from mentorship programs include increased academic performance and involvement in programs at the college or university (Brittian et al., 2009; Dahlvig, 2010), better transition and adjustment to the college environment (Smojver Ažić and Antulić, 2013), improved personal and career development (Kinkel, 2011), more degrees conferred and persistence through programs (Gross et al., 2015), and positive civic outcomes such as increased social responsibility and socially responsive leadership (Haddock et al., 2013). Outcomes from a Relationship Perspective Mentoring relationships can be characterized by the purpose, intensity, and duration of the relationship. Successful mentoring relationships result from a mentor’s intentional and purposeful commitment to helping the mentee succeed (Baker and Griffin, 2010). Additionally, mentorship may help develop students’ time management skills, study skills, communication skills, and other transferable skills sets, as well as helping them adjust to college (Michael et al., 2010; Salinitri, 2005). Helping to guide and engage stu- dents in research, providing direction in career goals, and creating a sense of belonging in college departments are strategies that have proved successful in mentorship programs (Crisp et al., 2017). Measuring outcomes from the mentoring relationship perspective highlights the value of having parallel measures from both sides of the relationship: that of the mentor and the mentee. Such parallel measures can elucidate the degree to which mentees and mentors have shared views about the mentoring relationship and mentoring activities, which can be an indicator of their working alliance. One example of parallel mentoring 14    nformal I and formal mentoring relationships are discussed in Chapter 4. PREPUBLICATION COPY—Uncorrected Proofs

144 Th e S c i e n c e o f E f f e c t i v e Me n to r sh i p i n ST E M M relationship measures is from the Howard Hughes Medical Institute Gilliam Fellow- ships for Advanced Study, a predoctoral program for UR students in STEMM. A survey posed questions to Gilliam mentors and mentees in dyadic pairs about behaviors in the mentoring relationships related to facilitating students’ research and career develop- ment and science identity. Results of the survey revealed a mismatch on some aspects in the mentoring relationship. Namely, mentors reported displaying more of the desired behavior, such as, mentors sharing their own research career pathway, highlighting and giving direction for improving mentees’ research outcomes, and affirming mentees’ ability to be a scientist, than their mentees reported perceiving (see Table 6-2) (Pfund, Byars-Winston, Black, 2019). These findings indicate that further inquiry into how dif- ferent views of mentoring activities influence mentorship outcomes could be useful, and they also point to the potential value of mentorship education in supporting mentors’ career facilitation for students. Few studies on mentorship outcomes appear to use theoretical frameworks focused on the relational elements of mentoring, such as social support, that emphasize how rela- tionships reduce stress and promote coping, or developmental support, which links men- torship to the college student developmental process. However, several studies (Aikens et al., 2017; Aikens et al., 2016) have used social capital theory as a framework for exam- TABLE 6-2  Results from a Paired Survey of Mentors-Mentees in the Howard Hughes Medical Institute Gilliam Fellowships for Advanced Study Program Mean Question for the Mentee Question for the Mentor Mentee Mentor p-value My mentor provided opportunities I provided opportunities for my 4.30 4.78 0.959 for me to draw upon my previous mentee to draw upon their previous knowledge to complete a new task. knowledge to complete a new task. My mentor discussed the pathway I discussed with my mentee the 3.68 4.57 0.021 he or she took to enter research. pathway I took to enter research. My mentor appeared aware of the I am aware of the skills and 3.55 4.32 0.530 skills and behaviors that he or she behaviors that I am modeling. was modeling. My mentor told me I have the ability I told my mentee they have the 4.14 4.67 0.040 to be a scientist. ability to be a scientist. My mentor acknowledged my I acknowledged my mentee’s 3.32 4.14 0.713 successes in real time. successes in real time. My mentor highlighted positive I highlighted positive outcomes 3.18 4.52 0.002 outcomes of my research as of my mentee’s research as well well as gave me clear steps for as gave them clear steps for improvement. improvement. NOTE: Bolded items indicate a measureable mismatch between mentor and mentee responses. SOURCE: Pfund, Byars-Winston, Black, 2019. PREPUBLICATION COPY—Uncorrected Proofs

A s s e s s m e n t a n d E va l u a t i o n 145 TABLE 6-3  Parallel Mentor and Mentee Measures Assessing Social Cognitive Career Theory (SCCT) Variables and Cultural Diversity Awareness of Mentors Administered to Measure Mentees Mentors SCCT Variablesa Research Self-Efficacy X X Sources of self-efficacy in mentoring (four subscales) Past performance X X Social persuasion X X Vicarious learning X X Emotional/affective states X X Cultural Diversity Awareness (CDA)b (three subscales) Attitudes toward CDA in mentoring relationships X X Behaviors displaying mentors’ CDA X X Confidence to enact CDA in mentoring relationship X SOURCES: aByars-Winston et al., 2016; bByars-Winston and Butz, in review, 2018. ining the effect of mentorship structures between students, doctoral and postdoctoral scholars, and faculty on various outcomes.15 These investigations found that in “closed mentorship triads,” which included a faculty mentor, a graduate student or postdoctoral mentor, and an undergraduate student mentee,16 interactions were the most beneficial for mentee outcomes such as science identity development (Aikens et al., 2017; Aikens et al., 2016), scholarly productivity, and intentions to pursue a STEM Ph.D. (Aikens et al., 2016). In addition, several researchers have developed parallel measures for mentors and mentees in STEMM based on social cognitive career theory and science identity as well as multicultural theory (Byars-Winston et al., 2016).17 These parallel mentor and mentee measures assess elements in the mentoring relationship related to mentees’ research self-efficacy beliefs and mentors’ cultural diversity awareness (see Table 6-3). Measuring Mentor Motivations and Correlates Assessment and measurement of mentorship could integrate how and why mentors participate in mentorship and what they gain from successful mentorship. For example, one qualitative case study found that graduate students and postdoctoral researchers 15    ocial S capital theory is described further in Chapter 2. 16    riad T configurations of mentorship are discussed in Chapter 4. 17    urther discussion of social cognitive career theory is in Chapter 2. F PREPUBLICATION COPY—Uncorrected Proofs

146 Th e S c i e n c e o f E f f e c t i v e Me n to r sh i p i n ST E M M who mentored undergraduates in research reported improved career preparation and qualifications, cognitive and socioemotional growth, improved teaching and commu- nication skills, greater enjoyment of their own apprenticeship experience, and twice as many benefits as challenges (Dolan and Johnson, 2009). Their motivations for engaging in mentorship were largely about how mentorship would serve as a means to an end, though the benefits and challenges they reported indicated a longer-term vision of how mentorship influenced their personal, cognitive, and professional growth. At the same time, some mentors in this study reported that mentorship of undergraduates made their work lives more enjoyable while generating emotional costs. Several investigators have reported that mentors benefit from a sense of personal fulfillment through knowledge and skill sharing, honing their leadership skills, career preparation, and cognitive growth (Dolan and Johnson, 2009; Eagan et al., 2013; Laursen et al., 2010). Another qualitative study determined that mentors had both career and intrinsic motivations for mentorship in the context of undergraduate research, which appeared to differ by career stage (Hayward et al., 2017). Career motivators for faculty included increased productivity, help in recruiting future students, increased prestige for the university resulting from students presenting at conferences, and helping prepare stu- dents for graduate work and careers. Intrinsic motivators included improved teaching and mentorship skills, feelings of doing something positive, preparing future scientists, and increased energy and enthusiasm in the lab. Faulty mentors of undergraduates were motivated by their belief that mentorship informed their teaching and added fun and enthusiasm to their work, while negative factors included the need for additional time, effort, and funding; increased tension; increased difficulty of gauging students’ research ability; and little recognition or reward (Dolan and Johnson, 2010). Forming mentoring relationships with graduate students helped faculty recruit undergraduates and gain a better sense of postgraduates, but study participants had trouble gauging the effective- ness of mentorship. Research outside of STEMM indicates that mentors’ commitment to the mentoring relationship matters for mentorship outcomes (Allen and Eby, 2008). Given competing role demands on mentors and mentees in STEMM and work settings, mentor commit- ment is not necessarily a given and is often an outcome of many factors (Aryee et al., 1996). In fact, research outside of STEMM indicates that mentors’ identities and their perceptions of the benefits of mentorship toward their own career goal progression play a role, along with factors such as altruism and the presence of effective schemata for developing and sustaining relationships with mentees (Ragins, 2009). Even though effective mentorship has been shown to relate to positive career out- comes for mentors in workplace settings (Ghosh and Reio, 2013; Rogers et al., 2016), the relationships between effective mentorship and career outcomes for mentors in STEMM settings are not always self-evident. Research on work performance (Kerr, 1995; Van Eerde and Thierry, 1996) suggests that individuals have to understand that certain tasks and the quality of task completion will factor into organizational reward systems and ulti- PREPUBLICATION COPY—Uncorrected Proofs

A s s e s s m e n t a n d E va l u a t i o n 147 mately the individual level compensation and rewards they receive. In other words, the value and attention paid to mentorship quality might change if it became a tracked and managed component of universities’ and research organizations’ performance appraisal system for faculty and other researchers who engage in STEMM mentorship (Aryee et al., 1996).18 It is important to note that there may be unintended consequences of efforts to track and manage mentorship, especially if mechanisms are not carefully identified and vetted by professional assessment developers to minimize inequities and bias. NEW AND EMERGING APPROACHES TO ASSESSMENT AND MEASUREMENT OF MENTORSHIP Reciprocal exchanges between mentors and mentees in postsecondary STEMM contexts warrant further study. However, existing research on relationship theory points to the essential nature of reciprocal exchanges between relational partners (Brown, 1991; Fiske, 1992) and can provide insight relevant to STEMM mentoring relationships. Here, the committee explores some recent advances in two methodologies—dyadic data a ­ nalysis and social network analysis of mentorship—and poses further questions for inquiry. Dyadic Data Analysis Relatively recent advances in statistical methodology now allow for characterizing reciprocal relationships through dyadic data analysis (Kenny, 1994; Kenny et al., 2006).19 Dyadic data analysis involves collecting data from both the mentor and the mentee over time to reveal how the perceptions and experiences of each influence the other. For example, a mentor’s perceptions of the mentee has the potential to influence the mentee’s self-perceptions, but this influence can only be examined if data are collected from both the mentor and the mentee over time. This methodology allows researchers to investigate dynamic feedback loops between mentor and mentee, where each informs the other regarding what is or is not needed from the relationship, how the relationship quality and characteristics such as trust development shift over time, and how this influences both mentor and mentee. For instance, this method could reveal how change in trust over time from both the mentor and the mentee perspective influences mentee percep- tions that they are receiving psychosocial support or mentee confidence in their ability to be successful in a STEMM career. One study used a dyadic approach to characterize reciprocal feedback between mentors and mentees in a STEMM research experience context (Griese et al., 2016). 18    ese Th topics are explored more deeply in Chapter 7. 19    yadic D data analysis is a general methodology that captures the reciprocal nature of a relationship and its influence on both members in the relationship (Kenny, 1994; Kenny et al., 2006). PREPUBLICATION COPY—Uncorrected Proofs

148 Th e S c i e n c e o f E f f e c t i v e Me n to r sh i p i n ST E M M Social Network Analysis of Mentorship Advancements in mentorship theory point to the importance of networks of men- toring relationships, particularly for individuals from historically UR groups (Downing et al., 2005; Glessmer et al., 2012; Higgins, 2000; Higgins and Kram, 2001; Higgins and Thomas, 2001; Packard, 2003a; Packard et al., 2004). Recent advancements in measure- ment and statistical methodology now allow researchers to capture and quantify charac- teristics of mentorship-related networks as social networks (Scott, 2017). Social networks can be conceptualized either as “whole networks” or as “ego networks.” Whole networks are systems such as a mentorship group. Whole network analysis could be used to analyze the value of collective or group mentorship, including the value of the network based on the resources offered by its members, such as expertise and information; the diversity of its members; which relationships within the network are most influential; how interconnected members must be for the network to be valu- able to its members; where there might be gaps in the network; and which members of the network serve as hubs for information or resources such as high-quality feedback. Several researchers have begun to measure and categorize beneficial triadic mentor network structures as the simplest form of a whole network (Aikens et al., 2017; Aikens et al., 2016; Morales et al., 2018) and to identify and characterize successful mentorship communities (Chariker et al., 2017), but much more can be done to determine how mentorship networks operate and their distinctive impact and value.20 Ego networks are the connections, or lack thereof, of a single individual and the resources available, or not, to the individual through their connections. Ego network analysis could be used to examine the mentorship resources available to a given ­ entee m and how these resources relate to their personal characteristics and outcomes. For example, mentees with different racial, ethnic, and gender identities can differ in their mentorship networks in ways that may or may not influence their outcomes (Aikens et al., 2017). Longitudinal ego network research is appropriate to determine whether ­mentees with different personal characteristics are more or less likely to develop mentor- ship networks that meet their needs. For instance, mentees who identify strongly with their mentor may perceive that they are receiving both career and psychosocial support and thus may require a simpler dyadic mentorship structure to meet their needs. Mentees who do not identify strongly with their mentor either personally or professionally may benefit from a more elaborate network of mentors, including others who share their identities or ­ articular career interests. These questions could be addressed through p systematic analysis of the ego networks of mentees related to their personal character- istics and outcomes. 20    nsights I from different forms of mentorship can be found in Chapter 4. PREPUBLICATION COPY—Uncorrected Proofs

A s s e s s m e n t a n d E va l u a t i o n 149 Further Questions for Inquiry Understanding the mechanisms by which mentorship is initiated, developed, and sustained, and if they are effectual, is important for theory building and for practical purposes. For example, if research can identify specific, favorable mentee and men- tor behaviors, it could be possible to enhance and encourage those behaviors through programming and evaluation systems, and thus improve mentoring relationships and resulting outcomes. Ideally, assessments would identify important milestones in developing mentoring relationships. In addition, assessments could provide details on whether relationships develop in a linear manner or if there are discontinuous changes or time-bounded needs of mentees, mentors, or mentoring relationships that must be taken into account to develop an effective mentoring relationship and fully realize its benefits to mentees, mentors, and the STEMM enterprise. For more quantitative data, statistical techniques could be used to identify unobservable subgroups based on measured variables or trends in larger data sets, such as probability-based latent class analysis (Bauer and Shanahan, 2007; Oberski, 2016; Pastor et al., 2007; Wachsmuth et al., 2017). More work is needed to minimize selection bias in assessing mentoring outcomes, for example, matched control groups or propensity score matching. Most mentorship theories suggest that mentoring relationships change over time,21 and most correlational research assumes that change is linear—that as trust increases, for example, so does relationship quality. However, experience implies that relationships can shift suddenly, such as when one act of betrayal irreversibly destroys a relationship or when one act of kindness transforms a struggling relationship. Research on turning points in close relationships suggests using both quantitative and qualitative methods to develop robust, explanatory theory. Research could potentially determine if there are predictable patterns of discontinuous change, identify experiences that fundamentally alter mentoring relationships, and learn if positive turning points can repair a previously damaged mentoring relationship (NRC, 2002; Warfa, 2016). There has been little research on multilevel influences arising from mentoring rela- tionships being nested within workgroups, academic departments, research laboratories, organizations including colleges and universities, and industries or academic disciplines. Research is also lacking on aggregate effects that go beyond the individual, such as work- group- or department-level effects. Multilevel modeling can help examine individual, dyadic, group, and organizational effects on the mentoring relationship. 21    uch S as the theories discussed in Chapter 2. PREPUBLICATION COPY—Uncorrected Proofs

PREPUBLICATION COPY—Uncorrected Proofs

Next: 7 Individuals, Relationships, and Institutional Responsibility: How Can Institutional Culture Better Support Mentorship? »
The Science of Effective Mentorship in STEMM Get This Book
×
Buy Prepub | $84.00 Buy Paperback | $75.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Mentorship is a catalyst capable of unleashing one’s potential for discovery, curiosity, and participation in STEMM and subsequently improving the training environment in which that STEMM potential is fostered. Mentoring relationships provide developmental spaces in which students’ STEMM skills are honed and pathways into STEMM fields can be discovered. Because mentorship can be so influential in shaping the future STEMM workforce, its occurrence should not be left to chance or idiosyncratic implementation. There is a gap between what we know about effective mentoring and how it is practiced in higher education.

The Science of Effective Mentorship in STEMM studies mentoring programs and practices at the undergraduate and graduate levels. It explores the importance of mentorship, the science of mentoring relationships, mentorship of underrepresented students in STEMM, mentorship structures and behaviors, and institutional cultures that support mentorship. This report and its complementary interactive guide present insights on effective programs and practices that can be adopted and adapted by institutions, departments, and individual faculty members.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!