6

Assessment and Evaluation:
What Can Be Measured in Mentorship, and How?

Assessments and evaluations enable institutions and individuals to determine if mentorship is achieving the desired goals and outcomes. However, there is a folly in hoping for a specific outcome if measures used to evaluate what is happening focus on something else entirely (Kerr, 1995). To fully understand mentorship, evaluation measures would ideally address both mentorship processes and mentorship outcomes and the system factors that can profoundly shape it.¹ Measurement and evaluation play a critical role in assessing interventions, determining organizational priorities, and developing and testing theory—three key elements that underlie understanding the effectiveness of mentorship. In addition, initiatives and their outcomes that are assessed consistently are better positioned to provide insights for improvement and long-term outcomes. Therefore, intentionality is needed when selecting measures to assess mentorship and outcomes of mentorship.²

This chapter draws on theories and frameworks from Chapter 2 to highlight how to evaluate mentorship in its various forms and contexts. Box 6-1 highlights how theory may inform the concepts that are discussed, like the process-oriented model shown in Figure 6-1. Evaluating the effectiveness of mentorship depends on both quantitative and qualitative measures and tools. Ideally, such measures identify how mentorship or specific mentorship factors contribute to desired outcomes and provide specific insights into how interventions work. Integrating theories is important because they make explicit the

___________________

¹ As articulated in the discussion of theories applicable to mentorship in Chapter 2.

²Intentionality is defined as a calculated and coordinated method of engagement to effectively meet the needs of a designated person or population within a given context.

mechanisms by which mentorship is expected to operate and therefore the appropriate measures to use to assess mentorship activities and programs. This chapter focuses on quantitative measures, although the importance and value of qualitative assessment is acknowledged.

CONSIDERATIONS IN ASSESSMENT OF MENTORSHIP

As mentioned in Chapter 4, it is challenging to determine how to assess effective mentorship at the program, institution, individual, and relationship levels across science, technology, engineering, mathematics, and medicine (STEMM) disciplines and career stages. In selecting appropriate measures, there are at least three important questions to consider:

1. How can we quantify “quality” mentoring relationships and programs—and at what time and from whose perspective? Similarly, what are the indicators that prevailing evidence suggests constitute quality in mentoring relationships?
2. What measures assess effective mentoring relationships in STEMM fields that allow for multiple mentoring relationships at one time?
3. What outcome measures are useful in assessing the most successful characteristics of mentoring relationships and programs?

Measures must be theoretically grounded, psychometrically sound, and reliable across demographic groups. This includes careful consideration of factors such as selection bias. Ideally, measures also provide information that can be used by mentors and mentees to adapt their behaviors to maximize positive outcomes, and by programs and institutions to help them improve their mentorship activities. This chapter discusses the work that has been done on developing and using such measures, some of the challenges in doing so, and potential areas of research to better assess the effectiveness of mentorship education and initiatives. It focuses on identifying validated quantitative measures for use in assessment efforts for mentorship improvement and summarizing qualitative work on outcomes, antecedents, and correlates of mentorship.³

THEORETICAL APPROACH

Existing research on mentorship tends to examine the relationships between mentoring functions, intervening processes, and individual-level outcomes, such as satisfaction, career progression, STEMM persistence, and retention. Still, there is opportunity for future work to augment our understanding of the intervening psychological, cognitive,

___________________

³ The measures highlighted in this chapter have been studied and are supported by some validity and reliability evidence. They have also been used in practice. However, it was beyond the scope of this report to determine how widely these instruments are used. Table 6-1 provides a list of measures that could be considered over measures that lack such reliability and validity evidence, along with context for those measures. In addition, Chapter 1 discussed what qualifies as evidence and reminded the reader that the committee endorses using both qualitative and quantitative methods.

affective, and behavioral processes that link the quality of the mentorship a mentee receives and outcomes in STEMM contexts.⁴

Assessing mentorship’s relational processes involves moving beyond cross-sectional studies and interpersonal analysis and including intrapersonal research methods such as experience sampling assessment or ecological momentary assessment designs (Shiffman et al., 2008).⁵ For example, experience sampling assessments can involve, but are not limited to, the use of cell phone and computer-based applications.⁶ Using an app, individuals could be prompted to record mentorship behaviors they experienced that day. Such methods allow for analysis of daily variations in mentoring functions as predictors of relationship development over time or relate mentor or mentee behaviors over time to other factors, such as institutional support, and outcomes. Effective tracking allows users and researchers to examine which factors are related to and predictive of happiness. Such approaches will facilitate the study of how relational experiences over time culminate to predict outcomes, which could provide important insights for understanding both immediate and cumulative effects of mentorship.

To highlight the available valid measures and the strength of evidence supporting them, the committee drew on a process-based model of mentorship that suggests key individual and relational characteristics and processes for mentees (Figure 6-1) (Eby et al., 2013). This model focuses on the individual level in the ecological systems model discussed in Chapter 2. As noted throughout this report, contexts are important for mentorship. However, the committee failed to find any valid measures for assessing a culture of mentorship for STEMM undergraduate students and graduate students at the level of the department, college, or institution, or professional associations or societies.⁷

___________________

⁴ Organizational scholarship has relevant information that may be useful to consider factors, such as resilience, that mediate mentorship (Kao et al., 2014).

⁵Experience sampling asks individuals to “provide systematic self-reports at random occasions during the waking life of a normal week. Sets of these self-reports from a sample of individuals create an archival file of daily experience” (Larson and Csikszentmihalyi, 2014). Ecological momentary assessment (EMA) “involves repeated sampling of subjects’ current behaviors and experiences in real time, in subjects’ natural environments. EMA aims to minimize recall bias, maximize ecological validity, and allow study of micro-processes that influence behavior in real-world contexts. EMA studies assess particular events in subjects’ lives or assess subjects at periodic intervals, often by random time sampling, using technologies ranging from written diaries and telephones to electronic diaries and physiological sensors” (Shiffman et al., 2008).

⁶ One example is the Track Your Happiness app. More information is available at https://www.trackyourhappiness.org/; accessed April 24, 2019.

⁷ In organizational behavior, culture and climate assessments are oftentimes aggregates of individual assessments: if a large number of people feel their organization is safety focused, a strong safety culture exists. However, it is unclear that a full parallel can be made to the mentorship culture for STEMM undergraduate and graduate students.

MEASURES OF MENTORING RELATIONSHIP PROCESSES IN STEMM CONTEXTS

Efforts to assess mentorship at any level are ideally a part of a larger evaluation effort of expected outcomes. Scholars have provided guidance for evaluating formal mentor programs (Lunsford, 2016), and such assessments may be formative—used to change mentorship behaviors or practice and to inform decision making about programs—or summative—used to demonstrate the effectiveness and significance of practices, behaviors, or programs.

As noted in Chapter 4, mentorship occurs formally and informally, but in all cases it is expected to result in an improved outcome for participants. Meta-analyses from a mentee perspective⁸ indicate four categories of outcomes for mentorship: attitudinal, behavior, career, and health related (Eby et al., 2013). Attitudinal outcomes change the fastest and include attitudes such as sense of belonging in and satisfaction with an academic major, department, discipline, or program. Behavioral outcomes refer to behaviors such as remaining in a major or a graduate program. Career outcomes refer to career prospects, such as gaining admission to graduate school or to a job. Health-related outcomes refer to strain or stress and self-efficacy, which are related to psychological health.

Process-Oriented Model

In the ideal case, measurement models would map onto theoretical models to test research questions and hypotheses of mentorship processes and outcomes. In the committee’s review of assessment methods, recent theoretical and empirical evidence supports a process-oriented model of mentorship (Figure 6-1) (Eby et al., 2013) that can be mapped onto assessment methodologies. This model holds that personal, contextual, and relational inputs shape the characteristics of the mentoring relationship processes, and these relationship processes influence cognitive, emotional, and behavioral outputs. Outputs from mentorship in STEMM contexts vary widely across the literature, with examples including psychological processes such as self-efficacy, learning or skill, scholarly achievement, and enhanced career aspirations and advancement including persistence in STEMM pathways (Crisp and Cruz, 2009; Eby et al., 2013; Gershenfeld, 2014; Ghosh, 2014; Ghosh and Reio, 2013; Jacobi, 1991; Pfund et al., 2016; Sadler et al., 2010; Syed et al., 2011).

According to this process model, mentorship includes active functions such as career support or instrumental support (i.e., sponsorship, coaching, exposure and visibility, protection, and challenging work assignments) and psychosocial support (i.e., acceptance, counseling, and friendship) that were discussed in Chapter 2. Additional roles include

___________________

⁸ A meta-analysis involves quantitatively combining and analyzing data from multiple studies to determine aggregate effect sizes for relationships between variables across multiple quantitative studies.

passive functions, such as role modeling,⁹ in which a mentor serves as an inspirational example of the norms, attitudes, and behaviors necessary to achieve success (Lockwood and Kunda, 1997). Mentorship also includes negative experiences, including mismatch within the mentorship dyad, distancing behavior, manipulative behavior, lack of expertise, and general dysfunctionality, as discussed in Chapter 5 (Eby et al., 2004; Eby, Durley et al., 2008; Kram, 1985b). Benevolent mentorship support functions—career support, psychosocial support, and role modeling, for example—promote relationship quality, which includes overall relationship satisfaction, trust, reciprocity, and effectiveness (Kram, 1985b), whereas negative experiences diminish relationship quality. Relationship quality, in turn, reciprocally influences future levels of provided and received mentor support functions (Eby et al., 2013). For research mentoring in STEMM, performance outputs encompass an array of research skills, as well as critical research products such as publications.

A landscape review conducted for this report identified 35 assessments of mentoring relationship processes in postsecondary educational STEMM contexts from the perspectives of mentees, mentors, or programs/institutions, many of which contain components that map onto process models (Hernandez, 2018). Most of these assessments have focused on measuring characteristics of the mentoring relationship from the mentee’s perspective, and the majority of those assessments focused on undergraduate and graduate students, with fewer looking at postdoctoral researchers. Of the few assessments focused on the mentor’s perspective of the mentoring relationship, most examined university faculty, graduate student, and postdoctoral researcher perceptions of the mentoring relationship they had with undergraduate mentees (Hernandez, 2018). Assessments of mentoring relationships from the program or institutional perspective drew on the perceptions of institutional staff members who run mentorship programs or faculty mentors involved in those programs.

The quantity and quality of validity evidence varies substantially across mentee, mentor, program, and institutional evaluation perspectives and within specific assessments from each perspective. Figure 6-2 summarizes the strength of the validity evidence based on assessment content, internal structure, and relationships among processes within the process-oriented model of mentorship (Eby et al., 2013). Table 6-1 lists the instruments that have moderate levels of validity evidence supporting their use (Hernandez, 2018).

Assessments from the mentee perspective examined types of career and psychosocial support mentees received as well as overall mentor relationship quality. Items in these assessments ranged from general support functions that apply across contexts, such as goal setting, to support functions that are specific to STEMM contexts, such as research collaboration. Assessments from the mentor perspective examine a variety of behaviors categorized as provision of career support and psychosocial support. Assessments at the

___________________

⁹ Role modeling, as a support function of mentorship, is sometimes broken out and sometimes subsumed in the psychosocial support functions (Crisp and Cruz, 2009).

TABLE 6-1 Assessments by Career Stage with Moderate Levels of Validity Evidence

SOURCE: Hernandez, 2018.

program or institution level included items that ranged from general support functions to items that are specific to STEMM contexts, such as fostering research independence.

USING EXISTING MEASURES AND TAILORING ASSESSMENTS TO STEMM CONTEXTS

There are several pathways for developing and selecting measures to evaluate mentoring relationships. First, a large body of research on mentorship measures in the organizational behavior literature delineates and differentiates between psychosocial and career support mentorship functions and sometimes role modeling functions. These measures can be adapted through minimal wording changes to STEMM contexts—by changing contextual components of items from “workplace” to “university” or “research group,” for example—and some of them have been used in assessments of academic mentorship (Eby, Allen et al., 2008; Pfund et al., 2016). Second, significant development and validation work on STEMM-specific measures can supplement broad mentorship measures with STEMM context-specific behaviors, competencies, and outcomes.

Two examples illustrate the benefits of adapting assessments or developing them for postsecondary STEMM contexts. The Global Measure of Mentorship Practices (GMMP) (Dreher and Ash, 1990) was developed as a comprehensive assessment of mentorship support received, and it was adapted for use in postsecondary STEMM contexts by omitting two questions that were irrelevant to graduate students and adding four additional questions that related to disseminating research and exploring career options (Tenenbaum et al., 2001). The resulting adapted GMMP instrument measures 10 behaviors of career and psychosocial support that are generally specified to mentee experiences in postsecondary STEMM (see Box 6-2). The adaptation of the GMMP was efficient and relatively low in cost, but without a more complete attempt to establish validity with the population of interest, it is possible that the modified instrument misses important career support behaviors unique to STEMM.

In contrast, the Mentoring Competency Assessment (MCA) (Fleming et al., 2013) is an example of an instrument developed specifically for postsecondary STEMM research contexts.¹⁰ The content validation process for this measure involved (1) an extensive review of the mentorship assessments, (2) cognitive interviews with mentors and mentees in postsecondary STEMM research contexts, and (3) aligning assessment content to a framework and learning objectives for an Entering Mentoring–based mentor education program (Fleming et al., 2013; Handelsman et al., 2005; Pfund et al., 2006, 2013). The resulting 26-item MCA measures six mentor competencies that are specific to postsecondary STEMM research contexts, with one version for mentors and one for mentees. The MCA includes sets of items, or subscales, that could be useful

___________________

¹⁰ Examples, designed for self-reflection, are available at https://ictr.wisc.edu/mentoring/mentorevaluation-form-examples/; accessed May 23, 2019.

for measuring elements of mentorship outside of STEMM or research contexts, such as active listening. Other subscales are specific to STEMM research, such as accurately estimating a mentees’ ability to conduct research. The decision to adapt or develop an assessment—and in particular, the content of an assessment—for postsecondary STEMM is not trivial, particularly given limited empirical evidence supporting the assertion that context-specific measures necessarily result in enhanced predictive and construct validity (AERA, 2014).

GAPS IN STEMM MENTORSHIP ASSESSMENT

Similar to the broader literature of the science of mentorship in postsecondary settings (Crisp and Cruz, 2009; Jacobi, 1991), a review of the mentorship assessment literature reveals there is little consensus on how to determine either the most essential specific forms of mentorship support or the programmatic or institutional structures that could enhance, incentivize, or reward mentorship support. This ambiguity is often related to a lack of valid measures at various levels or from various perspectives.

Program- and institution-level evaluations have attempted to evaluate mentorship support in a variety of ways, ranging from perceived costs and benefits to opportunities for professional development. However, to date there is a lack of theoretical or empirical work linking the content or aspects of institutional support structures for mentorship to dyadic mentorship processes, such as the perceptions of mentorship provided by a mentor to a mentee. As a result, the current assessments of mentorship from program and institutional perspectives do not align well with theoretical models of mentoring relationship processes such as career support, psychosocial support, role modeling, and negative experiences.

There are several measures of relationship quality in STEMM contexts from the mentee perspective (Byars-Winston et al., 2015; Dennehy and Dasgupta, 2017; Ensher et al., 2001; Hernandez et al., 2016), but a dearth of measures of relationship quality from mentors’ perspective. For example, negative mentoring experiences have been documented,¹¹ and there are robust assessments of negative mentorship experiences outside of STEMM contexts (Eby et al., 2004; Eby, Durley et al., 2008). These could be adapted and leveraged for use in STEMM contexts for both mentees and mentors. In addition, numerous measures are available for documenting mentee outcomes of mentoring relationships (Hernandez, 2018), but measures of mentor outcomes are scarce.

Finally, there is a shortage of assessments for STEMM mentorship at the department, college, university, and professional association levels. Development of these assessments could contribute to an enhanced understanding of contextual factors conducive or prohibitive to mentorship, such as departmental, institutional, or disciplinary culture. Preliminary evidence for what constitutes a mentorship-supportive culture is avail-

___________________

¹¹ Negative mentoring experiences are discussed further in Chapter 5.

able, and it has the potential to inform the development of assessments in this domain (Zachary, 2011).

MENTORSHIP OUTCOMES

Support for mentorship within STEMM contexts is more likely if comprehensive evidence shows how and why mentorship and specific mentorship processes are linked to desirable outcomes for mentees, mentors, and the research enterprise. One potential component of a greater assessment of a mentorship practice or program could be an evaluation of programs or campaigns to demonstrate how and why mentorship can benefit mentees. Therefore, in addition to gaining an in-depth understanding of mentorship experiences from both the mentors’ and mentees’ perspective, it is important to review different outcomes of mentorship for mentees, mentors, and their broader contexts. This section discusses outcomes of mentorship, with an emphasis on assessment and measurement practices.

A major purpose of STEMM mentorship is to improve outcomes for mentees, including improved academic and professional performance, increased persistence in pursuing a degree and career, greater self-efficacy, and a stronger sense of science identity and belonging, among others. Successful mentoring relationships can be measured by mentees’ successes in reaching individual milestones along their educational or career trajectory. In addition, successful mentoring relationships yield mentees with the ability to define their career goals, identify the skills they need to achieve those goals, and take the necessary steps to make progress toward those goals. In that way, a successful mentor will be one with the skills and knowledge to support mentees’ development by helping them gain the competencies, knowledge, and confidence they will need to reach their educational and career goals. Achieving success involves mentors understanding each mentee’s unique needs and desires, as well as being flexible and humble enough to adapt their mentoring behaviors to best meet the mentee’s needs and desires (Pfund, 2016). One example illustrating the link between mentor effectiveness and mentee efficiency in achieving academic milestones comes from Vanderbilt University, which is currently assessing the value and impact of mentorship on almost 1,000 basic biomedical sciences Ph.D. students (see Box 6-3).

A substantial body of research compiled over the past 30 years has examined the effect of the mentoring relationships individuals engage in during their careers. This research, conducted across a broad range of professional domains, indicates mentorship has a net positive effect on academic achievement, retention, and degree attainment (Campbell and Campbell, 1997; Crisp and Cruz, 2009; Nagda et al., 1998; Terenzini et al., 1996), as well as career success, career satisfaction, and career commitment (Cox, 1997; Schlosser et al., 2003).

Outcomes of Mentorship in STEMM for Mentees

For undergraduates in STEMM, participating in mentored research experiences has been linked to self-reported gains in research skills, productivity, and retention in STEMM (Laursen et al., 2010; Linn et al., 2015; Sadler and McKinney, 2010). Studies have also shown that research experiences combined with quality mentorship that includes providing psychosocial and career support and networking opportunities contributes to students feeling integrated into STEMM fields (Estrada et al., 2018). Effective mentoring relationships have been shown to influence undergraduate mentees’ confidence in

their research skills, a key predictor of persistence in STEMM (Byars-Winston et al., 2015). One investigator described STEMM environments as ideal for the development of undergraduate mentor-mentee relationships because there is often a focus on working in laboratories (DeAngelo, 2016), which places the faculty member and student in a one-on-one situation conducive to mentorship. Still, students and faculty have to initiate this pairing on their own.

As noted in a 2017 National Academies report on undergraduate research experiences in STEM (NASEM, 2017b), mentees perceive mentors who model ethical behaviors, kindness, and competence as exhibiting outstanding mentor qualities (Johnson,

2002; Mullen et al., 2000; Rice and Brown, 1990). In addition, research has shown that perceived mentor effectiveness indirectly predicts enrollment in science-related doctoral or medical degree programs (Byars-Winston et al., 2015).

Graduate students who have good mentoring relationships are more likely to persist in their academic decisions (McGee and Keller, 2007; Williams et al., 2016a), with positive mentorship cited as the most important factor in completing a science, technology, engineering, and mathematics (STEM) degree (Ashtiani and Feliciano, 2012; Solorzano and Yosso, 2000). Quality mentorship focusing on graduate students’ psychosocial needs appears to increase how mentees perceive the quality of the mentoring relationship and how satisfied they are with that relationship, which in turn enables them to see themselves as more competent STEMM researchers (Tenenbaum et al., 2001; Waldeck et al., 1997). Mentored graduate students and medical trainees are also more likely to publish their research than those who are not mentored (Steiner et al., 2002, 2004; Wingard et al., 2004).

The association between quality mentoring relationships and achievement among mentees from groups who are underrepresented (UR) in STEMM¹² is even stronger (NASEM, 2017a).¹³ Evidence suggests that positive mentor-mentee relationships and quality mentorship are particularly important for integrating women and UR students into the STEMM academic community (Anderson and Kim, 2006; Byars-Winston et al., 2015; Estrada et al., 2018; Felder, 2010; Good et al., 2000; Griffith, 2010; Huang et al., 2000; Lewis et al., 2016; Lisberg and Woods, 2018). Studies have also shown that quality mentorship increases recruitment of UR mentees into graduate school and research-related career paths (Hathaway et al., 2002; Junge et al., 2010; Nagda et al., 1998; Thiry and Laursen, 2011).

Outcomes of Mentorship in Higher Education Outside of STEMM for Mentees

Researchers have conducted a wide range of qualitative and quantitative studies on mentorship outcomes in higher education outside of STEMM. In qualitative studies, for example, investigators used case study methods and interviews to study recommended characteristics of mentorship, how students and mentors experience the mentoring relationship, and what both students and mentors expect from mentoring relationships and what their roles are in that relationship (Baker and Griffin, 2010; Bell and Treleaven, 2011; Griffin, 2013). For the most part, quantitative research has examined college adjustment (Apprey et al., 2014), career and personal development (Haddock et al., 2013; Kinkel, 2011; Sams et al., 2015), and measures of academic progress and success (Fox et al., 2010; Hu and Ma, 2010; Zell, 2011).

___________________

¹² This report refers to UR groups as including women of all racial/ethnic groups and individuals specifically identifying as Black, Latinx, and American Indian/Alaska Native. Where possible, the report specifies if the UR groups to which the text refers to Black, Latinx, or of American Indian/Alaska Native heritage.

¹³ This topic is explored in more depth in Chapter 3.

Most of these studies in higher education outside of STEMM did not distinguish between mentorship and other forms of supportive relationships, including those with advisors, institutional agents, developers, and coaches (Baker and Griffin, 2010; Bettinger and Baker, 2011; Museus and Neville, 2012; Tovar, 2015). Nonetheless, there are lessons from these studies that suggest what outcomes STEMM mentees might experience. This research suggests, for example, that informal mentorships are more likely to be successful for mentees and result in outcomes superior than with formal mentorship, which is when relationships are based on assigning students to mentors (Davidson and Foster-Johnson, 2001; Gandara and Maxwell-Jolly, 1999).¹⁴ This research also shows that career and psychosocial support in mentorship often contribute in different ways to different types of outcomes for mentees, and that career support typically results in better career outcomes, such as greater publication output for graduate students (Haeger and Fresquez, 2016; Tenenbaum et al., 2001). Psychosocial support results in outcomes that are crucial for student well-being and other criteria necessary for promotion and productivity, such as greater satisfaction with the mentoring relationship and commitment to one’s own academic program (Phinney et al., 2011). Other positive outcomes from mentorship programs include increased academic performance and involvement in programs at the college or university (Brittian et al., 2009; Dahlvig, 2010), better transition and adjustment to the college environment (Smojver Ažić and Antulić, 2013), improved personal and career development (Kinkel, 2011), more degrees conferred and persistence through programs (Gross et al., 2015), and positive civic outcomes such as increased social responsibility and socially responsive leadership (Haddock et al., 2013).

Outcomes from a Relationship Perspective

Mentoring relationships can be characterized by the purpose, intensity, and duration of the relationship. Successful mentoring relationships result from a mentor’s intentional and purposeful commitment to helping the mentee succeed (Baker and Griffin, 2010). Additionally, mentorship may help develop students’ time management skills, study skills, communication skills, and other transferable skills sets, as well as helping them adjust to college (Michael et al., 2010; Salinitri, 2005). Helping to guide and engage students in research, providing direction in career goals, and creating a sense of belonging in college departments are strategies that have proved successful in mentorship programs (Crisp et al., 2017).

Measuring outcomes from the mentoring relationship perspective highlights the value of having parallel measures from both sides of the relationship: that of the mentor and the mentee. Such parallel measures can elucidate the degree to which mentees and mentors have shared views about the mentoring relationship and mentoring activities, which can be an indicator of their working alliance. One example of parallel mentoring

___________________

¹⁴ Informal and formal mentoring relationships are discussed in Chapter 4.

relationship measures is from the Howard Hughes Medical Institute Gilliam Fellowships for Advanced Study, a predoctoral program for UR students in STEMM. A survey posed questions to Gilliam mentors and mentees in dyadic pairs about behaviors in the mentoring relationships related to facilitating students’ research and career development and science identity. Results of the survey revealed a mismatch on some aspects in the mentoring relationship. Namely, mentors reported displaying more of the desired behavior, such as, mentors sharing their own research career pathway, highlighting and giving direction for improving mentees’ research outcomes, and affirming mentees’ ability to be a scientist, than their mentees reported perceiving (see Table 6-2) (Pfund et al., 2019). These findings indicate that further inquiry into how different views of mentoring activities influence mentorship outcomes could be useful, and they also point to the potential value of mentorship education in supporting mentors’ career facilitation for students.

Few studies on mentorship outcomes appear to use theoretical frameworks focused on the relational elements of mentoring, such as social support, that emphasize how relationships reduce stress and promote coping, or developmental support, which links mentorship to the college student developmental process. However, several studies (Aikens et al., 2016, 2017) have used social capital theory as a framework for examining the effect of mentorship structures between students, doctoral and postdoctoral

TABLE 6-3 Parallel Mentor and Mentee Measures Assessing Social Cognitive Career Theory (SCCT) Variables and Cultural Diversity Awareness of Mentors

SOURCES: ^aByars-Winston et al., 2016; ^bByars-Winston and Butz, 2018.

scholars, and faculty on various outcomes.¹⁵ These investigations found that in “closed mentorship triads,” which included a faculty mentor, a graduate student or postdoctoral mentor, and an undergraduate student mentee,¹⁶ interactions were the most beneficial for mentee outcomes such as science identity development (Aikens et al., 2016, 2017), scholarly productivity, and intentions to pursue a STEM Ph.D. (Aikens et al., 2016). In addition, several researchers have developed parallel measures for mentors and mentees in STEMM based on social cognitive career theory and science identity as well as multicultural theory (Byars-Winston et al., 2016).¹⁷ These parallel mentor and mentee measures assess elements in the mentoring relationship related to mentees’ research self-efficacy beliefs and mentors’ cultural diversity awareness (see Table 6-3).

Measuring Mentor Motivations and Correlates

Assessment and measurement of mentorship could integrate how and why mentors participate in mentorship and what they gain from successful mentorship. For example, one qualitative case study found that graduate students and postdoctoral researchers who mentored undergraduates in research reported improved career preparation and

___________________

¹⁵ Social capital theory is described further in Chapter 2.

¹⁶ Triad configurations of mentorship are discussed in Chapter 4.

¹⁷ Further discussion of social cognitive career theory is in Chapter 2.

qualifications, cognitive and socioemotional growth, improved teaching and communication skills, greater enjoyment of their own apprenticeship experience, and twice as many benefits as challenges (Dolan and Johnson, 2009). Their motivations for engaging in mentorship were largely about how mentorship would serve as a means to an end, though the benefits and challenges they reported indicated a longer-term vision of how mentorship influenced their personal, cognitive, and professional growth. At the same time, some mentors in this study reported that mentorship of undergraduates made their work lives more enjoyable while generating emotional costs. Several investigators have reported that mentors benefit from a sense of personal fulfillment through knowledge and skill sharing, honing their leadership skills, career preparation, and cognitive growth (Dolan and Johnson, 2009; Eagan et al., 2013; Laursen et al., 2010).

Another qualitative study determined that mentors had both career and intrinsic motivations for mentorship in the context of undergraduate research, which appeared to differ by career stage (Hayward et al., 2017). Career motivators for faculty included increased productivity, help in recruiting future students, increased prestige for the university resulting from students presenting at conferences, and helping prepare students for graduate work and careers. Intrinsic motivators included improved teaching and mentorship skills, feelings of doing something positive, preparing future scientists, and increased energy and enthusiasm in the lab. Faulty mentors of undergraduates were motivated by their belief that mentorship informed their teaching and added fun and enthusiasm to their work, while negative factors included the need for additional time, effort, and funding; increased tension; increased difficulty of gauging students’ research ability; and little recognition or reward (Dolan and Johnson, 2010). Forming mentoring relationships with graduate students helped faculty recruit undergraduates and gain a better sense of postgraduates, but study participants had trouble gauging the effectiveness of mentorship.

Research outside of STEMM indicates that mentors’ commitment to the mentoring relationship matters for mentorship outcomes (Allen and Eby, 2008). Given competing role demands on mentors and mentees in STEMM and work settings, mentor commitment is not necessarily a given and is often an outcome of many factors (Aryee et al., 1996). In fact, research outside of STEMM indicates that mentors’ identities and their perceptions of the benefits of mentorship toward their own career goal progression play a role, along with factors such as altruism and the presence of effective schemata for developing and sustaining relationships with mentees (Ragins, 2009).

Even though effective mentorship has been shown to relate to positive career outcomes for mentors in workplace settings (Ghosh and Reio, 2013; Rogers et al., 2016), the relationships between effective mentorship and career outcomes for mentors in STEMM settings are not always self-evident. Research on work performance (Kerr, 1995; Van Eerde and Thierry, 1996) suggests that individuals have to understand that certain tasks and the quality of task completion will factor into organizational reward systems and ultimately the individual-level compensation and rewards they receive. In other words, the

value and attention paid to mentorship quality might change if it became a tracked and managed component of universities’ and research organizations’ performance appraisal system for faculty and other researchers who engage in STEMM mentorship (Aryee et al., 1996).¹⁸ It is important to note that there may be unintended consequences of efforts to track and manage mentorship, especially if mechanisms are not carefully identified and vetted by professional assessment developers to minimize inequities and bias.

NEW AND EMERGING APPROACHES TO ASSESSMENT AND MEASUREMENT OF MENTORSHIP

Reciprocal exchanges between mentors and mentees in postsecondary STEMM contexts warrant further study. However, existing research on relationship theory points to the essential nature of reciprocal exchanges between relational partners (Brown, 1991; Fiske, 1992) and can provide insight relevant to STEMM mentoring relationships. Here, the committee explores some recent advances in two methodologies—dyadic data analysis and social network analysis of mentorship—and poses further questions for inquiry.

Dyadic Data Analysis

Relatively recent advances in statistical methodology now allow for characterizing reciprocal relationships through dyadic data analysis (Kenny, 1994; Kenny et al., 2006).¹⁹ Dyadic data analysis involves collecting data from both the mentor and the mentee over time to reveal how the perceptions and experiences of each influence the other. For example, a mentor’s perceptions of the mentee have the potential to influence the mentee’s self-perceptions, but this influence can only be examined if data are collected from both the mentor and the mentee over time. This methodology allows researchers to investigate dynamic feedback loops between mentor and mentee, where each informs the other regarding what is or is not needed from the relationship, how the relationship quality and characteristics such as trust development shift over time, and how this influences both mentor and mentee. For instance, this method could reveal how change in trust over time from both the mentor and the mentee perspective influences mentee perceptions that they are receiving psychosocial support or mentee confidence in their ability to be successful in a STEMM career. One study used a dyadic approach to characterize reciprocal feedback between mentors and mentees in a STEMM research experience context (Griese et al., 2016).

___________________

¹⁸ These topics are explored more deeply in Chapter 7.

¹⁹Dyadic data analysis is a general methodology that captures the reciprocal nature of a relationship and its influence on both members in the relationship (Kenny, 1994; Kenny et al., 2006).

Social Network Analysis of Mentorship

Advancements in mentorship theory point to the importance of networks of mentoring relationships, particularly for individuals from historically UR groups (Downing et al., 2005; Glessmer et al., 2012; Higgins, 2000; Higgins and Kram, 2001; Higgins and Thomas, 2001; Packard, 2003a; Packard et al., 2004). Recent advancements in measurement and statistical methodology now allow researchers to capture and quantify characteristics of mentorship-related networks as social networks (Scott, 2017). Social networks can be conceptualized either as “whole networks” or as “ego networks.”

Whole networks are systems such as a mentorship group. Whole network analysis could be used to analyze the value of collective or group mentorship, including the value of the network based on the resources offered by its members, such as expertise and information; the diversity of its members; which relationships within the network are most influential; how interconnected members must be for the network to be valuable to its members; where there might be gaps in the network; and which members of the network serve as hubs for information or resources such as high-quality feedback. Several researchers have begun to measure and categorize beneficial triadic mentor network structures as the simplest form of a whole network (Aikens et al., 2016, 2017; Morales et al., 2018) and to identify and characterize successful mentorship communities (Chariker et al., 2017), but much more can be done to determine how mentorship networks operate and their distinctive impact and value.²⁰

Ego networks are the connections, or lack thereof, of a single individual and the resources available, or not, to the individual through their connections. Ego network analysis could be used to examine the mentorship resources available to a given mentee and how these resources relate to their personal characteristics and outcomes. For example, mentees with different racial, ethnic, and gender identities can differ in their mentorship networks in ways that may or may not influence their outcomes (Aikens et al., 2017). Longitudinal ego network research is appropriate to determine whether mentees with different personal characteristics are more or less likely to develop mentorship networks that meet their needs. For instance, mentees who identify strongly with their mentor may perceive that they are receiving both career and psychosocial support and thus may require a simpler dyadic mentorship structure to meet their needs. Mentees who do not identify strongly with their mentor either personally or professionally may benefit from a more elaborate network of mentors, including others who share their identities or particular career interests. These questions could be addressed through systematic analysis of the ego networks of mentees related to their personal characteristics and outcomes.

___________________

²⁰ Insights from different forms of mentorship can be found in Chapter 4.

Further Questions for Inquiry

Understanding the mechanisms by which mentorship is initiated, developed, and sustained, and if they are effectual, is important for theory building and for practical purposes. For example, if research can identify specific, favorable mentee and mentor behaviors, it could be possible to enhance and encourage those behaviors through programming and evaluation systems, and thus improve mentoring relationships and resulting outcomes.

Ideally, assessments would identify important milestones in developing mentoring relationships. In addition, assessments could provide details on whether relationships develop in a linear manner or if there are discontinuous changes or time-bounded needs of mentees, mentors, or mentoring relationships that must be taken into account to develop an effective mentoring relationship and fully realize its benefits to mentees, mentors, and the STEMM enterprise. For more quantitative data, statistical techniques could be used to identify unobservable subgroups based on measured variables or trends in larger data sets, such as probability-based latent class analysis (Bauer and Shanahan, 2007; Oberski, 2016; Pastor et al., 2007; Wachsmuth et al., 2017). More work is needed to minimize selection bias in assessing mentoring outcomes, for example, matched control groups or propensity score matching.

Most mentorship theories suggest that mentoring relationships change over time,²¹ and most correlational research assumes that change is linear—that as trust increases, for example, so does relationship quality. However, experience implies that relationships can shift suddenly, such as when one act of betrayal irreversibly destroys a relationship or when one act of kindness transforms a struggling relationship. Research on turning points in close relationships suggests using both quantitative and qualitative methods to develop robust, explanatory theory. Research could potentially determine if there are predictable patterns of discontinuous change, identify experiences that fundamentally alter mentoring relationships, and learn if positive turning points can repair a previously damaged mentoring relationship (NRC, 2002; Warfa, 2016).

There has been little research on multilevel influences arising from mentoring relationships being nested within workgroups, academic departments, research laboratories, organizations including colleges and universities, and industries or academic disciplines. Research is also lacking on aggregate effects that go beyond the individual, such as work-group- or department-level effects. Multilevel modeling can help examine individual, dyadic, group, and organizational effects on the mentoring relationship.

___________________

²¹ Such as the theories discussed in Chapter 2.

This page intentionally left blank.