Over the course of its research and deliberations, the committee identified several issues that apply to both master’s and Ph.D. levels of graduate science, technology, engineering, and mathematics (STEM) education. These crosscutting themes are the subject of this chapter. Issues particular to master’s or Ph.D. education are discussed in Chapters 4 and 5, respectively. The crosscutting issues include:
- improving STEM graduate education by adjusting faculty rewards and incentives as they pertain to teaching and mentoring;
- collecting and disseminating data to increase transparency for prospective and current STEM graduate students about institutional degree and career outcomes, among other metrics;
- increasing diversity, equity, and inclusiveness throughout STEM graduate programs to cultivate talent from all backgrounds and promote continued scientific leadership;
- building the ability of the STEM graduate education system to adjust to the dynamic nature of the scientific enterprise and the career options available to its students; and
- optimizing the experiences that graduate students have while in their programs.
One of the main themes of this report is its call for cultural change at the nation’s universities that puts students at the center of the graduate school experience. This change in culture, coupled with the set of actions laid out in this and subsequent chapters, would move the graduate education system significantly
closer to the ideal set of educational experiences articulated in Chapter 6 and in the Summary of this report. It would also help bridge the partial disconnect between the full range of faculty activities needed to prepare STEM graduate students for the 21st-century work environment writ large—including appropriate advising and mentoring and exposing them to career options outside of academia—and the incentives that drive faculty behavior in terms of tenure, promotion, and merit raises that are based largely on research productivity and results.
At most research universities, the incentive and reward system for faculty emphasizes publication rate and amount of grant funding as the main metrics for tenure and promotion. Although the evaluations of the quality of teaching and mentoring activities are collected, these qualities do not receive as much emphasis in the overall evaluation (NAS/NAE/IOM, 1996, 1997). This imbalance is well known throughout academia, and addressing it has been the subject of recommendations in previous reports, on both graduate STEM education and reform in higher education broadly (NRC, 2012).
In addition to the adverse effects that the current incentive structure can have on graduate education, many programs do not employ teaching or mentoring practices based on the emerging evidence base about the most effective pedagogical practices or about the ways adults learn. Research on undergraduate education, for example, has demonstrated that using effective pedagogical practices, such as “active learning,” increases student learning and retention (Freeman et al., 2014). Regarding mentoring, the Center for the Improvement of Mentored Experiences in Research at the University of Wisconsin–Madison1 has developed a wealth of resources for institutions to improve research mentoring relationships. Another potential resource for faculty members is the National Research Mentoring Network,2 supported by the National Institutes of Health (NIH), which is developing a research base around the science of mentorship. The network has already developed a variety of curricular offerings to promote effective and inclusive mentoring practices.
Although high-quality and student-focused faculty mentoring and advising are essential to the education of STEM graduate students, the academic ecosystem does not reward these behaviors as highly as it does research productivity, publications, and other traditional metrics of success. Faculty members should be given the time, resources, know-how, and incentives to devote attention to mentorship. Early-career faculty in the process of establishing themselves in a
department may require a primer on effective approaches to mentoring, while more senior faculty would benefit from establishing a baseline of existing mentoring and advising skills and from periodic refreshers to explore new skills or techniques in supporting student success.
Education researchers have noted how important it is for institutional leaders to create a set of rewards, incentives, and well-defined criteria (Filetti, 2009; Law et al., 2014; Marcellino, 2011) that encourage advisors and mentors to deliver quality guidance (Drake, 2008; Habley, 2007) and research opportunities (Davis et al., 2015; Schultheis et al., 2011). Although some faculty may strongly resist changing tenure and promotion policies to reward activities outside of research achievements (Brownell and Tanner, 2012), at least a few institutions have overcome faculty objections to reward such activities as they relate to undergraduate students (Purdue University, 2015; University of Arkansas, 2011). Institutional leadership plays a critical role in promoting and sustaining faculty incentive policies that acknowledge the importance of engaging in student-centered activities (Fountain and Newcomer, 2016).
In its advice to new graduate students, the NIH Office of Intramural Training and Education says the best mentors are advisors, coaches, counselors, and supporters all at the same time.3 They are experienced scientists who guide graduate students, but also challenge them to develop their independence. A good mentor helps students define their research goals and then supports them in their quest to achieve those goals.
Both the paper by Margaret Blume-Kohout that was commissioned for this study and findings in the mentoring literature claim that those students who were most satisfied with their mentors reported that those mentors had several attributes in common (Blume-Kohout, 2017). These mentors challenged and stimulated their students’ thinking, were helpful and encouraging, were enthusiastic about the student’s research, and contributed to the student’s professional development. They were also approachable and generous with their time, gave appropriate amounts of freedom and direction, and provided regular and constructive feedback on both research and academic progress. (Lovitts, 2004; Zhao et al., 2007).
It is uncommon that a student will find a single individual or advisor who has all these attributes. In many instances, students can benefit from having multiple mentors (Higgins, 2000) given that different mentors can provide guidance on different topics. At least one institution, the Watson School of Biological Sciences at Cold Spring Harbor, requires that students have a “research mentor,” and separately, an “academic mentor.”
One of the key themes that students raised in the focus groups conducted by Research Triangle International (RTI) for this study4 was how much their gradu-
4 See http://sites.nationalacademies.org/cs/groups/pgasite/documents/webpage/pga_186164.pdf (accessed May 16, 2018).
ate school experience depended on their supervising faculty advisor and their relationship with him or her. Participants noted challenges they faced when their educational and career goals differed from those of their advisor. Many of these students did not feel comfortable pursuing courses, workshops, or other professional development opportunities outside the focus of the advisor’s research for fear of putting the relationship at risk. The stigma, whether real or perceived, associated with a student’s pursuit of a career outside academia also needs to be addressed. Adjusting the incentive system, perhaps by including metrics on student outcomes beyond traditional measures, such as placement of students at research-intensive institutions, is one approach that may be effective in achieving the goals discussed in this section.
Given the importance of effective teaching and mentoring to providing a more effective, student-centered graduate school experience, the committee makes the following two recommendations:
RECOMMENDATION 3.1—Rewarding Effective Teaching and Mentoring: Advancement procedures for faculty, including promotion and tenure policies and practices, should be restructured to strengthen recognition of contributions to graduate mentoring and education.
- Federal and state funding agencies should align their policies and award criteria to ensure that students in the programs they support experience the kind of graduate education outlined in this report and achieve the scientific and professional competencies articulated here, whether they are on training or research grant mechanisms.
- Institutions should increase priority and reward faculty for demonstrating high-quality teaching and inclusive mentoring practices for all graduate students, including the recognition of faculty teaching in master’s degree programs, based on the results of restructured evaluations.
- Institutions should include teaching and mentoring performance as important considerations for reappointment, promotion, annual performance review, and tenure decisions. Institutions should also nominate faculty for external awards (such as those from technical societies) that reward teaching excellence.
RECOMMENDATION 3.2—Institutional Support for Teaching and Mentoring: To improve the quality and effectiveness of faculty teaching and mentoring, institutions of higher education should provide training for new faculty and should offer regular refresher courses in teaching and mentoring for established faculty.
- Institutions should require faculty and postdoctoral researchers who have extensive contact with graduate students to learn and demonstrate evidence-based and inclusive teaching and mentoring practices.
- Graduate programs should facilitate mentor relationships between the graduate student and the primary research advisors, as well as opportunities for students to develop additional mentor or advisor relationships, including with professionals in industry, government laboratories, and technical societies.
- Graduate schools should provide extra-departmental mentoring and support programs.
- Graduate students should seek multiple mentors to meet their varied academic and career needs.
The ability to understand the current state of and emerging trends in the graduate STEM enterprise depends on the quality, breadth, and transparency of data and research about graduate education. For data, a number of organizations collect information ranging from longitudinal datasets to periodic collections held by institutions or professional societies. These data shed light on specific components of the system, such as individual disciplines; however, previous reports on graduate education have called on the primary collectors of data, including federal agencies, institutions, and professional societies, to collect and, more critical, share a broader array of standardized, common metrics on a regular basis. Those recommendations have not generally been implemented or have been implemented unevenly across institutions (Hussain, 2017). As a result, it is difficult, for example, to track how graduate students fare during their programs and after graduation.
The federal government is one of the principal collectors of longitudinal STEM graduate education data. With specific attention to the post-baccalaureate-level population, the National Science Foundation’s (NSF’s) National Center for Science and Engineering Statistics conducts the following surveys:
- National Survey of College Graduates, a longitudinal biennial survey conducted since the 1970s that provides data on the nation’s college graduates, with particular focus on those in the science and engineering workforce;
- Survey of Graduate Students and Postdoctorates in Science and Engineering, an annual census of all U.S. academic institutions;
- Survey of Earned Doctorates, an annual census of all research doctorate recipients;
- Survey of Doctoral Recipients, a sample designed to provide a cross-sectional estimate of the activities of research doctorate recipients; and
- Early Career Doctorates Survey, which gathers in-depth information about individuals who earned their first doctoral degree (Ph.D., M.D., or equivalent) in the past 10 years.
In addition, the National Center for Education Statistics (NCES) conducts a variety of national assessments that include longitudinal data on postsecondary education, including graduate-level metrics across all disciplines. NCES also administers the Baccalaureate and Beyond Longitudinal Study, which examines students’ education and work experiences after they complete a bachelor’s degree, with a special emphasis on the experiences of new elementary and secondary teachers. The Bureau of Labor Statistics (BLS) also provides career outcome data by levels of educational attainment.
Outside of the federal government, professional societies also play a key role in collecting and sharing data for their constituent audiences. The Council of Graduate Schools (CGS) has provided a report on annual Graduate Enrollments and Degrees, a joint effort with the Graduate Record Examinations (GRE) Board, since 1986 (Okahana and Zhou, 2017). CGS also releases data through a variety of other annual surveys, such as the International Graduate Admissions and Pressing Issues Survey, and through issue-oriented projects, such as the Doctoral Initiative on Minority Attrition and Completion and the recently launched data phase of the Understanding PhD Career Pathways for Improvement Program. Disciplinary professional societies also collect information on graduate education as a part of the information-gathering activities on the state of the field. In December 2017, a coalition of 10 university presidents—the Coalition for Next Generation Life Sciences—announced plans for their institutions to collect and report comprehensive data on graduate student outcomes (Blank et al., 2017).5
A lack of publicly available data on graduate student outcomes at the institutional levels makes it difficult for students to make informed choices about their training activities and for universities to prepare graduate students for a full range of careers (Blank et al., 2017). Lacking the information needed to fully understand the path ahead of them can limit students’ ability to choose a suitable institution and discipline and be clear about the career options available to them with the degree they decide to pursue (Polka et al., 2015). National data-gathering efforts, such as the NSF surveys described above, provide coverage at a broad STEM or discipline level. However, there are data at the institutional level that should be captured to provide potential students the ability to compare programs and for departments to make data-driven decisions to inform continuous im-
provement. Many institutions collect these data for internal purposes, though few choose to share their data with the public. The critical data points, which would benefit from regularly scheduled updates, include
- number of applicants and accepted students;
- metrics on current student population by gender, race and ethnicity, and visa status;
- time to candidacy, time to degree, and completion rate, aggregated for all students;
- percentage of students funded with tuition and living expenses, along with percentages of source of funding (assistantships, traineeships, fellowships, faculty members’ research grants);
- debt level of undergraduates entering the graduate program and of graduates upon completion for an understanding of how incoming financial aid may affect student outcomes and career decisions;
- positions obtained by alumni of graduate programs at 1, 5, 10, and 15 years after graduation in all workforce sectors, and salary and job satisfaction at those time points; and
- student satisfaction with their graduate program and career options and opportunities for employers to provide feedback regarding strengths and gaps in skills and competencies of hired graduates.
In addition to collecting data about students, universities—and particularly financial aid offices—should provide information to all entering students on national salary data by field. Because salary information does not create a complete picture of student satisfaction, institutions should make additional efforts to include other metrics to indicate how graduates relate their education to their career. Financial aid offices should also provide entering graduate students with information on the total cost of their graduate experience, including living expenses, and the various forms of financial support, including student loans, available for graduate students.
There is some promising activity under way to address the current lack of transparency. The Association of American Universities (AAU) in 2017 issued a policy statement (Flaherty, 2017) calling on “all Ph.D. granting universities and their respective Ph.D. granting colleges, schools and departments, to make a commitment to providing prospective and current students with easily accessible information.” The AAU stated explicitly that such data should include student demographics, average time to finish a degree, financial support, and career paths and outcomes both inside and outside academia. The presidents of 10 leading research-intensive universities announced that they would collect and make publicly available comprehensive data on graduate student outcomes (Blank et al., 2017), and the University of Michigan Rackham Graduate School has been collecting such data since 2003. The University of Michigan data are now accessible for every graduate department through an interactive dashboard highlighted as
the first item on the graduate school’s home page.6 Some federal grant programs, such as the National Institute of General Medical Sciences’ T32 Institutional Predoctoral Training grants, also require reporting on career outcomes.7
The CGS Understanding PhD Career Pathways for Improvement Program builds on 3 years of work with institutions, survey researchers, professional and disciplinary associations, and labor force economists. This effort will provide more extensive information on the nature of work by collecting information about currently enrolled doctoral students and alumni from 61 participating institutions. Unlike previous efforts that broadly describe occupational functions or provide information only on tenure-track academic positions, the CGS project will capture data on all graduates, including those employed in industry, government, and nonprofits, who no longer produce peer-reviewed publications and were not supported on federal traineeships, research assistantships, or fellowships during graduate school.
The Institute for Research on Innovation and Science (IRIS), based at the University of Michigan, provides a platform for linking university administrative records with U.S. Census data, as well as linking administrative data with other databases, including patents and publications.8 One recent study using the IRIS system was able to identify where recent science and engineering Ph.D.’s are finding jobs (Zolas et al., 2015). In 2016, the University of Minnesota released IPUMS Higher Ed,9 a publicly available tool that harmonizes multiple NSF datasets—the National and International Survey of Doctoral Researchers10 databases and Survey of College Graduates11 and National Survey of Recent College Graduates12 databases—from 1990 to 2013. IPUMS Higher Ed provides a user-friendly data extraction system to track career trajectories of Ph.D.’s across different occupations, including in academia, government, industry, and other types of research involvement.
These actions are important steps toward addressing the information gap and increasing transparency about STEM graduate education and career outcomes. As these initiatives and other efforts continue to increase the amount of data available, one challenge that may arise is the ability to compare or to cross-walk metrics from one dataset to the next. In July 2017, NORC at the University of Chicago held a stakeholder workshop on existing efforts to track the career paths and professional outcomes of graduate degree holders. This workshop also
6 See https://tableau.dsc.umich.edu/t/UM-Public/views/ProgramStatisticsPhD2016/ProgramStatistics?embed=y&showShareOptions=true&display_count=no&FOSDParameter=All%20Rackham&:isGuestRedirectFromVizportal=y&:embed=y (accessed March 20, 2018).
7 See https://www.nigms.nih.gov/Training/InstPredoc/Pages/default.aspx (accessed January 22, 2018).
examined ways that programs can use data to inform program effectiveness, as well as how to support coordination and partnerships to improve the network of efforts in this area.13 At the institutional level, there is an opportunity to define and to collect metrics in a standardized way across departments that could make aggregation feasible with other institutions and scale to a national level.
The availability of data also improves the capacity of the research community to understand the graduate STEM education enterprise and help plan any needed adjustments. While there has been increased attention to the pedagogy and practice in effective STEM undergraduate education, which includes critical components of active learning, blended classrooms, and discipline-based education research, there is a relatively smaller proportion of educational research targeted toward understanding effective models and practices in graduate education.14 Given the importance to the future of the STEM research enterprise of increasing retention and degree completion rates for historically underrepresented minorities, there is a critical need for research on the programs and models that most effectively support those students (NAE, 2014; NAS/NAE/IOM, 2011).
Currently, most studies of this sort employ a small sample size from a single discipline, making it challenging to determine whether the findings can be generalized to other groups of students or to other disciplines. One notable exception is the NIH Broadening Experiences in Scientific Training (BEST) program.15 Started in 2013, the BEST program funded 17 sites with the focus on improving career exploration in the biomedical sciences. Each site designed an experimental approach, based on the local context of the institution, as well as an assessment to evaluate the outcomes and impact of the work. The predominant goal of the BEST program has been to identify evidence-based practices that other institutions could adopt or adapt. Toward that end, the program includes a continuous and rigorous evaluation process. Given the significant level of investment from federal agencies in graduate-level STEM training and education programs, the products of those investments have the potential to grow beyond the individual student participants and expand into a set of evidence-based practices for other institutions to adopt or adapt. For example, the University of California, San Francisco, and the University of California, Davis, reported that the programs students participate in to expand career development skills and promote career exploration did not increase median time to degree for the 217 Ph.D. students in the BEST programs at those institutions (Schnoes et al., 2018).
13 See https://www.spencer.org/graduate-degree-holder-career-paths-workshop (accessed January 22, 2018).
14 See online resource at http://sites.nationalacademies.org/cs/groups/pgasite/documents/webpage/pga_186176.pdf for literature review.
RECOMMENDATION 3.3—Comprehensive National and Institutional Data on Students and Graduates: Graduate programs should collect, update, and make freely and easily accessible to current and prospective students information about master’s- and Ph.D.-level educational outcomes. In addition, to make appropriate future adjustments in the graduate education system, it is essential that comprehensive datasets about the system, its participants, and its outcomes be collected in a standard format, be fully transparent, and be easily accessible and transferable across multiple computer and statistical analysis platforms.
- Federal and state funding agencies should require institutions that receive support for graduate education to develop policies mandating that these data be collected and made widely available to qualify for traineeships, fellowships, and research assistantships.
- Institutions should develop a uniform, scalable, and sustainable model for data collection that can operate beyond the period of extramural funding. The data collection should follow standard definitions that correspond with national STEM education and workforce surveys to help inform benchmarking or higher education research.
- Departments and programs should review their own data from current students and alumni to inform curricula and professional development offerings, and they should provide these data to current and prospective students.
- Prospective students should use these data to inform graduate program selection, educational goal development, and career exploration.
RECOMMENDATION 3.4—Funding for Research on Graduate STEM Education: The National Science Foundation, other federal and state agencies, and private funders of graduate STEM education should issue calls for proposals to better understand the graduate education system and outcomes of various interventions and policies, including but not limited to the effect of different models of graduate education on knowledge, competencies, mind-sets, and career outcomes.
- Funders should support research on the effect of different funding mechanisms on outcomes for doctoral students, including traineeships, fellowships, teaching and research assistantships; the effects of policies and procedures on degree completion, disaggregated by gender, race and ethnicity, and citizenship; and the effect of expanding eligibility of international students to be supported on federal fellowships and training grants.
Diversity in science refers to cultivating talent and promoting the full inclusion of excellence across the social spectrum, including people from backgrounds that are traditionally underrepresented and those from backgrounds that are traditionally well represented (Gibbs, 2014). The STEM graduate education enterprise as a whole must seek to enable students of all backgrounds to succeed by implementing mentoring practices and pedagogies that create an inclusive institutional environment in terms of gender, age, culture, ethnicity, and nationality; that make available opportunities for productive dialogue; and that encourage diverse perspectives that can lead to a deeper understanding of how people from different backgrounds may approach learning and problem solving in different ways (Gibbs, 2014).
Expansion of educational opportunities and engagement of a broader and more diverse cross section of the U.S. population in STEM fields is a national priority (NAS/NAE/IOM, 2011) given the importance of drawing on the unique perspectives that people from different backgrounds and with different experiences bring to addressing the most challenging scientific problems facing society today. Research has shown, in fact, that scientific creativity benefits from having multiple viewpoints shaped by the different life experiences of its group members (Ferrini-Mundy, 2013; Page, 2008; Reagans and Zuckerman, 2001; Roberge and Van Dick, 2010; Saxena, 2014). In addition, the enormous output of the U.S. scientific community depends on a constant supply of scientific talent, and a lack of diversity represents a loss of talent (NAS/NAE/IOM, 2011). Diversity of ideas also comes with diversity of the talent pool. As NIH Director Francis Collins and Lawrence Tabak, NIH’s principal deputy director, have stated, a lack of diversity in the STEM student body and workforce leads to “the inescapable conclusion that we are missing critical contributions” (Tabak and Collins, 2011, p. 941). Given that the demographics of the U.S. population have shifted over the past 50 years—most children born today in the United States are not white, and a growing fraction of the workforce is female (U.S. Census Bureau, 2012)—tapping into every available pool of talent is essential for the United States to retain its world leadership in science, engineering, and technology.
Moreover, improving representation in STEM graduate education is critical to future employment needs and to ensuring equity for the growing minority-majority population in the United States. According to the BLS, employment in STEM and STEM-associated occupations is projected to grow faster than the average for all occupations (Vilorio, 2014). BLS estimates that overall STEM employment will grow approximately 13 percent between 2012 and 2022, faster than the 11 percent projected growth for all occupations over the same period. Moreover, BLS predicts that nearly all STEM occupations, which pay on average nearly double the median wage for all workers, will experience growth during that time.
Currently, graduate programs do not attract or develop talent from all sec-
tors of the nation’s population; women and certain racial and ethnic groups remain underrepresented in many (but not all) disciplines. As a result, graduate programs need to strengthen the culture of equity and inclusion that prepares all students for successful careers and that equips all students with the career skills needed to overcome the challenges they will face in graduate school and beyond. Comparative research to understand how certain departments, institutions, and disciplines have been successful in increasing both the number and success of underrepresented students in graduate programs could help others who are facing similar challenges.
The past few decades have seen increases in the participation of students from historically underrepresented groups and female students; however, progress toward parity looks different from discipline to discipline. Overall, data from the NSF show that the number of minority students pursuing graduate STEM degrees more than doubled in the two decades from 1989 to 2009, with the number of Hispanic and Latino/a graduate students in STEM programs nearly tripling and the number of black or African American students more than doubling (Einaudi, 2011, p. 4). The inclusion of historically underrepresented minorities, notably at the doctoral level, is intertwined with the challenges in developing the equitable representation around faculty. As noted in the National Academies report, Expanding Underrepresented Minority Participation, “Not only does it provide underrepresented minorities [doctoral students] an opportunity to contribute to teaching and research, but it is at this level that increases can have a multiplier effect. . . . As the number of underrepresented minorities in faculty positions increases, the more role models underrepresented minority students have who ‘look like them’ and the higher rate at which underrepresented minority students enroll and graduate.” (NAS/NAE/IOM, 2011, pp. 46-47)
Research has already shown that mentoring that intentionally addresses the challenges faced by underrepresented groups can be highly effective at empowering student success (Carver et al., 2017; Griffin et al., 2010; Lewis et al., 2016; May, 2016; Packard, 2015). The development of scalable and sustainable initiatives comes with significant challenges. Revisiting admissions policies can expand traditional definitions of merit to include characteristics that recognize student potential, particularly to the benefit of students from historically underrepresented groups. For example, the Fisk-Vanderbilt Master’s-to-PhD Bridge Program, designed to provide underrepresented minority (URM) students a pathway to doctoral studies, added a question to its selection process to assess the applicants’ understanding of their own grit16 and resilience. Since 2004, the program has demonstrated positive results, with 81 percent of those entering the program having gone on to enter doctoral programs (Posselt, 2016).
The extensive effort associated with various intervention programs, edu-
16 Grit is a predisposition for pursuing long-term, challenging goals with passion and perseverance. From http://fisk-vanderbilt-bridge.org/grit-better-than-gre-for-predicting-grad-student-success/ (accessed January 18, 2018).
cational approaches, and modified federal policies has driven the increase in underrepresented student populations (Covington et al., 2017; Fleming et al., 2013; Maton and Hrabowski, 2004; Rincon and George-Jackson, 2016; Stassun et al., 2011; Tanner, 2013; Valantine et al., 2016). One program is the NSF’s undergraduate-focused Louis Stokes Alliances for Minority Participation Program17 (LSAMP), an alliance-based initiative that helps universities and colleges transform undergraduate STEM education through innovative, evidence-based recruitment and retention strategies and relevant educational experiences in support of racial and ethnic groups historically underrepresented in STEM discipline. The last full evaluation of this program in 2005 found that the vast majority of program graduates sought additional education after their bachelor’s degrees, and two-thirds of participants later enrolled in graduate school, working toward a master’s, Ph.D., or professional degree (Clewell et al., 2005). At the time of this evaluation, one in four LSAMP graduates had completed a STEM graduate degree. In addition, the majority of LSAMP graduates reported that the program had been helpful as they sought their bachelors’ degrees in STEM and had influenced their decisions to attend graduate school. A comparison between LSAMP and a nationally representative sample of URMs and white and Asian students revealed that LSAMP participants pursued post-bachelor’s coursework, enrolled in graduate programs, and completed advanced degrees at greater rates than did the national comparison groups.
Similarly, the NSF’s Alliances for Graduate Education and the Professoriate18 (AGEP) program seeks to advance knowledge about models to improve pathways to the professoriate and success for historically underrepresented minority doctoral students, postdoctoral researchers, and faculty in specific STEM disciplines and/or STEM education research fields. AGEP has, in fact, enhanced institutions’ efforts to recruit underrepresented minorities into STEM graduate programs. According to a 2011 evaluation of the AGEP program, alliances and institutions funded by the program experienced both successes and challenges in their recruitment efforts (Rodriguez et al., 2011). Reported successes included increased URM enrollments in specific disciplines and a change in campus culture to one that was more supportive and welcoming of diverse students into STEM programs. Challenges included the limited pool of students that universities were drawing from and competition with industry or other institutions.
Evaluations of two programs developed by the National Institute of General Medical Sciences—the Maximizing Access to Research Careers Undergraduate Student Training in Academic Research (MARC U-STAR) Program19 and the Postbaccalaureate Research Education Program (PREP)20—have also shown
17 See https://www.nsf.gov/funding/pgm_summ.jsp?pims_id=13646 (accessed December 21, 2017).
18 See https://www.nsf.gov/funding/pgm_summ.jsp?pims_id=5474 (accessed December 21, 2017).
19 See https://www.nigms.nih.gov/Training/MARC/Pages/USTARAwards.aspx (accessed January 22, 2018).
20 See https://www.nigms.nih.gov/Training/PREP/Pages/default.aspx (accessed January 22, 2018).
gains with respect to entry and completion of doctoral graduate degrees by groups that have been historically underrepresented in the biomedical sciences. Among alumni of the MARC U-STAR Program, which provide trainees with multiyear structured training programs and a summer research experience at a research-intensive institution outside the home institution, approximately 59 percent enrolled in Ph.D. programs and two-thirds completed their degrees (Hall et al., 2016). In addition, 65 percent of PREP scholars, who receive support to work as apprentice scientists in a mentor’s laboratory and participate in courses for skills development, matriculated into Ph.D. programs and were found to complete at or above the national average for other students from underrepresented minority groups (Hall et al., 2015). Nonetheless, despite these and other effective programs, such as the NIH Ruth L. Kirschstein National Research Service Award (NRSA) Institutional Research Training Grant (T32), most racial and ethnic groups other than whites and some Asian groups remain underrepresented in the STEM graduate student population compared to the composition of the U.S. population.21
Research has demonstrated that GRE scores used in isolation may have only modest predictive power for many measures of graduate school performance and that reliance on standardized tests can lead to disproportionate selection bias against women and scientists from URM backgrounds (Hall et al., 2017; Miller and Stassun, 2014; Moneta-Koehler et al., 2017). That is not to say that institutions should abandon traditional measures, such as undergraduate grade point average or GRE scores, entirely. However, programs should also be aware that the Educational Testing Service itself, the sponsoring body of the GRE, describes the scores as an “inexact measure” and that “a cut-off score (i.e., a minimum score) should never be used as the only criterion for denial of admission or awarding of a fellowship” (Educational Testing Service, 2017). Alternatively, departments can use data on completion rates and other student metrics to evaluate the degree to which the admissions process is inclusive and equitable.
STEM master’s degree and Ph.D. programs should continuously expose students to multiple worldviews, promote interdisciplinary activities involving individuals from different backgrounds, welcome international students, and employ diverse approaches to teaching and learning. Toward those ends, graduate schools should design programs that account for the complexity of how cultural diversity and career diversity interface with one another (Godwin et al., 2016; Layton et al., 2016). As the trainee pool becomes more diverse, faculty and administrators should consider how to design pedagogical experiences and training opportunities that are inclusive of cultural and societal differences. Attrition rates can be deeply impacted by bringing students from historically underrepresented backgrounds into environments that are not inclusively designed to maximize the likelihood of their success. In other words, if the trainers and environment in science and engineering are largely the reason various branches of these enterprises historically have not been an inclusive space, increasing the number of trainees from
historically underrepresented backgrounds but placing them in this same historic environment will keep them in a system that is biased against their success.
Professional development modules should be required for faculty to learn how to advise and mentor students from different backgrounds and to raise awareness and accountability about their role in changing the training and mentoring environment (Carver et al., 2017; Museus and Liverman, 2010; Packard, 2015). Examples of the types of issues that modules could cover include a step-by-step walk-through on implementing a mentoring compact, how faculty have an impact on trainee self-efficacy, and how to provide the same quality of mentorship to each student without bias.
Graduate programs will have to increase programmatic flexibility to be able to tailor training and career preparation for each student while considering the different needs and cultural values of each individual, and how those values affect future career decisions. All programs also should have access to experts in this sort of work who can assist in correctly identifying and determining student needs for mentoring, personal and professional development, career advice, etc. This may require hiring faculty, administrators, or other experts within each school.
RECOMMENDATION 3.5—Ensuring Diverse, Equitable, and Inclusive Environments: The graduate STEM education enterprise should enable students of all backgrounds, including but not limited to racial and ethnic background, gender, stage of life, culture, socioeconomic status, disability, sexual orientation, gender identity, and nationality, to succeed by implementing practices that create an equitable and inclusive institutional environment.
- Faculty and administrators involved in graduate education should develop, adopt, and regularly evaluate a suite of strategies to accelerate increasing diversity and improving equity and inclusion, including comprehensive recruitment, holistic review in admissions, and interventions to prevent attrition in the late stages of progress toward a degree.
- Faculty should cultivate their individual professional development skills to advance their abilities to improve educational culture and environments on behalf of students.
- Institutions, national laboratories, professional societies, and research organizations should develop comprehensive strategies that use evidence-based models and programs and include measures to evaluate outcomes to ensure a diverse, equitable, and inclusive environment.
- Institutions should develop comprehensive strategies for recruiting and retaining faculty and mentors from demographic groups historically underrepresented in academia.
- Federal and state agencies, universities, professional societies, and nongovernmental organizations that rate institutions should embed diversity and inclusion metrics in their criteria.
- Federal and state funding agencies and private funders that support graduate education and training should adjust their award policies and funding criteria to include policies that incentivize diversity, equity, and inclusion and include accountability measures through reporting mechanisms.
As noted in the Summary and Chapter 1, the research enterprise itself—the way research is done in the United States and abroad—is constantly evolving. This evolution is largely the result of technological advances that have enabled new methods of inquiry, analysis, and collaboration, and advances in knowledge that are enabling the research enterprise to tackle bigger problems that can best—or only—be addressed through a combination of disciplinary approaches and technologies (Disis and Slattery, 2010; NRC and IOM, 2003; Van Noorden, 2015). From basic science through applied research and development, the opportunities made available through the impressive advances of recent decades in instrumentation and across all of science, as well as in the ability and focused efforts to turn scientific advances into technological ones, and vice versa, have yielded robust opportunities (NRC and IOM, 2003). The era of big data and cloud computing, team science that crosses disciplinary boundaries, online publishing, artificial intelligence, nanotechnology, gene editing, and other developments are expanding the ways science is done and the ways researchers think about conducting their studies. STEM graduate education must actively embrace and integrate these new areas and approaches to scientific research to continue educating and training the STEM workforce of the future.
Many fields of science were historically rather solitary activities that, over time, evolved into a “typical” form where a single principal investigator pursued research projects along with a small cadre of graduate students and perhaps a few postdoctoral fellows. That structure, however, is not well adapted to address more complex multidimensional and multidisciplinary problems that require multiple levels of analysis at the same time. As a result, team science is becoming a more “typical” model of research—two analyses of research papers and patents issued over five decades found that teams increasingly dominate the production of knowledge (Plume and van Weijen, 2014; Wuchty et al., 2007)—and thus graduate students trained across STEM disciplines and with collaborators should become comfortable working in teams and with collaborators who may approach research problems differently (NRC, 2015).
Although working in teams has long been a tradition in some fields, such as high-energy physics and many engineering disciplines, many other disciplines have yet to focus on developing these skills in their next generation of researchers. Additionally, industry has long valued employees with the capability to work in teams and collaborate across departments, locations, or with partners. Industry
leaders who spoke with the committee stressed that they increasingly require individuals who can lead, navigate, and work in teams with others from diverse backgrounds with regard to gender, race, culture, country of origin, and academic discipline (Hart Research Associates, 2015). Moreover, while specialization is as important as ever, scientists and engineers now also need a broader general literacy to enable them to know enough outside of their specialties to appreciate how another strategy, approach, or technology can contribute to solving the problem on which they are working (NRC, 2009, 2014).
Multiple reports have described a current and future world in which societal pressures present scientific and technological challenges that require multi- and transdisciplinary approaches to problem solving. Again, the graduate STEM education enterprise will need to adapt to prepare students to contribute meaningfully to the solutions to those challenges. For example, a 2009 National Research Council report articulates a future world “where there is abundant, healthful food for everyone; where the environment is resilient and flourishing; where there is sustainable, clean energy; and where good health is the norm” (NRC, 2009, p. 9). This report describes these goals as daunting, interconnected challenges that cannot be achieved independently of the others at a time when population growth threatens to outstrip food and energy production, and environmental degradation due to agricultural practices, climate change, and unsustainable manufacturing practices is accelerating (NRC, 2009).
Similarly, a report from the American Academy of Arts and Sciences (2013) argues that these “formidable, urgent, and interconnected societal challenges” present levels of complexity that will require teams of researchers to utilize approaches from the physical sciences, engineering, information sciences, environmental sciences, and social sciences together with an evermore sophisticated understanding of the underlying biology. These reports and others describe in detail how convergent thinking by integrative and collaborative research teams could effectively address these challenges if provided with new educational and training paradigms at the graduate level.
Looking at yet another trend in STEM activity, the 21st-century STEM enterprise is experiencing an explosion of what is called “big data,” a term used to describe the emergence of very large datasets that aggregate information from many studies and/or many individuals. This presents a critical, emerging need for analytical training and tools that can enable researchers to integrate and manage those datasets to transform information into knowledge. Some 90 percent of the data in the world today was created in the past 2 years (Hale, 2017), and estimates place the growth of data at 40 percent a year, which if correct would mean the digital universe will comprise 44 zettabytes, or 44 trillion gigabytes, of data by 2020 (NASEM, 2017b; Turner, 2014) (Figure 3-1).
Big data and team science considerations aside, as the rate of scientific and technical advances increases, these advances affect people’s everyday lives more than ever. However, they also lead to real concerns about privacy, data
quality, and technology-driven social and workforce changes. Consequently, it is more important than ever that scientists and engineers consider the societal impacts of science and technology and what they can do to steward responsible discovery and innovation (NASEM, 2017a; NRC, 2011). Today, all STEM graduate students supported by NSF and NIH training grants must take courses on the responsible conduct of research—the “microethics” of authorship rules, research misconduct, and publishing norms, among others—but few graduate programs teach or discuss the “macroethics” of scientific and technological impacts on society (Herkert, 2004). Too few graduate students have opportunities to grapple with the big questions at the intersection of science, technology, and ethics that frequently appear in the news, such as climate change or gene editing (Interacademy Partnership, 2016). Advances in the conduct of STEM research call for graduates who have had the opportunity to learn both foundational and state-of-the-art methods and research skills that reflect the ways in which STEM fields carry out their work, and to develop an understanding of the societal and ethical issues that accompany advances in science and technology. The adoption of evidence-based teaching practices and the regular evaluation and assessment of curricula can help administrators and faculty measure the degree to which the program fulfills these objectives.
One consistent message that speakers from outside of academia made to the committee was that new employees with graduate degrees are well trained regarding their ability to do research. The curricula of graduate education center on building disciplinary knowledge, but there are other professional skills that students can develop during their education based on individual goals, such as appropriate teaching techniques for future faculty or management approaches for
positions in industry, government, or nonprofits. One study, conducted in 2017 by the consulting firm PwC for the Business-Higher Education Forum, found that employer demand for students with data science and analytical skills, in addition to their other training, is triple that of the supply of such students (PwC and Business-Higher Education Forum, 2017). The speakers from outside of academia also noted the need for graduates who have broad literacy across STEM fields and the humanities to enable the convergent, interdisciplinary, and team-based research that is needed to solve increasingly complex research problems.
Given the complex nature of interdisciplinary problems, institutions face challenges establishing and sustaining programs across disciplines. In a paper commissioned by the committee, Jennifer Lebrón notes: “There are numerous organizational barriers that interdisciplinary research within institutions including tenure processes which value disciplinary contributions, faculty reward structures which are tied to departments, and financial systems of institutions that discourage faculty from crossing disciplines for collaborative research or teaching” (Lebrón, 2017). The paper goes on to review lessons learned from awards made by the Institute of Education Sciences (IES) Predoctoral Interdisciplinary Research Training Program that was designed “to train a new generation of education researchers to carry out methodologically rigorous research that is relevant and accessible to education practitioners and policymakers.” Since IES established the program in 2004, there have been three iterations of multiyear funding to 20 institutions. These institutions developed training programs crossing several disciplines (e.g., economics, education, psychology, public policy, and statistics) and supported 600 doctoral students who have graduated within a traditional academic discipline and earned an Education Sciences Certificate.
Current and previous participants noted that the structure of the IES program, which includes interdisciplinary lectures, provided them with exposure to new subject matter and helped them gain understanding of different disciplines. Internships, which could be hosted at a variety of institutions (e.g., policy organizations, K-12 classrooms, or other independent research centers), allowed students to understand the value of research in practice and to explore career pathways, including the professional skills needed to apply and interview for positions outside of academia. Upon completion of the program, however, alumni integrated their roles as interdisciplinary scholars to mixed effect. For example, alumni who sought positions in policy and research reported that they were able to apply their interdisciplinary training by conceptualizing problems across disciplines and through their ability to adapt discipline-specific language.
From discussions with IES program directors, it remains unclear the degree to which the awards have shifted any of the underlying structures of their institutions, which can limit the ability for faculty to sustain an interdisciplinary program. The resulting challenges can range from logistical, such as scheduling and course offerings; to social and identity-based (language used and expectations set by different departments and disciplines); to financial (sustained fund-
ing for the program at the conclusion for the award). Additionally, the program directors raised concerns about the ability to recruit students from historically underrepresented groups into the program and hoped that they could work with the institution to create more inclusive admission policies, thus enlarging the pool of students eligible to apply for the fellowship.
The IES program highlights the nature of the challenges that face the future of higher education. Traditional departmental structure of the university has historically provided researchers and students the ability to develop deep disciplinary knowledge within a structured network. Going forward, the U.S. graduate education system needs to also reflect well the changing nature of scientific work, such as the increasing emphasis on team and multidisciplinary22 science, as well as the changing needs of an increasingly diverse student body with goals and aspirations that differ from those of many who currently compose the professoriate.
RECOMMENDATION 3.6—A Dynamic Graduate STEM Education System: The STEM education system should develop the capabilities to adjust dynamically to continuing changes in the nature of science and engineering activity and of STEM careers. This includes mechanisms to detect and anticipate such changes, experiment with innovative approaches, implement appropriate educational methods, and support institutional mechanisms on a larger scale.
- Faculty and graduate departments and programs should periodically review and modify curricula, dissertation requirements, and capstone projects to ensure timeliness and alignment with the ways relevant work is conducted, and to provide students with opportunities to work in teams that promote multidisciplinary learning.
- Professional societies and nonprofit organizations should convene and lead discussions with graduate programs, employers, and other stakeholders and disseminate innovative approaches.
- Federal and state funding agencies, professional societies, and private foundations that support or conduct education research should support studies on how different STEM disciplines can integrate the changing scientific enterprise into graduate education programs and curricula.
- Graduate students should learn how to apply their expertise in a variety of professional contexts and seek guidance from faculty, research mentors, and advisors on strategies to gain work-related experience while enrolled in graduate school.
22 Multidisciplinarity juxtaposes two or more disciplines focused on a question, problem, topic, or theme. Interdisciplinarity integrates information, data, methods, tools, concepts, and/or theories from two or more disciplines focused on a complex question, problem, topic, or theme (NRC, 2014).
Physical, mental, and emotional well-being are critical for students to develop and perform at their highest level (Graduate Assembly, 2014). For many students, graduate school is a positive experience. However, there is an expanding body of research suggesting that today’s students overall are more stressed in ways that are qualitatively different from those of previous generations of graduate students (Levecque et al., 2017; Pain, 2016, 2017; Tsai and Muindi, 2016), and it is likely that this stress impairs their ability to learn and optimally contribute to their chosen discipline. In addition, the number of graduate students reporting mental health disorders has been rising over the past several years (Garcia-Williams et al., 2014; Kemsley, 2017; Levecque et al., 2017).
According to input from current and recent graduate students, one frequent source of student stress and anxiety stems from the longstanding “power differential” that exists between students and advisors that can reduce a student’s ability to comfortably advocate for him- or herself. Investigators have noted that students’ and advisors’ goals can conflict and that students can be at a considerable disadvantage when conflicts arise, largely because students have no formal power (Chesler and Franklin, 1968; Chesler and Lohman, 1971; Miles, 1967). There have, however, been some recent advances in responding to the negative impacts of power imbalances at universities, including in some graduate programs. For example, several U.S. and Canadian universities have recognized and addressed the power imbalances directly by providing classes about mastering relationships with power imbalances, outlets for voicing concerns, counseling services, and third-party mentors to graduate students (Kim et al., 1998).
Another factor that appears to affect the graduate school experience negatively is stagnant or declining federal research budgets and how they might affect career aspirations. Many authors have commented on the growing hypercompetitive nature of the research environment today (Alberts et al., 2014; Cyranoski et al., 2011; Kimble et al., 2015). In most fields of research, access to funding is limited, variable, and uncertain, creating an important source of stress for graduate students both in its own right and through observing the kinds of stress that their faculty advisors also may be experiencing.
Graduate education can be isolating, too. This is particularly true for students from groups traditionally underrepresented in STEM if they are also underrepresented in their graduate programs and for international students who are dealing with a new culture and language. This isolation can contribute to outsized prevalence of mental health issues in Ph.D. students compared to the highly educated population in general (Graduate Assembly, 2014; Pain, 2016, 2017; Patel, 2016; Tsai and Muindi, 2016). High-pressure environments, cloudy career prospects, an imbalance of work and life, and leadership style of one’s advisor also contribute to health problems or unhealthy mental status of graduate students.
Another issue related to campus climate is the ways in which institutions address sexual harassment. Reports from graduate students of sexual harassment
by faculty or other graduate students have increased, and there have been increasing numbers of high-profile sexual harassment cases in STEM in recent years. Related to the power differential discussed above, when students feel powerless, they are less likely to seek help in the case of harassment. In 2016, the National Academies of Sciences, Engineering, and Medicine Committee on Women in Science, Engineering, and Medicine began a 2-year consensus study on the influence of sexual harassment in academia on the career advancement of women in the scientific, technical, and medical workforce. The scope of the project includes a review of the research on the extent to which women in the fields of science, engineering, and medicine are victimized by sexual harassment in academic settings; an examination of existing information on the extent to which sexual harassment in academia negatively impacts the recruitment, retention, and advancement of women’s careers; and the identification and analysis of policies, strategies, and practices that have been the most successful in preventing and addressing sexual harassment in these settings.23
There are steps that universities and students can take to ameliorate student stress and anxiety. Better policies at the graduate program level that help both students and faculty address issues such as the need for parental leave, financial support, unconscious bias training, and harassment training could improve the graduate student experience and increase retention of students from all backgrounds. So, too, would consistent and transparent training environments that enable students to clearly understand exactly what their graduate program entails and the requirements for success.
Students, with the help of their institutions, could form and join supportive communities with their peers across a university’s graduate education program. Doing so would enable students to both broaden their perspectives about other disciplines and career options and develop networks that can serve them well throughout graduate school and in their future careers. Graduate programs could help with these student-driven efforts by encouraging students to engage in activities and experiences outside of the laboratory with fellow graduate students from within and outside of their departments. Research has shown that such group-based activities can reduce student isolation and improve student success (Fenning, 2004; Wisker et al., 2007).
RECOMMENDATION 3.7—Stronger Support for Graduate Student Mental Health Services: Institutions should provide resources to help students manage the stresses and pressures of graduate education and maximize their success. Institutions of higher education should work with their faculty to recognize and
23 See http://sites.nationalacademies.org/shstudy/index.htm (this URL reflects the most current website for the National Academies report on the Sexual Harassment of Women: Climate, Culture, and Consequences in Academic Sciences, Engineering, and Medicine updated after the public release on June 12, 2018).
ameliorate behaviors that exacerbate existing power differentials and create unnecessary stress for graduate students. Toward that end:
- Institutions should administer periodic climate surveys of graduate students at the departmental level to assess their well-being in the aggregate and make adjustments when problems are identified.
- Institutions should take extra steps to provide and advertise accessible mental health services, such as those already available to veterans and most undergraduate students, at no cost to graduate students.
- Institutions should develop clear policies and reporting procedures for instances of sexual harassment and bullying.
- Graduate programs should fully incorporate awareness of mental health issues into the training experience for both students and faculty and should assess services to ensure that they are meeting the needs of graduate students.
- Faculty should be regularly informed on how to support and engage with students requiring or seeking mental health services.
- Graduate programs should encourage students to engage as a group in activities and experiences outside of traditional academic settings as a means of increasing feelings of inclusion and normalizing feelings associated with negative phenomena, such as imposter syndrome, that can reduce productivity and success in the training experience and extend time to degree.
- Graduate programs should allow students to have an active and collaborative voice to proactively engage in practices that support holistic research training and diverse career outcomes and that allow students to provide feedback on their experiences.
Alberts, B., M. W. Kirschner, S. Tilghman, and H. Varmus. 2014. Rescuing U.S. biomedical research from its systemic flaws. Proceedings of the National Academy of Sciences of the United States of America 111(16):5773-5777.
American Academy of Arts and Sciences. 2013. Arise 2: Unleashing America’s research and innovation enterprise. Cambridge, MA: American Academy of Arts and Sciences.
Blank, R., R. J. Daniels, G. Gilliland, A. Gutmann, S. Hawgood, F. A. Hrabowski, M. E. Pollack, V. Price, L. R. Reif, and M. S. Schlissel. 2017. A new data effort to inform career choices in biomedicine. Science 358(6369):1388-1389.
Blume-Kohout, M. 2017. On what basis? Seeking effective practices in graduate STEM education. Available: http://sites.nationalacademies.org/cs/groups/pgasite/documents/webpage/pga_186176.pdf (accessed May 7, 2018).
Brownell, S. E., and K. D. Tanner. 2012. Barriers to faculty pedagogical change: Lack of training, time, incentives, and … tensions with professional identity? CBE—Life Sciences Education 11(4):339-346.
Carver, S., J. Van Sickle, J. Holcomb, C. Quinn, D. Jackson, A. Resnick, S. Duffy, N. Sridhar, and A. Marquard. 2017. Operation STEM: Increasing success and improving retention among first-generation and underrepresented minority students in STEM. Journal of STEM Education 18(3):30-39.
Chesler, M. A., and J. Franklin. 1968. Interracial and intergenerational conflict in secondary schools. Paper presented at Annual Meeting of the American Sociological Association, Boston, MA.
Chesler, M. A., and J. E. Lohman. 1971. Changing schools through student advocacy. In Organization development in schools, edited by R. Schmuck and M. Miles. Palo Alto, CA: National Press Books, pp. 185-212.
Clewell, B. C., C. C. de Cohen, L. Tsui, L. Forcier, E. Gao, N. Young, N. Deterding, and C. West. 2005. Final report on the evaluation of the National Science Foundation Louis Stokes Alliances for Minority Participation program. Washington, DC: Urban Institute.
Covington, M., T. Chavis, and A. Perry. 2017. A scholar-practitioner perspective to promoting minority success in STEM. Journal for Multicultural Education 11(2):149-159.
Cyranoski, D., N. Gilbert, H. Ledford, A. Nayar, and M. Yahia. 2011. Education: The PhD factory. Nature 472(7343):276-279.
Davis, S. N., D. Mahatmya, P. W. Garner, and R. M. Jones. 2015. Mentoring undergraduate scholars: A pathway to interdisciplinary research? Mentoring & Tutoring: Partnership in Learning 23(5):427-440.
Disis, M. L., and J. T. Slattery. 2010. The road we must take: Multidisciplinary team science. Science Translational Medicine 2(22):22cm9.
Drake, J. K. 2008. Recognition and reward for academic advising in theory and practice. In Academic advising: A comprehensive handbook. 2nd ed., edited by V. N. Gordon, W. R. Habley, and T. J. Grites. San Francisco: Jossey-Bass, pp. 396-412.
Educational Testing Service. 2017. GRE guide to the use of scores, 2017-2018. Princeton, NJ: Educational Testing Service.
Einaudi, P. 2011. Two decades of increasing diversity more than doubled the number of minority graduate students in science and engineering. InfoBrief. NSF 11-319. Arlington, VA: National Science Foundation.
Fenning, K. 2004. Cohort based learning: Application to learning organizations and student academic success. College Quarterly 7(1). Available: http://collegequarterly.ca/2004-vol07-num01winter/fenning.html (accessed March 16, 2018).
Ferrini-Mundy, J., 2013. Driven by diversity. Science 340(6130):278.
Filetti, J. S. 2009. Assessing service in faculty reviews: Mentoring faculty and developing transparency. Mentoring & Tutoring: Partnership in Learning 17(4):343-352.
Flaherty, C. 2017. AAU sets expectation for data transparency on Ph.D. program outcomes. Inside Higher Ed. Available: https://www.insidehighered.com/quicktakes/2017/09/20/aau-sets-expectation-data-transparency-phd-program-outcomes (accessed January 22, 2018).
Fleming, M., M. S. House, M. V. Shewakramani, L. Yu, J. Garbutt, R. McGee, K. Kroenke, M. Z. Abedin, and D. M. Rubio. 2013. The mentoring competency assessment: Validation of a new instrument to evaluate skills of research mentors. Academic Medicine 88(7):1002.
Fountain, J., and K. E. Newcomer. 2016. Developing and sustaining effective faculty mentoring programs. Journal of Public Affairs Education 22(4):483-506.
Freeman, S., S. L. Eddy, M. McDonough, M. K. Smith, N. Okoroafor, H. Jordt, and M. P. Wenderoth. 2014. Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences of the United States of America 111(23):8410-8415.
Garcia-Williams, A. G., L. Moffitt, and N. J. Kaslow. 2014. Mental health and suicidal behavior among graduate students. Academic Psychiatry 38(5):554-560.
Gibbs, K., Jr. 2014. Diversity in STEM: What it is and why it matters. Scientific American. Voices blog. Available https://blogs.scientificamerican.com/voices/diversity-in-stem-what-it-is-and-why-it-matters/ (accessed March 16, 2018).
Godwin, A., G. Potvin, Z. Hazari, and R. Lock. 2016. Identity, critical agency, and engineering: An affective model for predicting engineering as a career choice. Journal of Engineering Education 105(2):312-340.
Graduate Assembly. 2014. Graduate student happiness & well-being. Berkeley: University of California.
Griffin, K. A., D. Pérez, A. P. E. Holmes, and C. E. P. Mayo. 2010. Investing in the future: The importance of faculty mentoring in the development of students of color in STEM. New Directions for Institutional Research 2010(148):95-103.
Habley, W. R. 2007. Putting students first in the campus community: Pathway five. In Fostering student success in the campus community, edited by G. L. Kramer. San Francisco: Jossey-Bass, pp. 407-431.
Hale, T. 2017. How much data does the world generate every minute? IFLScience. Available: http://www.iflscience.com/technology/how-much-data-does-the-world-generate-every-minute/ (accessed January 22, 2018).
Hall, A., J. Mann, and M. Bender. 2015. Analysis of scholar outcomes for the NIGMS Postbaccalaureate Research Education program. Available: https://www.nigms.nih.gov/News/reports/Documents/PREP-outcomes-report.pdf (accessed March 17, 2018).
Hall, A. K., A. Miklos, A. Oh, and S. D. Gaillard. 2016. Educational outcomes from the Maximizing Access to Research Careers Undergraduate Student Training in Academic Research (MARC U-STAR) program. Available: https://www.nigms.nih.gov/News/reports/Documents/MARCpaper031416.pdf (accessed March 17, 2018).
Hall, J. D., A. B. O’Connell, and J. G. Cook. 2017. Predictors of student productivity in biomedical graduate school applications. PloS ONE 12(1):e0169121. Available: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0169121 (accessed March 17, 2018).
Hart Research Associates. 2015. Falling short? College learning and career success. Prepared for the Association of American Colleges and Universities. Available: https://www.aacu.org/sites/default/files/files/LEAP/2015employerstudentsurvey.pdf (accessed March 17, 2018).
Herkert, J. 2004. Microethics, macroethics, and professional engineering societies. In Emerging technologies and ethical issues in engineering: Papers from a workshop. Washington, DC: The National Academies Press, pp. 107-114.
Higgins, M. C. 2000. The more, the merrier? Multiple developmental relationships and work satisfaction. Journal of Management Development 19(4):277-296.
Hussain, Y. 2017. Key Recommendations from Selected Recent Reports on Graduate Education (1995-2017). Commissioned paper prepared for the Committee. Available: http://sites.nationalacademies.org/cs/groups/pgasite/documents/webpage/pga_186162.pdf (accessed May 7, 2018).
Interacademy Partnership. 2016. Doing global science: A guide to responsible conduct in the global research enterprise. Princeton, NJ: Princeton University Press.
Kemsley, J. 2017. Grappling with graduate student mental health and suicide. Chemical & Engineering News 95(32):28-33.
Kim, S. H., R. H. Smith, and N. L. Brigham. 1998. Effects of power imbalance and the presence of third parties on reactions to harm: Upward and downward revenge. Personality and Social Psychology Bulletin 24(4):353-361.
Kimble, J., W. M. Bement, Q. Chang, B. L. Cox, N. R. Drinkwater, R. L. Gourse, A. A. Hoskins, A. Huttenlocher, P. K. Kreeger, P. F. Lambert, M. R. Mailick, S. Miyamoto, R. L. Moss, K. M. O’Connor-Giles, A. Roopra, K. Saha, and H. S. Seidel. 2015. Strategies from UW-Madison for rescuing biomedical research in the US. Elife 4:e09305.
Law, A. V., M. M. Bottenberg, A. H. Brozick, J. D. Currie, M. V. DiVall, S. T. Haines, C. Jolowsky, C. P. Koh-Knox, G. A. Leonard, S. J. Phelps, D. Rao, A. Webster, and E. Yablonski. 2014. A checklist for the development of faculty mentorship programs. American Journal of Pharmaceutical Education 78(5):98.
Layton, R. L., P. D. Brandt, A. M. Freeman, J. R. Harrell, J. D. Hall, and M. Sinche. 2016. Diversity exiting the academy: Influential factors for the career choice of well-represented and underrepresented minority scientists. CBE—Life Sciences Education 15(3), doi:10.1187/cbe.16-01-0066.
Lebrón, J. 2017. Forming Interdisciplinary Scholars: An Evaluation of the IES Predoctoral Interdisciplinary Training Program. Commissioned paper prepared for the Committee. Available: http://sites.nationalacademies.org/cs/.
Levecque, K., F. Anseel, A. De Beuckelaer, J. Van der Heyden, and L. Gisle. 2017. Work organization and mental health problems in PhD students. Research Policy 46(4):868-879.
Lewis, V., C. A. Martina, M. P. McDermott, P. M. Trief, S. R. Goodman, G. D. Morse, J. G. LaGuardia, D. Sharp, and R. M. Ryan. 2016. A randomized controlled trial of mentoring interventions for underrepresented minorities. Academic Medicine 91(7):994-1001.
Lovitts, B. E. 2004. Research on the structure and process of graduate education: Retaining students. In Paths to the professorship: Strategies for enriching the preparation of future faculty, edited by D. H. Wulff and A. E. Austin. San Francisco: Josey-Bass, pp. 115-136.
Marcellino, P. A. 2011. Fostering sustainability: A case study of a pilot mentoring program at a private university. Mentoring & Tutoring: Partnership in Learning 19(4):441-464.
Maton, K. I., and F. A. Hrabowski III. 2004. Increasing the number of African American PhDs in the sciences and engineering: A strengths-based approach. American Psychologist 59(6):547-556.
May, W.-C. J. 2016. Characteristics of U.S. students that pursued a STEM major and factors that predicted their persistence in degree completion. Universal Journal of Educational Research 4(6):1495-1500.
Miles, M. B. 1967. Some properties of schools as social systems. In Change in school systems, edited by G. Watson. Washington, DC: Cooperative Project for Educational Development, National Training Laboratories.
Miller, C., and K. Stassun. 2014. A test that fails. Nature 510:303-304.
Moneta-Koehler, L., A. M. Brown, K. A. Petrie, B. J. Evans, and R. Chalkley. 2017. The limitations of the GRE in predicting success in biomedical graduate school. PloS ONE 12(1):e0166742.
Museus, S. D., and D. Liverman. 2010. High-performing institutions and their implications for studying underrepresented minority students in STEM. New Directions for Institutional Research 2010(148):17-27.
NAE (National Academy of Engineering). 2014. Advancing diversity in the US industrial science and engineering workforce: Summary of a workshop. Washington, DC: The National Academies Press.
NAS/NAE/IOM (National Academy of Sciences, National Academy of Engineering, and Institute of Medicine). 1996. Careers in science and engineering: A student planning guide to grad school and beyond. Washington, DC: National Academy Press.
NAS/NAE/IOM. 1997. Adviser, teacher, role model, friend: On being a mentor to students in science and engineering. Washington, DC: National Academy Press.
NAS/NAE/IOM. 2011. Expanding underrepresented minority participation: America’s science and technology talent at the crossroads. Washington, DC: The National Academies Press.
NASEM (National Academies of Sciences, Engineering, and Medicine). 2017a. Fostering integrity in research. Washington, DC: The National Academies Press.
NASEM. 2017b. Refining the concept of scientific inference when working with big data: Proceedings of a workshop. Washington, DC: The National Academies Press.
NRC (National Research Council). 2009. A new biology for the 21st century. Washington, DC: The National Academies Press.
NRC. 2011. Challenges and opportunities for education about dual use issues in the life sciences. Washington, DC: The National Academies Press.
NRC. 2012. Research universities and the future of America: Ten breakthrough actions vital to our nation’s prosperity and security. Washington, DC: The National Academies Press.
NRC. 2014. Convergence: Facilitating transdisciplinary integration of life sciences, physical sciences, engineering, and beyond. Washington, DC: The National Academies Press.
NRC. 2015. Enhancing the effectiveness of team science. Washington, DC: The National Academies Press.
NRC and IOM (National Research Council and Institute of Medicine). 2003. Enhancing the vitality of the National Institutes of Health: Organizational change to meet new challenges. Washington, DC: The National Academies Press.
Packard, B. W.-L. 2015. Successful STEM mentoring initiatives for underrepresented students: A research-based guide for faculty and administrators. Sterling, VA: Stylus.
Page, S. E. 2008. The difference: How the power of diversity creates better groups, firms, schools, and societies Princeton, NJ: Princeton University Press.
Pain, E. 2016. Trainees and mental health: Let’s talk. In Science Careers. Washington, DC: American Association for the Advancement of Science.
Pain, E. 2017. Ph.D. students face significant mental health challenges. Science Careers Blog. Washington, DC: American Association for the Advancement of Science.
Patel, V. 2016. Grad schools try to ease “culture problem” of anxiety and isolation. Chronicle of Higher Education 15-17.
Plume, A., and D. van Weijen. 2014. Publish or perish? The rise of the fractional author. Research Trends 38(3):16-18.
Polka, J. K., K. A. Krukenberg, and G. S. McDowell. 2015. A call for transparency in tracking student and postdoc career outcomes. Molecular Biology of the Cell 26(8):1413-1415.
Posselt, J. R. 2016. Inside graduate admissions merit, diversity, and faculty gatekeeping. Cambridge, MA: Harvard University Press.
Purdue University. 2015. Trustees approve new guidelines for promotion and tenure to highlight mentoring. News Release. Available: http://www.purdue.edu/newsroom/releases/2015/Q4/trustees-change-purdue-polytechnic-department-name-to-reflect-enhancements.html (accessed January 22, 2018).
PwC and Business-Higher Education Forum. 2017. Investing in America’s data science and analytics talent: The case for action. Available: https://www.pwc.com/us/en/publications/assets/investingin-america-s-dsa-talent-bhef-and-pwc.pdf (accessed March 17, 2018).
Reagans, R., and E. W. Zuckerman. 2001. Networks, diversity, and productivity: The social capital of corporate R&D teams. Organization Science 12(4):502-517.
Rincon, B. E., and C. E. George-Jackson. 2016. STEM intervention programs: Funding practices and challenges. Studies in Higher Education 41(3):429-444.
Roberge, M. É., and R. Van Dick. 2010. Recognizing the benefits of diversity: When and how does diversity increase group performance? Human Resource Management Review 20(4):295-308.
Rodriguez, C., R. Kirshstein, C. Tanenbaum, C. Storey, T. Chan, L. Turk-Bikakci, S. Francois, R. Miller, M. Cillier, and A. Alsbrooks. 2011. National evaluation of the alliances for graduate education and the professoriate. Washington, DC: American Institutes for Research.
Saxena, A. 2014. Workforce diversity: A key to improve productivity. Procedia Economics and Finance 11(Suppl. C):76-85.
Schnoes, A. N., A. Caliendo, J. Morand, T. Dillinger, M. Naffziger-Hirsch, B. Moses, J. C. Gibeling, K. R. Yamamoto, B. Lindstaedt, R. McGee, and T. C. O’Brien. 2018. Internship experiences contribute to confident career decision making for doctoral students in the life sciences. CBE—Life Science Education 17(1). Available: https://doi.org/10.1187/cbe.17-08-0164 (accessed May 9, 2018).
Schultheis, A. S., T. M. Farrell, and E. L. Paul. 2011. Promoting undergraduate research through revising tenure and promotion policy. CURQuarterly 31(4):25-31.
Stassun, K. G., S. Sturm, K. Holley-Bockelmann, A. Burger, D. J. Ernst, and D. Webb. 2011. The Fisk-Vanderbilt Master’s-to-Ph.D. Bridge Program: Recognizing, enlisting, and cultivating unrealized or unrecognized potential in underrepresented minority students. American Journal of Physics 79(4):374-379.
Tabak, L. A., and F. S. Collins. 2011. Weaving a richer tapestry in biomedical science. Science 333(6045):940-941.
Tanner, K. D. 2013. Structure matters: Twenty-one teaching strategies to promote student engagement and cultivate classroom equity. CBE—Life Sciences Education 12(3):322-331.
Tsai, J. W., and F. Muindi. 2016. Towards sustaining a culture of mental health and wellness for trainees in the biosciences. Nature Biotechnology 34(3):353-355.
University of Arkansas at Little Rock. 2011. Faculty roles and rewards, I: Tenure track. Policy 403.20. Available: http://ualr.edu/policy/home/facstaff/faculty-roles-and-rewards-i/ (accessed January 22, 2018).
U.S. Census Bureau. 2012. Most children younger than age 1 are minorities. Census Bureau Reports (news release). Washington, DC: U.S. Census Bureau.
Valantine, H. A., P. K. Lund, and A. E. Gammie. 2016. From the NIH: A systems approach to increasing the diversity of the biomedical research workforce. CBE—Life Sciences Education 15(3):fe4; doi:10.1187/cbe.16-03-0138.
Van Noorden, R. 2015. Interdisciplinary research by the numbers. Nature News 525(7569):306-307.
Vilorio, D. 2014. STEM 101: Intro to tomorrow’s jobs. Occupational Outlook Quarterly (Spring). Available: https://www.bls.gov/careeroutlook/2014/spring/art01.pdf (accessed March 20, 2018).
Wisker, G., G. Robinson, and M. Shacham. 2007. Postgraduate research success: Communities of practice involving cohorts, guardian supervisors and online communities. Innovations in Education and Teaching International 44(3):301-320.
Wuchty, S., B. F. Jones, and B. Uzzi. 2007. The increasing dominance of teams in production of knowledge. Science 316(5827):1036-1039.
Zhao, C.-M., C. M. Golde, and A. C. McCormick. 2007. More than a signature: How advisor choice and advisor behavior affect doctoral student satisfaction. Journal of Further and Higher Education 31(3):263-281.
Zolas, N., N. Goldschlag, R. Jarmin, P. Stephan, J. O. Smith, R. F. Rosen, B. M. Allen, B. A. Weinberg, and J. I. Lane. 2015. Wrapping it up in a person: Examining employment and earnings outcomes for Ph.D. recipients. Science 350(6266):1367-1371.