This chapter focuses on the studies that have been done on student learning experiences in undergraduate research experiences (UREs), whereas later chapters address the context in which the UREs happen by discussing mentorship, faculty, institutional administration, and policy issues.1 While there are many opportunities for undergraduates to engage in research, as discussed in the previous chapters, the goals and structures of these experiences vary significantly. The studies on UREs also vary greatly in their content, approach, and perspectives; in gathering information about UREs, the committee was not able to find data on all of the topics we sought. Even so we learned of many interesting and creative programs at various types of institutions around the country.
This chapter examines information from those programs that have been the subject of focused study. Some of the more recent studies have focused on course-based UREs (CUREs) specifically. In examining the evidence that has been gathered to date on the outcomes of UREs, the committee found that much of the published data are for retention in the major and graduation rates. Although this focus may be because many UREs were set up specifically to promote the participation of students in research to support their retention in science, technology, engineering, and mathematics (STEM) fields, it may also be because these data are readily available, as
1 This chapter includes outcomes from participation in course-based undergraduate research experiences (CUREs) from a paper commissioned by the committee titled Course-Based Undergraduate Research Experiences: Current Knowledge and Future Directions by Erin Dolan (2016).
they are already collected for other purposes. These data are discussed here in the section on participation and retention. Other studies are discussed in sections on increased understanding of STEM practices (e.g., content, concepts, and research skills) and integration of students into STEM culture (e.g., belonging, teamwork, ownership). Overall, the studies that document outcomes of UREs are relatively new and were developed by researchers and instructors who are early adopters of this high-impact practice; therefore, the motivation and self-selection of students included is not always articulated for each study. When possible, information about the selection of students and student motivation is highlighted for each study and discussed here. This would also include describing the comparison group when possible; the importance of proper comparison group selection and the impact on research design is discussed in detail in Chapter 7.
A wide range of outcomes have been proposed as potential benefits of UREs for students, for faculty, and for the institution as a whole. However, few of these benefits have been well documented. Clearly, with such a wide range of potential outcomes, there are different approaches to gathering evidence, not only for each type of outcome, but also with regard to the evidence of benefit that is being sought.
As discussed in Chapter 1, there are three types of evidence that might be collected to support claims of benefit for these outcomes: (1) evidence that provides a description of outcomes from UREs or suggests ways in which UREs may influence outcomes, (2) evidence that provides a causal explanation for the outcomes of UREs, and (3) evidence that supports improved understanding of the mechanisms by which UREs affect outcomes. Descriptive evidence may come from institutional or national datasets (such as the Cooperative Institutional Research Program Freshman Survey and the National Survey of Student Engagement) that have information about student enrollment and persistence; it can also be obtained from student self-reports, surveys, pre and post testing, and interviews. However, unless a careful experimental or quasi-experimental design is used, gathering this type of descriptive evidence is unlikely to provide causal evidence for any changes that are observed after participation in UREs.
Similarly, studies that attempt to show why participation in a URE might bring about a particular outcome—that is, to provide a mechanism for fostering desired outcomes—must also be carefully designed. Some insight into mechanisms may emerge from phenomenographic studies—which require URE participants to describe their experiences. Table 4-1 outlines reliable measurement tools that have been used to measure student outcomes from UREs.
TABLE 4-1 Measurement Tools for UREs
|Instrument Name||Domains Measured||Further Information|
|Experimental Design Ability Test||Students’ understanding of experimental design criteria through open-ended prompt||Differentiates between students’ scientific thinking gains in research-based vs. traditional course lab sections. Can be used in pre and post testing format; test is independent of disciplinary content (can be used in a variety of contexts) (Sirum and Humberg, 2011). Has been modified to be more sensitive for students majoring in biology (Expanded-Experimental Design Ability Test; Brownell et al., 2014).|
|Laboratory Course Assessment Survey||Students’ perceptions of 3 design features of CUREs: collaboration, discovery and relevance, and iteration||Self-report survey. The discovery and relevance and iteration scales differ for CUREs versus traditional lab courses (Corwin et al., 2015b).|
|Networking Survey||Students’ personal and professional networks through self-report of degrees of conversation||Self-report survey. Student networking related to project ownership. Survey can differentiate between research experiences with low-networking or high-networking design (Hanauer and Hatfull, 2015).|
|Project Ownership Survey||Extent of students’ project ownership within research experience||Self-report survey. Results support argument that project ownership is one design aspect of UREs that fosters increased retention. Defines five categories of project ownership (Hanaeur and Dolan, 2014).|
|Rubric for Experimental Design Knowledge and Difficulties||Knowledge of experimental design and ability to diagnose problems in research design||Assessment that can be used in UREs and CUREs in pre and post format. Examines students’ difficulties with: identifying variable properties of experimental subject, manipulation of variables, measurement of outcomes, accounting for variability, and recognizing the scope of inferences appropriate for experimental findings (Dasgupta et al., 2014).|
|Survey of Undergraduate Research Experiences (SURE)||Cognitive (understanding research process, etc.); skills; personal (confidence, temperament)||Self-report survey. Students report gains in all areas. Highest gains are in understanding research process and learning lab techniques. Personal gains rated second highest (Lopatto, 2004).|
|Instrument Name||Domains Measured||Further Information|
|URSSA Survey||Thinking and working like a scientist; personal gains; skills; attitudes and behaviors||A self-report survey. The four domains (survey constructs) are separate but related. Analysis shows that “attitudes and behaviors” items act like satisfaction items and measure similar constructs. Comparison of Likert-scale and open-ended items showed inflation in students rating themselves as more likely to go to graduate school (Weston and Laursen, 2015).|
Many studies on the outcomes of UREs have focused on outcomes of participation, retention, and persistence. Data on these outcomes are often already gathered by the institution, thereby providing a reasonably accessible entrée for faculty interested in examining the results of UREs. Obtaining information on whether URE participants continue on to graduate school or into STEM careers is more difficult to gather, though the National Student Clearinghouse does track national degree completion, and analysis of existing information could provide important insights into the effects of UREs.2
Performance and Continued Enrollment in STEM Major
One prevalent argument for UREs is that participation in a research experience improves students’ academic outcomes, such as retention in STEM majors, college completion, and grade point average (GPA) (Graham et al., 2013).
Nagda and colleagues (1998) conducted one of the few studies to randomly select applicants for research experiences, notably before UREs were widely available, to measure outcomes associated with retention. They found that for students who applied and were randomly selected for a URE program, there was a statistically significant decrease in attrition (retention in major) for those students who participated compared to those who did not, although findings varied by racial/ethnic groups. The difference in retention rate was strongest and statistically significant for African Americans. Non-Hispanic white students who had participated in research showed half the STEM attrition rate of the matched group of control
students, though the difference was not statistically significant. Moreover, Hispanic students had a slightly higher, though not statistically significant, retention rate compared with control students.
The remaining studies report outcome data on students who self-selected into UREs, and although they were matched for demographic characteristics with comparison groups who did not participate in a URE, it is not clear that the groups were matched for motivation or other characteristics that may have contributed to their success in college and in continuing in a STEM major. A recent study by Rodenbusch and colleagues (2016) examined GPA, graduation rates, and retention in STEM majors among students who chose to participate in CUREs as part of the Freshman Research Initiative at The University of Texas at Austin, which offers students up to three sequenced courses in which they engage in research at increasing levels of independence.3 The study used propensity score matching to account for selected student-level differences4 and concluded that students who participated in the full three-semester sequence were more likely to graduate with a STEM degree and more likely to graduate within 6 years. In contrast to the usual observation of greater minority attrition in STEM majors and STEM degree completion, students from historically underrepresented groups participating in this initiative succeeded at the same rate as other students; that is, they were more likely to stay in the STEM major and graduate with a STEM degree. This study found no difference in GPA between those students participating in the URE compared to those who did not. However, a study comparing research5 and nonresearch students at another university showed that extended participation in research for more than a semester was associated with an increase in GPA, even after controlling for SAT scores, though this GPA gain was not evident in students with a single semester of research experience (Fechheimer et al., 2011).
UREs may also contribute to subsequent course-taking patterns in STEM. After controlling for background characteristics such as early college coursework, GPA, math SAT scores, gender, and minority status, Junge and colleagues (2010) found that students who chose to participate in the
4 The model included 13 variables that were used to create the comparison group. These variables included: gender, race/ethnicity, parental education levels, parental income level, Pell grant eligibility, SAT total score or ACT equivalent, number of high school science credits earned, number of high school math credits earned, whether students graduated from a Texas or out-of-state high school, enrollment year at UT Austin, first semester enrolled (e.g., Fall), first college students entered at The University of Texas at Austin, and enrollment in the Texas Interdisciplinary Program.
5 The definition of undergraduate research used “invokes the traditional one-faculty-mentor-to-one-student relationship focused on a directed-research project” (Fechheimer et al., 2011, p. 157). No demographic data were provided for each group (research, nonresearch) beyond gender and SAT scores.
Summer Undergraduate Research at Emory [University] program took significantly more science courses and earned higher grades in those courses than nonparticipants. There is some evidence that student participation in UREs correlates with a shorter time to degree. Based on student transcript data at a single site, 98.5 percent of undergraduates in a summer research program graduated within 5 years, compared to the overall graduate rate of 82 percent (Craney et al., 2011).
Deek and colleagues (2003) conducted a study to examine research experiences as a factor in academic achievement. The study compared 39 students who participated in a one-semester engineering Research Experiences for Undergraduates program with 230 students who did not; the two groups were matched on demographics and academic performance prior to the research semester. Comparisons between the groups on retention, cumulative GPA, and ratio of earned and attempted credit hours showed a statistically significant difference between the groups. Overall, students who participated in research had higher grades, earned more credits relative to attempted credits, and were more likely to persist in the program after completing the URE. Moreover, analysis of survey responses from both faculty and students found that the program increased students’ motivation and interest toward research.
Studies Focusing on Historically Underrepresented Students
Other studies have documented the educational and career benefits of apprentice-style UREs for historically underrepresented students in particular. In recent years, a variety of research programs, using quasi-experimental designs and statistical modeling, have started to show consistent evidence that research experience correlates with higher likelihood of degree completion and persistence in interest in STEM careers (Chemers et al., 2011; Jones et al., 2010; Schultz et al., 2011). For example, TheScienceStudy6 tracked a cohort of 1,400 historically underrepresented students who were participating in the Research Initiative for Scientific Enhancement (RISE) program, an initiative funded by the National Institutes of Health to increase the participation of students from underrepresented populations in the biomedical sciences. Schultz and colleagues (2011) found that students with science research experiences (e.g., in classes, working independently with a faculty member, or at a job) who reported no active enrollment in a co-curricular science program retained interest in science careers more strongly than those who did not engage in research but were enrolled in
6TheScienceStudy is a nationwide longitudinal study of the academic and professional experiences of students and professionals. It was sponsored by the National Institutes of Health. See https://ssl1.csusm.edu/thesciencestudy [September 2016].
a science program that did not include any hands-on research experience. All students who were a part of this study had reported high intention to pursue a biomedical career when the study began. Interestingly, participation in the RISE undergraduate research program did not increase the career interest of these already interested students. Rather, it appeared to buffer students from losing interest. The “match” students were not enrolled in any undergraduate research programs but at the beginning of the study shared a similar interest in the biomedical sciences with the RISE students. The study did not report whether the “match” students had similar access to UREs.
Chang and colleagues (2014) found that participation in UREs by students who were from groups historically underrepresented in STEM and who entered college with high grades and aspirations moderated the negative correlation between being an underrepresented minority and persistence. “Five college experiences significantly predicted the likelihood of historically URM students [underrepresented minority students] following through on their freshman intentions to major in STEM. The strongest of these predictors was participation in an undergraduate research program. URM students who participated in programs that exposed them to research were 17.4 percentage points more likely to persist in STEM than those who did not” (Chang et al., 2014, p. 567).
A comprehensive analysis of the transcripts and admissions applications of 7,664 University of California, Davis students who declared biology as a major between 1995 and 1999 found that underrepresented minority students who participated in a research experience, especially during their first 2 years, were more likely to have high academic performance and persist in biology, as well as go on to graduate, than those who did not (Jones et al., 2010). More specifically, for Hispanic and African American students, those students who participated in a URE were more likely to obtain a biology degree than those students who did not participate in research. Similarly, research by Villarejo and colleagues found that participants in the Biology Undergraduate Scholars Program, an undergraduate research enrichment program for underrepresented students in biology, were more likely than other students to persist to graduation with a biology major (Barlow and Villarejo, 2004; Villarejo and Barlow, 2007). Their analyses suggested that research participation contributed to persistence.
Together, these studies were almost all of highly motivated students who do not lose their motivation when they participate in UREs. Although these studies do not describe how to build motivation, they do describe how to sustain it.
The Meyerhoff Scholars program, a long-standing, comprehensive program to provide academic and social support to increase the retention of underrepresented minority STEM students at University of Maryland-
Baltimore County, has collected nearly 20 years of outcomes data on participants.7 As part of the program, all students are required to participate in on-campus, academic-year research. However, the structure and intensity of that research can vary. For example, some students only participate in a yearly undergraduate research symposium, some students complete research courses for academic credit, and others participate in the Minority Access to Research Careers (MARC)8 Undergraduate Student Training in Academic Research (U-STAR) program. The MARC U-STAR program is designed for students who intend to pursue a Ph.D. in biomedical research. Carter and colleagues (2009) examined the educational outcomes of 13 cohorts of students in relation to the structure (annual symposium, course, or MARC U-STAR program) and the intensity (symposium, two semesters, or more than two semesters) of the students’ on-campus, academic-year research experiences. They found that those students who participated in the more structured and/or intense experiences (i.e., participation in MARC U-STAR, more than three semesters of research courses, or both) were significantly more likely to enroll in a STEM Ph.D. program after graduation than students who did not participate in such research experiences. As noted by Carter and colleagues, participants in this program are from a highly select group and thus the findings may not be generalizable.
Graduate School and Future Career Choice
As detailed below, URE alumni have reported that the experience allowed them to test their fit with the profession; develop a close relationship with a faculty member; and gain insight into the social, cultural, and intellectual processes of science. Socialization into the professional STEM community might also help to shape students’ future interests and goals (Corwin et al., 2015a; Litzinger et al., 2011). A few researchers have explored the processes by which UREs may shape students’ career or educational decisions; some of these studies are described below.
Mastery of research skills might have a predictive effect on students’ efficacy beliefs, which in turn can be predictive of their graduate school aspirations (Adedokun et al., 2013). Particular student characteristics or aptitudes, such as curiosity about the unknown, a desire for autonomy and independence, and openness to the unknown in their career path, may be predictive of research students’ pursuit of a Ph.D. (McGee and Keller, 2007), although these same traits may be what motivates some students to seek out UREs. Experiences that take place during UREs, such as developing a close relationship with a faculty member, have helped students to
confirm that graduate school was the correct path and have clarified their field of interest for their graduate program (Laursen et al., 2010). These types of benefits were also documented from other types of STEM professional work under the guidance of a mentor, such as internships or co-ops, yet only research experiences helped students to clarify whether a research career or pursuing a Ph.D. was the correct path for them, as indicated through structured interviews (Thiry et al., 2011).
Many studies have relied on the assertions of current URE students about the influence of the research experience on their future career and educational plans (Adedokun et al., 2012; Grimberg et al., 2008). Several of these studies have used comparisons with students without research experiences (“nonresearch students”), but it is often not clear whether students’ career or educational goals differed prior to their research experience. For instance, Eagan and colleagues (2013) compared demographically matched groups of research and nonresearch students and found that students who had participated in apprentice-style UREs had stronger graduate school aspirations, but these differences may have existed prior to the experience. Likewise, a study of student researchers and nonresearchers reported that the former group felt that research increased their awareness of what graduate school is like and increased their aspirations for a Ph.D. degree, yet only 19 percent of students had a “new” expectation of receiving a Ph.D., which may reflect the selection bias inherent in students who are chosen for research experiences (Russell et al., 2007). Another study comparing apprentice-style student researchers and nonresearchers found that the research students held high expectations prior to their research experience that remained unchanged with respect to the value of research for facilitating their future career path (Craney et al., 2011). Other studies have reported that apprentice-style URE students felt that the experience prepared them for graduate school and STEM careers (Hunter et al., 2007; Sabitini, 1997; Seymour et al., 2004). Some studies found that students often enter UREs because they are interested in learning more about research or determining whether graduate school might be the right path for them. The results obtained show that research experiences played an important role in confirming or clarifying prior goals for these students (Gonzalez-Espada and LaDue, 2006; Hunter et al., 2007; Lopatto, 2004; Pacific and Thompson, 2011; Seymour et al., 2004).
Other studies have tracked URE participants’ postbaccalaureate outcomes through retrospective accounts from apprentice-style URE alumni. Zydney and colleagues (2002) compared retrospective accounts of research and nonresearch alumni at a single institution and found that research students were more likely to go to graduate school and more likely to cite a faculty member as influential in their career choice. Alumni of a biosciences research program at Emory University were three times more likely to pursue a Ph.D. than nonresearch students (Junge et al., 2010). Likewise, students
who conducted research in a formal research program had higher rates of graduate school attendance than students with individually reported research experiences that were not part of a formal program, suggesting that the professional development offered through programs can have an impact beyond the research experience itself (Bauer and Bennett, 2003). Taking these studies together, their retrospective approach does not control for initial motivation and interests. Those who initially selected UREs may have had greater interest in research careers. Thus, these findings may imply a correlational relationship that is not causal.
One of the few studies of UREs to use random assignment of students to research positions documented that URE participants were more likely to enroll in graduate school compared to nonresearch students who had applied for a research position and were not randomly selected (Hathaway et al., 2002). In fact, 82 percent of all research students enrolled in graduate degree programs, while only 65 percent of nonresearch students did so, although participation in UREs may have made the research students more competitive in their graduate school applications. This effect was even more pronounced for students of color, as underrepresented minority research students attended graduate school at rates similar to other research students, yet only 56 percent of nonresearch students of color attended graduate school.
Summary of Findings for Increased Retention and Participation of STEM Students
Students who participate in UREs are generally more likely to remain in STEM fields as undergraduates than are STEM students who do not participate. However, most of the studies cited in reaching this conclusion were not able to address differences in initial interest or motivation prior to the URE exposure (or lack thereof). Some studies have found higher grades and graduation rates for URE participants as well. Self-report data from students suggest that UREs can confirm the students’ intention to attend graduate school in STEM and that these students perceive mentorship as an important component of the experience. Thus, supporting and maintaining student interest in a STEM major and subsequent career may be an important function of UREs.
The outcomes associated with promoting STEM disciplinary practices include what a student learns, understands, and knows regarding the state of his own knowledge of that STEM discipline through participation in a
URE. The specific content and concepts a student learns will vary, depending on the discipline or disciplines of that student’s URE. However, in addition to learning traditional disciplinary content, a student in a URE is afforded an opportunity to engage in disciplinary practices common across STEM fields, such as analyzing and interpreting data, identifying the next steps in an experiment or research activity, and identifying gaps in knowledge that are worthy of further research.
Although this report considers a wide range of fields included within STEM, many of the studies reported in the literature examined a narrow range of fields, and much of the existing literature focuses heavily on bench science, rather than, for example, mathematics. Moreover, despite the fact that many of the findings focus on science and do not consider other STEM fields, the committee decided that these studies nevertheless provide useful insight for understanding the impact of UREs.
Content and Concepts
Several studies of apprentice-style summer research have documented that students perceive that they gained content knowledge, were able to relate their research projects to the larger field of study, and understood the context of the project (Craney et al., 2011; Kardash, 2000), yet these studies were conducted at a single site, relied on self-reports, and did not use a control or comparison group. Kardash (2000) also included research mentors’ ratings of students’ knowledge gains, which were similar to students’ ratings of their own gains. In Kardash’s study, female students rated their own cognitive gains from research as lower than male students rated their gains.
Because students in CUREs are exposed to standardized course content, CUREs may lend themselves to more uniform assessment of knowledge gains. Indeed, a study of one biosciences CURE found that students in the research-based laboratory section showed statistically significantly greater pre- and post gains on disciplinary content assessments than did students in the nonresearch-based lab sections (Russell et al., 2015), although this difference occurred only in the principles of biology and cell biology sections and not the ecology sections. Drew and Tiplett (2008) demonstrated that students made substantial increases in genomics knowledge from the beginning to the end of the CURE, though this study encompassed a single course with no comparison group.
One of the few studies to randomly assign students to either a research-based or traditional lab section and to use multiple methods to measure outcomes found that students in the research-based lab section believed that they were better able to explain the concepts in the experiment and had a better understanding of the research process than students in the traditional lab section (Szteinberg and Weaver, 2013). Two studies of the
multi-institution Genomics Education Partnership (GEP) CURE found that CURE students made greater gains in content knowledge than comparison students from participating schools who completed prerequisites but had not engaged in the GEP curriculum. The content gains were larger when the faculty member devoted more class time to the CURE projects (Shaffer et al., 2010, 2014). Lopatto and colleagues (2008) found that those GEP students had larger increases in their positive attitude about research, as measured by the Survey of Undergraduate Learning Experiences, a learning survey (see Table 4-1). Shaffer and colleagues (2010) reported that GEP students also had higher scores on quizzes about the content and processes used in the course, compared to students not participating in the GEP program.
Students participating in UREs report that they gain experience with the practices and skills of conducting STEM research, such as data collection, analysis, and interpretation and understanding of research design. A comparative study of summer URE students and nonresearch students at Emory University found that the URE students felt more prepared to select appropriate data analysis strategies and apply research ethics principles than students who had not had research experience (Junge et al., 2010). In another study, URE alumni perceived greater growth in science, math, logic, and/or problem solving skills than did nonresearch alumni (Bauer and Bennett, 2003). These findings are consistent with the work of Lopatto (2004). Kardash (2000) reported that both students and faculty mentors in apprentice-style research experiences rated students’ gains in collecting and analyzing data as some of the highest rated skills developed in UREs. In addition to these perceived gains in research skills, students’ performance on exams revealed gains in the ability to analyze and interpret data—a goal of many CUREs.
In a performance-based assessment that was blindly scored, students in an introductory biology CURE were tested three times on their ability to design experiments and interpret data. Over the course of the exams, students showed significant increases in their ability to analyze and interpret data and describe their results (Brownell et al., 2015).9 The authors argued that data analysis activities, collaboration, and discussion of research results within the course all promoted growth in students’ STEM thinking and skills.
9 It should be noted that although the participants are participating in a course required for all introductory biology students, there was no comparison sample for this study. Therefore, it is not certain whether the growth in ability to analyze and interpret data was any greater for the CURE students than it would have been in a traditional introductory biology course.
Knowledge of Experimental Design
Study authors have argued that students gain knowledge of experimental design from their work in apprentice-style UREs. A large study of research and nonresearch students across multiple institutions found that the research students (both sponsored and nonsponsored) reported gaining an understanding of experimental design at much higher rates than did the nonresearch students (Russell et al., 2007). A qualitative study of apprentice-style research at four liberal arts colleges found that students and faculty both reported student gains in understanding of experimental design, enhanced ability to connect their research experience to coursework, improved understanding of the role of theory in STEM research, and increased ability to troubleshoot research problems (Hunter et al., 2007; Seymour et al., 2004). Thiry and colleagues (2011) used interviews to gather data comparing outcomes from apprentice-style UREs with those from STEM coursework and other out-of-class STEM professional experiences. They found that undergraduate researchers who chose to enroll in UREs had developed a more sophisticated understanding of the research process and the nature of STEM knowledge than nonresearch students.
Research on student outcomes from CUREs has also documented gains in students’ conceptions of experimental design. A pre and post study of research-based versus traditional course labs found that the only statistically significant difference between the groups was greater increases in the research students’ perception of both their problem solving skills and their understanding of experimental design (Russell et al., 2015). Using a pre and post assessment of students performance, Kloser and colleagues (2013) documented that students made statistically significantly greater gains in understanding of experimental design and data interpretation. Open-ended surveys of students’ self-confidence in research design showed similar gains.
Understanding Disciplinary Research Practices
One of the most widely discussed outcomes from UREs in the research literature is developing scientific thinking skills or habits of mind. Some studies have argued that UREs help students develop a scientific approach to problems; a general understanding of the nature of the research process; and an understanding of how disciplinary knowledge is constructed, debated, and evaluated. One study examined the development over time of students’ abilities to perform tasks typical of STEM researchers, as well as their conceptions of STEM research (see Box 4-1).
Three in-depth ethnographic studies involving interviews, surveys, participant observation, and/or reflective journals have documented that students gain maturity in their beliefs about science from UREs, including a
more sophisticated understanding of the validity of knowledge claims, the role of theory in shaping research questions, taking scientific approaches to problems, and understanding the practice of science as a collaborative and detail-oriented activity (Cartrette and Melro-Lehrman, 2012; Ryder et al., 1999; Thompson et al., 2016). However, none of these studies included a comparison group.
CUREs have been argued to contribute to students’ understanding of the nature of scientific knowledge. Brownell and colleagues (2015) reported a change in students’ conceptions of scientific thinking through a blindly scored pre and post open-ended survey. That is, students had a more expert-like conception of research that was more grounded in the research experience (focusing on collaboration and data analysis) rather than viewing research simply as the development of hypotheses and using the scientific method. A pre and post survey also found that these students also perceived that their scientific thinking had matured. Russell and Weaver (2011) compared traditional, inquiry-based and research-based labs at five universities and found that students in the research-based laboratory section demon-
strated the most gains in understanding the nature of scientific knowledge, including more nuanced understandings of the role of creativity in science and their conceptions of science as a process. Other studies have also concluded that students learn about the scientific research process and the practice of science from CUREs (Harrison et al., 2011; Rowland et al., 2012).
Growth of Student Skills and Technical Knowledge
Another goal of UREs is growth in students’ skills and technical knowledge. Through UREs, students are provided with an avenue for practicing application of knowledge as they address a research problem. Although UREs and CUREs are often touted as fostering important skills, such as technical or laboratory skills, critical thinking, teamwork, and communication skills, few if any studies have measured these outcomes beyond self-report methods. Other studies have reported that students engaged in apprentice-style research perceived that they have gained communication skills (Craney et al., 2011; Junge et al., 2010), with at least one of these studies reporting that Black and Latino students perceived higher communication skills gains than their counterparts (Craney et al., 2011).
Several studies have compared student and faculty reports of URE outcomes and have documented that the faculty and students both perceived that the students developed communication skills, organizational and time management skills, technical skills, collaboration skills, and the ability to read and interpret primary literature from apprentice-style UREs (Hunter et al., 2007; Kardash, 2000). Additionally, Kardash (2000) found that students’ highest rated skills gains were in the oral communication of research results, such as presenting a poster, and their lowest rated skills gains were in writing a research paper for publication, suggesting the types of opportunities that students are likely to encounter during apprentice-style UREs. Few studies of CUREs have documented gains in skills, though students showed statistically significant increases in their comfort with reading and interpreting primary literature in a study of a CURE at a single institution (Drew and Tiplett, 2008).
Summary for Promoting an Understanding of STEM Disciplinary Knowledge and Practices
Following UREs, students frequently report gains in understanding of STEM content, data analysis, the nature of experiments, and a range of skills. In some cases, these outcomes have been corroborated by various assessments and scoring rubrics that look at disciplinary knowledge or abilities in experimental design. However, most of the studies use self-reporting and few include comparison groups to document causal claims;
hence, they indicate perceived improvements. Assuming the reports reflect the students’ true beliefs, this may be sufficient to bolster persistence in a STEM major (see below).
Integration of students into STEM culture is another goal of participation in UREs. The following section describes some studies to date on students’ feelings of autonomy and agency, belief in their own self-efficacy (i.e., feeling one “can” engage in a particular skill) and ability to act on their own, motivation, happiness, and commitment to persist in their field. For example, Healy and Rathbun (2013) reported that students developed more self-confidence in general as a result of their UREs. Estrada and colleagues (2011) showed that factors of self-efficacy, identity, and value endorsement may be important to the retention of historically underrepresented students’ interest and persistence in STEM by creating a type of “inoculation effect” that prevents loss of interest among students already pursuing a STEM degree. These results can begin to provide insights into the mechanism of why one finds improvements in a wide range of other outcomes.
UREs may foster an array of outcomes, such as increasing student confidence and self-efficacy; strengthening students’ sense of belonging in the discipline; providing professional socialization experiences; and fostering the traits, attitudes, and temperament of scientists. For instance, a qualitative study of students and faculty in multiple summer research programs in computer science reported that UREs promoted a sense of belonging in the discipline, especially for women in a male-dominated field such as computing (Barker, 2009). Another study in computer science documented the positive influence of the Affinity Research Group model of research mentoring on students’ identification with the larger professional community and the transformation of identity from student to researcher, especially for underrepresented minority students (Villa et al., 2013). The Affinity Research Group model outlines specific methods to socialize students into the research group through group orientation, scaffolding of students’ responsibility within the group’s work, and providing social and intellectual support within the group.
Several studies involving interviews or surveys have documented students’ increased confidence in research abilities, general self-confidence, and increased independence gained from apprentice-style research experiences (John and Creighton, 2012; Russell et al., 2007). Interviews with students and faculty at four liberal arts colleges also elicited the influence
of apprentice-style research on students’ beliefs in their ability to contribute to science, sense of ownership of a project, patience and perseverance with research work, tolerance for ambiguity, sense of responsibility and maturity, and development of a scientific identity (Hunter et al., 2007; Seymour et al., 2004). A study of research interns and their mentors in two sponsored-research programs found that personal and professional dispositions fostered from research experiences were mentioned more often than other outcomes such as cognitive gains or research skills, suggesting the importance of these outcomes to students (Kardash and Edwards, 2012).
Chemers and colleagues (2011) conducted a study with 327 undergraduates and 338 graduate students and postdoctoral fellows to examine students’ science support experiences (research experience, mentoring, and community involvement), psychological variables (science self-efficacy, leadership/teamwork self-efficacy, and identity as a scientist), and commitment to pursue a science career. They found that for the undergraduate students in this dataset, there was a strong relationship between science self-efficacy and both research experience and instrumental mentoring, suggesting that those who were more confident that they could perform the functions of a scientist (science self-efficacy) had more involvement in professional science activities (research experience) and with instrumental mentoring (helping students learn tasks of science career development). Moreover, students who were more likely to identify themselves as a scientist also stated that they were more likely to go on to work in scientific research (commitment to a science career). Estrada and colleagues (2011) found that science identity and endorsing the value of the scientific community were better predictors of persistence than was students’ sense of self-efficacy.
A study by Raelin and colleagues (2014) of engineering undergraduates looked at reciprocal relationships between work self-efficacy and co-op participation and between academic self-efficacy and academic achievement to see whether these factors played a critical role in retention. Academic achievement and academic self-efficacy, as well as contextual support in all time periods, were found to be critical to retention. Work self-efficacy, developed by students between their second and fourth years, was also an important factor in retention, though it was strongly tied to the students’ participation in co-op programs. The study also noted that higher retention was associated positively with the number of co-op experiences completed by students.
Several studies of CUREs have also documented enhanced positive attitudes and confidence of students enrolled in research-based courses. Shapiro and colleagues (2015) compared CURE students with students in traditional faculty-mentored research experiences on a single campus and
concluded that in both types of experiences, students developed a sense of independence, interest, and ownership of their project when they perceived that they had agency and choice within the work. Several studies of CUREs have documented that students gained confidence in their ability to perform laboratory tasks (Kloser et al., 2013; Rowland et al., 2012). A study of CUREs found that students had more positive attitudes toward research, collaboration, and peer critique; higher self-confidence in research-based laboratory tasks; and increased interest in pursuing future research experiences, compared to students in a traditional lab section. However, career interests did not change for either the CURE students or the comparison group (Brownell et al., 2012). A comparative study of CURE versus traditional lab sections found that students in the experimental section reported positive attitudinal changes toward understanding inquiry and the nature of science; increasing problem-solving ability; designing experiments; understanding how to conduct research; and the likelihood of choosing a STEM career in pre and post testing, whereas students in the control sections experienced declines in attitudes from the beginning to the end of the semester (Russell et al., 2015).
One of the most widely studied predictors of academic perseverance is self-efficacy. This line of research emerges from Bandura (1997, p. 3), who described self-efficacy as “the belief in one’s capabilities to organize and execute courses of action required to produce given attainments.” Bandura found that a person’s self-appraisal of ability is a strong predictor of the person’s likelihood to perform those actions in the future. Estrada and colleague’s (2011) research, using panel data from TheScienceStudy, found that when research experiences were more strongly correlated with perceived science self-efficacy, there was a greater intention to persist in biomedical careers.
Promoting Professional Identity
Another way that UREs promote identity development is by introducing students to the social and cultural processes underlying STEM practices, such as collaboration, critique, collegiality, mentorship, and peer review. An ethnographic study of student researchers and their faculty mentors found that students generated social ties with peers, postdocs, and faculty that they drew on for resources, information, and support (Thompson et al., 2016). Other qualitative studies have shown that apprentice-style research allowed students to enter into a community of practice where they learned the habits of mind, values, norms, and practices of researchers by working with experts who served as role models of STEM practices (Dolan and Johnson, 2010; Hunter et al., 2007; John and Creighton, 2012; Laursen et al., 2010; Thiry and Laursen, 2011). Apprentice-style research experiences,
in particular, have been argued to strengthen student-faculty interactions and provide a mentoring relationship for students that exposes them to STEM thinking and practices, which boosts their confidence that they can be scientists (Hunter et al., 2007; Laursen et al., 2010; Seymour et al., 2004). Students who worked closely with peers and faculty in UREs were more likely to report that the research experience had increased their interest in graduate school (Craney et al., 2011).
Research with historically underrepresented students describes how UREs have increased a sense of belonging and inclusion. Strong evidence exists that historically underrepresented students’ sense of belonging in academic environments is complex and often impeded (Hurtado and Carter, 1997). A sense of belonging influences the extent to which a student integrates into the academic community, which in turn affects intentions to persist (Hausmann et al., 2007). UREs designed specifically for promoting access for women and historically underrepresented students to engage in research have found that students describe an increasing sense of belonging and inclusion that may have been absent in the larger institution or STEM academic environments. Students who experience race-positive interactions10 while pursuing a STEM degree report feeling a greater sense of belonging (Lee and Davis, 2000; Mendoza-Denton et al., 2002), and URE programs aimed toward historically underrepresented populations claim to counteract the effects of perceived exclusion (Hurtado et al., 1998).
Stereotype threat research has shown that when there are “signals” or context contingencies that communicate to historically underrepresented students that they do not belong in the academic or STEM community, the students’ performances decline while cognitive vigilance increases (Murphy et al., 2007). Woodcock and colleagues (2012) found that participation in URE programs (specifically RISE and MARC) served to buffer students from the effects of stereotype threat, although they still experienced it.
Project designs that encourage students to take ownership of their part of their URE’s project have been associated with increased student retention in STEM (Hanauer and Dolan, 2014). Others have argued that facets of the research experience promote student development, such as a student’s intellectual engagement in the project and in the opportunity to work independently with appropriate guidance (Laursen et al., 2010; Thiry et al.,
10 In this context, Mendoza-Denton and colleagues (2002, p. 914) describe a race-positive experience as involving “interactions with same-race peers in settings where concern about the possibility of race-based rejection was absent.”
2011). In addition, engaging in STEM practices for communicating results, such as preparing and presenting posters or attending conferences, has been linked to positive intellectual and psychosocial outcomes from apprentice-style research (Hunter et al., 2007; Laursen et al., 2010).
Within CUREs, Brownell and Kloser (2015) identified five critical components that serve to define CUREs and should be present in a research-based course: using the tools of a scientist, thinking and communicating like a scientist, collaboration, iteration, and discovery and relevance. Corwin and colleagues (2015a) developed a model from a review of the literature that identified project ownership, success in overcoming problems, and collaborative work with peers as additional critical components of CUREs. They also asserted that working with peers helps students to make improvements in technical skills—because peers may model or provide feedback about how to perform tasks—and that a sense of ownership over their work promotes students’ sense of belonging to the STEM community (Corwin et al., 2015a; Hanauer et al., 2012).
Value of Teamwork
Working as part of a research group in a scientific community can lead to social benefits such as belonging and inclusion. This relationship suggests that research examining the value of teamwork and collaboration can provide mechanistic support for the benefits of UREs. A comparison of various approaches to collaboration found that requiring students to consider ideas of their peers that differ from their own is more effective than allowing students to consider only ideas that are consistent with their own (Matuk and Linn, 2015). The authors concluded that their results on collaboration provide support for a mechanism associated with the Knowledge Integration Framework, introduced by Linn and Eylon (2011).
The composition of the teams has also been shown to be important. Female engineering students were randomly assigned to one of three engineering groups of varying sex composition: 75 percent women, 50 percent women, or 25 percent women. For first-year students, group composition had a large effect: women in female-majority and sex-parity groups felt less anxious than women in female-minority groups. However, among advanced students, sex composition had no effect on anxiety. An important result was that group composition did have a statistically significant effect on verbal participation, regardless of women’s academic seniority: women participated more in female-majority groups than in the sex-parity or female-minority groups (Dasgupta et al., 2015).
Carter and colleagues (2016) examined the impact of undergraduate re-
search, broadly defined,11 in engineering and focused on three specific learning outcomes: communication, teamwork, and leadership. They studied 5,126 students across 31 colleges of engineering. After propensity score adjustment, the study found no statistically significant effect on teamwork or leadership skills, but it did find that URE participation was a significant predictor of perceived communication skills. This study highlights the importance of taking into account selection bias when assessing the effect of co-curricular programs12 on student learning. Implications of the study include expanding undergraduate research opportunities when possible and incorporating communication and leadership skill development into the required course curriculum.
Summary for Integrating Students into STEM Culture
Multiple studies indicate that students who participate in UREs feel more comfortable in STEM, have positive attitudes about STEM, and show increased confidence in being able to contribute to research after participation. Some work indicates that this feeling of confidence leads to greater engagement in STEM. The exposure to the culture, practices, and processes of STEM seems to increase students’ feelings of belonging and sustain a professional identity, and such exposure may buffer students from the effects of stereotype threat. These increases in confidence, engagement, and identification with STEM, as well as endorsement of community values, may help provide some mechanistic explanation for why participation in UREs also improves outcomes such as retention and skills development.
As discussed previously, there remain many unanswered questions about who participates in UREs and whether these experiences have a differential effect on specific subpopulations. While the committee was not able to find comprehensive data on the mix of men and women who did
11Carter and colleagues (2016) did not distinguish between the different types of undergraduate research in which a student could participate (e.g., research as part of the curriculum or a program of research such as in a faculty members lab).
12 Co-curricular activities were measured as the months spent in: engineering internships, engineering cooperative education experiences, study abroad or international school-related tours, humanitarian engineering projects, student design projects/competitions beyond class requirements, involvement in an engineering club or student chapter of a professional society, engineering-related clubs or programs for women and/or minority students, and other clubs or activities (e.g., civic or church organizations, campus publications, student government, Greek life, sports) (Carter et al., 2016, p. 371).
undergraduate research, they did locate some studies on the topic. When taken as a group, the results of these studies appear inconsistent, with some reporting differential impacts by gender (Campbell and Skoog, 2004; Gregerman, 2008; Junge et al., 2010) but many others showing similar outcomes for both genders (Craney et al., 2011; McGee and Keller, 2007; Russell et al., 2007; Thiry et al., 2012).
The committee found a limited amount of data on first generation college students’ participation in UREs, largely consisting of descriptive, qualitative studies that document the perceptions of the students who have engaged in UREs (Carpi and Lents, 2013; Ishiyama, 2007; Kwong Caputo, 2013; Stephens et al., 2014; Van Soom and Donche, 2014). Results from these studies suggest that programs of undergraduate research may be beneficial for women and first generation students, although it is not clear whether the benefits of participation are any different than for the majority of students. Nonetheless, if these students are at greater risk of leaving a STEM major, which appears to be the case in some fields, retention at the same rate as the average for all students would be a plus.
Although almost all of the research on UREs documents positive outcomes for research participants, several studies have noted less than desirable outcomes associated with poorly designed or poorly implemented research experiences, typically affecting only a small group of students (Craney et al., 2011; Harsh et al., 2011; Thiry et al., 2011). A poorly designed and/or implemented URE could involve students who have not received the proper training to do the work. It could also involve students lacking access to important resources due to an unexpected loss of research funds. Students could also be involved in projects that do not have the appropriate comparison samples built into the design. This could limit the types of claims that could be made about the research and reduce the quality of the work. Some studies have equated a lack of adequate mentoring with poor outcomes, such as loss of interest in graduate school or in the major (Barker, 2009; Thiry et al., 2011). Negative student-faculty interactions within research experiences can be particularly detrimental for students who are underrepresented in their fields, such as women in computing (Barker, 2009). These findings support the value of professional development for URE mentors. Other aspects of poorly designed research experiences, such as a lack of autonomy, inadequately selected projects (i.e., students take on an overly ambitious project given the time designated for the URE), or a general lack of structure have contributed to students’ loss of confidence or loss of interest in STEM careers (Harsh et al., 2011; Thiry et al., 2011).
The evidence suggests that programs with research experience components are more likely to contribute to developing and sustaining student interest in STEM fields. The majority of studies cited here, however, have focused on persistence in biology or biomedical fields. More research concerning the impact of research experience in engineering, mathematics, other sciences, and technology fields is needed. At the same time, future research could examine how the duration of the URE, the timing of the experience in the academic career, and the quality of the research experience influences STEM interest and persistence.
It is clear that there is a need for additional evidence on the impacts of UREs. Perhaps most important is the need for more well-designed studies that can provide more than descriptive evidence about the effect of UREs. For example, there is a need for research that accounts for differences in URE and non-URE populations upon entering research experiences beyond simple demographic or GPA differences (e.g., differences in interest, motivation, aspirations, and confidence prior to the research experience). In turn, this points to the need for better instruments and a well-articulated experimental design, to measure these differences both before and after the URE. Although it may be difficult in most instances to design randomized controlled trial of UREs for students, it is certainly possible to improve the selection of the populations that are under study.
Educational and career outcomes have been among the most studied aspects of UREs or CUREs. However, many studies rely on students’ self-report of aspirations or alumni retrospective accounts of the influence of research on their career or educational decisions, rather than longitudinally tracking students’ educational or career outcomes. Such long-term studies can be logistically and financially challenging, but they would greatly enhance the claims that research experiences inspire students to enroll in graduate degree programs or strengthen their commitment to STEM careers.
There is also a need for more research on nontraditional populations (pre-service teachers or current teachers in research experiences, community college students in both workforce and transfer programs, students of nontraditional ages, veterans, students with disabilities, etc.), who typically do not have access to UREs. CUREs can provide one potential approach for understanding the potential benefits for these populations as students simply need to sign up for a course. This could effectively enable more students to engage in research opportunities beyond traditional apprentice-style programs (Bangera and Brownell, 2014). As more opportunities develop, it will become important to understand whether all populations benefit (or do not benefit) in the same ways as the populations on whom research has
already been done. Additionally, there is a need for research on non-STEM majors in UREs to document whether outcomes for nonmajors differ from those of STEM majors.
Perhaps surprisingly, one of the least-documented effects of UREs is improvement in STEM thinking abilities for students. This includes improvements in disciplinary expertise, research design, and understanding of the research process. Most of the existing research has relied on self-report to document these important outcomes. Studies that include supplemental measures to self-report do not often include enough detail or description of what these instruments measure and how the instruments may have been piloted or validated. For example, whereas some studies of CUREs show improvement for students in course examinations, there is typically no discussion of what these examinations measure. Improvement in course grades is encouraging, but to be convincing such studies must provide information on what the course assessments are designed to measure and how and why a URE has changed student responses on these items.
UREs are diverse in their structure and goals, so it is not surprising that the questions and methodologies used to investigate the effectiveness of UREs in achieving those goals are similarly diverse. Much of the published literature focuses on outcomes of participation, retention, and persistence. Additional research has examined the potential benefit of UREs on developing an understanding of STEM disciplinary practices (e.g., content knowledge, concepts, and corresponding research skills) and integrating students into the STEM culture (e.g., project ownership, sense of belonging, teamwork). The committee’s review of the literature shows that most of the studies of UREs to date either are descriptive case studies or use correlational designs. Only a few studies have generated the causal evidence necessary to draw conclusions about the precise effects of UREs. However, the information currently available suggests that UREs may be beneficial for students due to their potential to improve participation and retention of students in STEM majors, as well as improving students’ knowledge of career options, experimental design, and related disciplinary thinking (Graham et al., 2013; Rodenbusch et al., 2016). Multiple studies also indicate that students who participate in UREs feel more comfortable in STEM, have more positive attitudes, and show increased confidence in being able to contribute to research upon URE completion. A few studies have documented reliable improvements relating to degree completion and persistence of interest in STEM careers for historically underrepresented students (Byars-Winston et al., 2015; Chemers et al., 2011; Jones et al., 2010; Nagda et al., 1998; Rodenbusch et al., 2016; Schultz et al., 2011). Additional
research is needed to better understand the mechanisms that explain why participation in UREs could lead to improved student outcomes.
Adedokun, O.A., Zhang, D., Carleton Parker, L., Bessenbacher, A., Childress, A., and Burgess, W.D. (2012). Understanding how undergraduate research experiences influence student aspirations for research careers and graduate education. Journal of College Science Teaching, 42(1), 82-90.
Adedokun, O.A., Bessenbacher, A.B., Parker, L.C., Kirkham, L.L., and Burgess, W.D. (2013). Research skills and STEM undergraduate research students’ aspirations for research careers: Mediating effects of research on self-efficacy. Journal of Research in Science Teaching, 50(8), 940-951.
Adedokun, O.A., Parker, L.C., Childress, A., Burgess, W., Adams, R., Agnew, C.R., Leary, J., Knapp, D., Shields, C., Lelievre, S., and Teegarden, D. (2014). Effect of time on perceived gains from an undergraduate research program. CBE–Life Sciences Education, 13, 139-148.
Bandura, A. (1997). Self-Efficacy: The Exercise of Control. New York: Freeman.
Bangera, G., and Brownell, S.E. (2014). Course-based undergraduate research experiences can make scientific research more inclusive. CBE–Life Sciences Education, 14, 602-606.
Barlow, A.E., and Villarejo, M. (2004). Making a difference for minorities: Evaluation of an educational enrichment program. Journal of Research in Science Teaching, 41(9), 861-881.
Bauer, K.W., and Bennett, J.S. (2003). Alumni perceptions used to assess undergraduate research experience. The Journal of Higher Education, 74(2), 210-230.
Barker, L. (2009). Student and faculty of undergraduate research experiences in computing. ACM Transactions on Computing Education, 9(1), 1-28.
Brownell, S.E., and Kloser, M.J. (2015). Toward a conceptual framework for measuring the effectiveness of course-based undergraduate research experiences in undergraduate biology. Studies in Higher Education, 40(3), 525-544.
Brownell, S.E., Kloser, M.J., Fukami, T., and Shavelson, R. (2012). Undergraduate biology lab courses: Comparing the impact of traditionally based “cookbook” and authentic research-based courses on student lab experiences. Journal of College Science Teaching, 41(4), 36-45.
Brownell, S.E., Wenderoth, M.P., Theobald, R., Okoroafor, N., Koval, M., Freeman, S., Walcher-Chevillet, C.L., and Crowe, A.J. (2014). How students think about experiemental design: Novel conceptions revealed by in-class activities. BioScience, 64(2), 125-137.
Brownell, S.E., Hekmat-Scafe, D.S., Singla, V., Chandler Seawell, P., Conklin Imam, J.F., Eddy, S.L., Sterns, T., and Cyert, M.S. (2015). A high enrollment course-based undergraduate research experience improves student conceptions of scientific thinking and ability to interpret data. CBE–Life Sciences Education, 41, 1-14.
Byars-Winston, A.M., Branchaw, J., Pfund, C., Leverett, P., and Newton, J. (2015). Culturally diverse undergraduate researchers’ academic outcomes and perceptions of their research mentoring relationships. International Journal of Science Education, 37(15), 2533-2554.
Campbell, A., and Skoog, G. (2004). Preparing undergraduate women for science careers. Journal of College Science Teaching, 33(5), 24-26.
Carpi, A., and Lents, N.H. (2013). Research by undergraduates helps underfinanced colleges as well as students. Chronicle of Higher Education, 60(9), B30-B31.
Carter, F.D., Mandell, M., and Maton, K.I. (2009). The influence of on-campus, academic year undergraduate research on STEM Ph.D. outcomes: Evidence from the Meyerhoff Scholarship Program. Educational Evaluation and Policy Analysis, 31(4), 441-462.
Carter, D.F., Ro, H.K., Alcott, B., and Lattuca, L. (2016). Curricular connections: The role of undergraduate research experiences in promoting engineering students’ communication, teamwork, and leadership skills. Research in Higher Education, 57(3), 363-393.
Cartrette, D.P., and Melro-Lehrman, B.M. (2012). Describing changes in undergraduate students’ preconceptions of research activities. Research in Science Education, 42, 1073-1100.
Chang, M.J., Sharkness, J., Hurtado, S., and Newman, C.B. (2014). What matters in college for retaining aspiring scientists and engineers from underrepresented racial groups. Journal of Research in Science Teaching, 51(5), 555-580.
Chemers, M.M., Zurbriggen, E.L., Syed, M., Goza, B.K., and Bearman, S. (2011). The role of efficacy and identity in science career commitment among underrepresented minority students. Journal of Social Issues, 67(3), 469-491.
Corwin, L.A., Graham, M.J., and Dolan, E.L. (2015a). Modeling course-based undergraduate research experiences: An agenda for future research and evaluation. CBE–Life Sciences Education, 14, es1. Available: http://www.lifescied.org/content/14/1/es1.full.pdf+html [January 2017].
Corwin, L.A., Runyon, C., Robinson, A., and Dolan, E.L. (2015b). The Laboratory Course Assessment Survey: A tool to measure three dimensions of research-course design. CBE–Life Sciences Education, 14(4), ar37. Available http://www.lifescied.org/content/14/4/ar37.abstract [November 2016].
Craney, C., McKay, T., Mazzeo, A., Morris, J., Prigodich, C., and de Groot, R. (2011). Cross-discipline perceptions of the undergraduate research experience. The Journal of Higher Education, 82(1), 92-113.
Dasgupta, A.P., Anderson, T.R., and Pelaez, N. (2014). Development and validation of a rubric for diagnosing students’ experimental design knowledge and difficulties. CBE–Life Sciences Education, 13(2), 265-284.
Dasgupta, N., Scircle, M.M., and Hunsinger, M. (2015). Female peers in small work groups enhance women’s motivation, verbal participation, and career aspirations in engineering. Proceedings of the National Academy of Sciences, 112(16), 4988-4993.
Deek, F., Briller, V., Friedman, R., and Joshi, K. (2003). Active research experience for undergraduates increases students’ motivation and academic performance. Pp. 8.161.1-8.161.18. Paper presented at 2003 ASEE Annual Conference and Exposition, Nashville, TN. Available: https://peer.asee.org/12353 [November 2016].
Dolan, E. (2016). Course-Based Undergraduate Research Experiences: Current Knowledge and Future Directions. Paper commissioned for the Committee on Strengthening Research Experiences for Undergraduate STEM Students. Board on Science Education, Division of Behavioral and Social Sciences and Education. Board on Life Sciences, Division of Earth and Life Studies. Washington, DC: National Academies of Sciences, Engineering, and Medicine. Available: http://nas.edu/STEM_Undergraduate_Research_CURE. [February 2016].
Dolan, E., and Johnson, D. (2010). The undergraduate-postgraduate-faculty triad: Unique functions and tensions associated with undergraduate research at research universities. CBE–Life Sciences Education, 9, 543-553.
Drew, J.C., and Tiplett, E.W. (2008). Whole genome sequencing in the undergraduate classroom: Outcomes and lessons from a pilot course. Journal of Microbiology & Biology Education, 9, 3-11.
Eagan, M.K., Hurtado, S., Chang, M.J., Garcia, G.A., Herrara, F.A., and Garibay, J.C. (2013). Making a difference in science education: The impact of undergraduate research programs. American Educational Research Journal, 50(4), 683-713.
Estrada, M., Woodcock, A., Hernandez, P.R., and Schultz, P. (2011). Toward a model of social influence that explains minority student integration into the scientific community. Journal of Educational Psychology, 103(1), 206-222.
Fechheimer, M., Webber, K., and Kleiber, P. (2011). How well do undergraduate research programs promote engagement and success of students? CBE–Life Sciences Education, 10, 156-163.
Feldman, A., Divoll, K., and Rogan-Klyve, A. (2009). Research education of new scientists: Implications for science teacher education. Journal of Research in Science Teaching, 46(4), 442-459.
Gilmore, J., Vieyra, M., Timmerman, B., Feldon, D., and Maher, M. (2015). The relationship between undergraduate research participation and subsequent research performance of early career STEM graduate students. The Journal of Higher Education, 86(6), 834-863.
Graham, C.R., Woodfield, W., and Harrison, J.B. (2013). A framework for institutional adoption and implementation of blended learning in higher education. The Internet and Higher Education, 18, 4-14.
Gregerman, S. (2008). Promising Practices in STEM Education. Paper commissioned for the Workshop on Evidence on Promising Practices in Undergraduate STEM Education. National Research Council, Washington, DC. Available: http://sites.nationalacademies.org/cs/groups/dbassesite/documents/webpage/dbasse_072631.pdf [November 2016].
Grimberg, S.J., Langen, T.A., Compeau, L.D., and Powers, S.E. (2008). A theme-based seminar on environmental sustainability improves participant satisfaction in an undergraduate summer research program. Journal of Engineering Education, 97(1), 95-102.
Gonzalez-Espada, W.J., and LaDue, D.S. (2006). Evaluation of the impact of the NWC REU program compared with other undergraduate research experiences. Journal of Geoscience Education, 54(5), 541-549.
Hanauer, D.I., and Dolan, E.L. (2014). The Project Ownership Survey: Measuring differences in scientific inquiry experiences. CBE–Life Sciences Education, 13(1), 149-158.
Hanauer, D.I., and Hatfull, G. (2015). Measuring networking as an outcome variable in undergraduate research experiences. CBE–Life Sciences Education, 14(4), ar38. Available http://www.lifescied.org/content/14/4/ar38.full [November 2016].
Hanauer D.I., Frederick J., Fotinakes B., and Strobel S.A. (2012). Linguistic analysis of project ownership for undergraduate research experiences. CBE–Life Sciences Education, 11, 378–385.
Harrison, M., Dunbar, D., Ratmansky, L., Boyd, K., and Lopatto, D. (2011). Classroom-based science research at the introductory level: Changes in career choice and attitude. CBE–Life Sciences Education, 10, 279-286.
Harsh, J.A., Maltese, A.V., and Tai, R.H. (2011). Undergraduate research experiences from a longitudinal perspective. Journal of College Science Teaching, 41(1), 84-91.
Hathaway, R.S., Nagda, B.A., and Gregerman, S.R. (2002). The relationship of undergraduate research participation to graduate and professional education pursuit: An empirical study. Journal of College Student Development, 43(5), 1-18.
Hausmann, L.R., Schofield, J.W., and Woods, R.L. (2007). Sense of belonging as a predictor of intentions to persist among African American and White first-year college students. Research in Higher Education, 48(7), 803-839.
Healy, N., and Rathbun, L.C. (2013). Developing Globally Aware Scientists and Engineers in Nanoscale Science and Engineering. Paper presented at the 120th ASEE Annual Conference & Exposition [June 23-26, 2013) in Atlanta, GA. Available: http://www.asee.org/public/conferences/20/papers/6264/download [November 2016].
Hunter, A.B., Laursen, S.L., and Seymour, E. (2007). Becoming a scientist: The role of undergraduate research in cognitive, personal and professional development. Science Education, 91(1), 36-74.
Hurtado, S., and Carter, D. F. (1997). Effects of college transition and perceptions of the campus racial climate on Latino college students’ sense of belonging. Sociology of Education, 70(4), 324-345.
Hurtado, S., Milem, J.F., Clayton-Pedersen, A.R., and Allen, W.R. (1998). Enhancing campus climates for racial/ethnic diversity: Educational policy and practice. The Review of Higher Education, 21, 279-302.
Ishiyama, J. (2007). Expectations and perceptions of undergraduate research mentoring: Comparing first generation, low income white/Caucasian and African American students. College Student Journal, 41(3), 540-549.
John, J., and Creighton, J. (2012). ‘In practice, it doesn’t always work out like that.’ Undergraduate experiences in a research community of practices. Journal of Further and Higher Education, 37(6), 1-19.
Jones, M., Amy, T., Barlow, E., and Villarejo, M. (2010). Importance of undergraduate research for minority persistence and achievement in biology. Journal of Higher Education, 81, 82-115.
Junge, B., Quiñones, C., Kakietek, J., Teodorescu, D., and Marsteller, P. (2010). Promoting undergraduate interest, preparedness, and professional pursuit in the sciences: An outcomes evaluation of the SURE program at Emory University. CBE–Life Sciences Education, 9(2), 119-132.
Kardash, C.M. (2000). Evaluation of an undergraduate research experience: Perceptions of undergraduate interns and their faculty mentors. Journal of Educational Psychology, 92(1), 191-201.
Kardash, C.M., and Edwards, O.V. (2012). Thinking and behaving like scientists: Perceptions of undergraduate science interns and their faculty mentors. Instructional Science, 40, 875-899.
Kloser, M.J., Brownell, S.E., Shavelson, R.J., and Fukami, T. (2013). Effects of a research-based ecology lab course: A study of nonvolunteer achievement, self-confidence, and perception of lab course purpose. Journal of College Science Teaching, 42(3), 72-81.
Kwong Caputo, J.J. (2013). Undergraduate Research and Metropolitan Commuter University Student Involvement: Exploring the Narratives of Five Female Undergraduate Students. [Dissertation]. Available at http://pdxscholar.library.pdx.edu/open_access_etds/1006/ [November 2016].
Laursen S.L., Hunter A.B., Seymour E., Thiry H., and Melton G. (2010). Undergraduate Research in the Sciences: Engaging Students in Real Science. San Francisco: Jossey-Bass.
Lee, R.M., and Davis III, C. (2000). Cultural orientation, past multicultural experience, and a sense of belonging on campus for Asian American college students. Journal of College Student Development, 41(1), 110-115.
Linn, M.C., and Eylon, B.S. (2011). Science Learning and Instruction: Taking Advantage of Technology to Promote Knowledge Integration. New York: Routledge.
Litzinger, T.A., Lattuca, L.R., Hadgraft, R.G., and Newstetter, W.C. (2011). Engineering education and the development of expertise. Journal of Engineering Education, 100(1), 123-150.
Lopatto, D. (2004). Survey of Undergraduate Research Experiences (SURE): First findings. Cell Biology Education, 3, 270-277.
Lopatto, D., Alvarez, C., Barnard, D., Chandrasekaran, C., Chung, H., Du, C., Eckdahl, T., Goodman, A. Hauser, C., Jones, C.J., Kopp, O.R., Kuleck, G.A., McNeil, G., Morris, R., Myka, J.L., Nagengast, A., Overvoorde, P.J., Poet, J.L., Reed, K., Regisford, G., Revie, D., Rosenwald, A., Saville, K., Shaw, M., Skuse, G.R., Smith, C., Spratt, M., Stamm, J., Thomspon, J.S., Wilson, B.A., Witkowski, C., Youngblom, J., Leung, W., Shaffer, C.D., Buhler, J., Mardis, E., and Elgin, S.C.R. (2008). Genomics Education Partnership. Science, 322(5902), 684-685.
Matuk, C., and Linn, M.C. (2015). Examining the real and perceived impacts of a public idea repository on literacy and science inquiry. CSCL’15: Proceedings of the 11th International Conference for Computer Supported Collaborative Learning, Gothenburg, Sweden. International Society of the Learning Sciences. Available at https://www.isls.org/cscl2015/papers/MC-0178-FullPaper-Matuk.pdf [November 2016].
McGee, R., and Keller, J.L. (2007). Identifying future scientist: Predicting persistence into research training. CBE–Life Sciences Education, 6, 316-331.
Mendoza-Denton, R., Downey, G., Purdie, V.J., Davis, A., and Pietrzak, J. (2002). Sensitivity to status-based rejection: Implications for African American students’ college experience. Journal of Personality and Social Psychology, 83(4), 896-918.
Murphy, T.J., Shehab, R.L., Reed-Rhoads, T., Foor, C.E., Harris, B.J., Trytten, D.A., Walden, S.E., Besterfield-Sacre, M., Hallbek, M.S., and Moor, W.C. (2007). Achieving parity of the sexes at the undergraduate level: A study of success. Journal of Engineering Education, 96(3), 241-252.
Nagda, B.A., Gregerman, S.R., Jonides, J., von Hippel, W., and Lerner, J.S. (1998). Undergraduate student-faculty research partnerships affect student retention. The Review of Higher Education, 21(1), 55-72.
Pacific, L.B., and Thompson, N. (2011). Undergraduate science research: A comparison of influences and experiences between premed and non-premed students. CBE–Life Sciences Education, 10, 199-208.
Raelin, J.A., Bailey, M.B., Hamann, J., Pendleton, L.K., Reisberg, R., and Whitman, D.L. (2014). The gendered effect of cooperative education, contextual support, and self-efficacy on undergraduate retention. Journal of Engineering Education, 103(4), 599-624.
Rodenbusch, S.E., Hernandez, P.R., Simmons, S.L., and Dolan, E.L. (2016). Early engagement in course-based research increases graduation rates and completion of science, engineering, and mathematics degrees. CBE–Life Sciences Education, 15(2), ar20. Available at http://www.lifescied.org/content/15/2/ar20 [November 2016].
Rowland, S.L., Lawrie, G.A., Behrendorff, J.B.Y.H., and Gillam, E.M.J. (2012). Is the undergraduate research experience (URE) always best? Biochemistry and Molecular Biology Education, 40(1), 46-62.
Russell, C.B., and Weaver, G.C. (2011). A comparative study of traditional, inquiry-based, and research-based laboratory curricula: Impacts on understanding of the nature of science. Chemistry Education Research & Practice, 12, 57-67.
Russell, J.E., D’Costa, A.R., Runck, C., Barnes, D.W., Barrera, A.L., Hurst-Kennedy, J., Sudduth, E.B., Quinlan, E.L., and Schlueter, M. (2015). Bridging the undergraduate curriculum using an integrated course-embedded undergraduate research experience (ICURE). CBE–Life Sciences Education, 14, 1-10
Russell, S.H., Hancock, M.P., and McCullough, J. (2007). Benefits of undergraduate research experiences. Science, 316, 548-549.
Ryder, J., Leach, J., and Driver, R. (1999). Undergraduate science students’ images of science. Journal of Research in Science Teaching, 36(2), 201-219.
Sabitini, D.A. (1997). Teaching and research synergism: The undergraduate research experience. Journal of Professional Issues in Engineering Education and Practice, 123(3), 98-102.
Salsman, N., Dulaney, C., Chinta, R., Zascavage, V., and Joshi, H. (2013). Student effort in and perceived benefits from undergraduate research. College Student Journal, 47(1), 202-211.
Schultz, P.W., Hernandez, P.R., Woodcock, A., Estrada, M., Chance, R.C., Aguilar, M., and Serpe, R.T. (2011). Patching the pipeline: Reducing educational disparities in the sciences through minority training programs. Educational Evaluation and Policy Analysis, 33(1), 95-114.
Seymour, E., Hunter, A.B., Laursen, S.L., and DeAntoni, T. (2004). Establishing the benefits of research experiences for undergraduates in the sciences: First findings from a three-year study. Science Education, 88, 493-534.
Shaffer, C.D., Alvarez, C., Bailey, C., Barnard, D., Bhalla, S., Chandrasekaran, C., Chandrasekaran, V., Chung, H.M., Dorer, D.R., Du, C., Eckdahl, T.T., Poet, J.L., Frohlich, D., Goodman, A.L., Gosser, Y., Hauser, C., Hoopes, L.L., Johnson, D., Jones, C.J., Kaehler, M., Kokan, N., Kopp, O.R., Kuleck, G.A., McNeil, G., Moss, R., Myka, J.L., Nagengast, A., Morris, R., Overvoorde, P.J., Shoop, E., Parrish, S., Reed, K., Regisford, E.G., Revie, D., Rosenwald, A.G., Saville, K., Schroeder, S., Shaw, W., Skuse, G., Smith, C., Smith, M., Spana, E.P., Spratt, M., Stamm, J., Thompson, J.S., Wawersik, M., Wilson, B.A., Youngblom, J., Leung, W., Buhler, J., Mardis, E.R., Lopatto, D., and Elgin, S.C. (2010). The Genomics Education Partnership: Successful integration of research into laboratory classes at a diverse group of undergraduate institutions. CBE–Life Sciences Education, 9, 55-69.
Shaffer, C.D., Alvarez, C.J., Bednarski, A.E., Dunbar, D., Goodman, A.L., Reinke, C., Rosenwald, A.G., Wolyniak, M.J., Bailey, C., Barnard, D., Bazinet, C., Beach, D.L., Bedard, J.E.J., Bhalla, S., Braverman, J., Burg, M., Chandrasekaran, V., Chung, H.-M., Clase, K., DeJong, R.J., DiAngelo, J.R., Du, C., Eckdahl, T.R., Eisler, H., Emerson, J.A., Frary, A., Frohlich, D., Gosser, Y., Govind, S., haberman, A., Hark, A.T., Hauser, C., Hoogewerf, A., Hooper, L.L.M., Howell, C.E., Johnson, D., Jones, C.J., Kadlec, L., Kaehler, M., Silver Key, S.C., Kleinschmit, A., Kokan, N.P., Kopp, O., Kuleck, G., Leatherman, J., Lopilato, J., MacKinnon, C., Martinez-Cruzado, J.C., McNeil, G., Mel, S., Mistry, J., Nagengast, A., Overvoorde, P., paetkau, D.W., Parrish, S., Peterson, C.N., Preuss, M., Reed, L.K., Revie, D., Robic, S., Roecklein-Canfield, J., Rubin, M.R., Saville, K., Scjrpeder. S., Sharif, K., Shaw, M., Skuse, G., Smith, C.D., Smith, M.A., Smith, S.T., Spana, E., Spratt, M., Sreenivasan, A., Stamm, J., Szauter, P., Thompson, J.S., Wawersik, M., Youngblom, J., Zhou, L., Mardis, E.R., Buhler, J., Leung, W., Lopatto, D., and Elgin, S.C.R. (2014). A course-based research experience: How benefits change with increased investment of instructional time. CBE–Life Sciences Education, 13, 111-130.
Shapiro, C., Moberg-Parker, J., Toma, S., Ayon, C., Zimmerman, H., Roth-Johnson, E.A., Hancock, S.P., Levis-Fitzgerald, M., and Sanders, E.R. (2015). Comparing the impact of a course-based and apprentice-based research experiences in a life sciences laboratory curriculum. Journal of Microbiology & Biology Education, 16(2), 189-197.
Sirum, K., and Humburg, J. (2011). The Experimental Design Ability Test (EDAT). Bioscene: Journal of College Biology Teaching, 37(1), 8-16.
Stephens, N.M., Hamedani, M.G., and Destin, M. (2014). Closing the social-class achievement gap a difference-education intervention improves first-generation students’ academic performance and all students’ college transition. Psychological Science, 25(4), 943-953.
Szteinberg, G.A., and Weaver, G.C. (2013). Participants’ reflections two and three years after an introductory chemistry course-embedded research experience. Chemistry Education Research and Practice, 14, 23-35.
Thiry, H., and Laursen, S.L. (2011). The role of student-advisor interactions in apprenticing undergraduate researchers into a scientific community of practice. Journal of Science Education and Technology, 20(6), 771-784.
Thiry, H., Laursen, S.L., and Hunter, A.B. (2011). What experiences help students become scientists? A comparative study of research and other sources of personal and professional gains for STEM undergraduates. Journal of Higher Education, 82(4), 358-389.
Thiry, H., Weston, T.J., Laursen, S.L., and Hunter, A.B. (2012). The benefits of multi-year research experiences: Differences in novice and experienced students’ reported gains from undergraduate research. CBE–Life Sciences Education, 11, 1-13.
Thompson, J.J., Conaway, E., and Dolan, E.L. (2016). Undergraduate students’ development of social, cultural, and human capital in a networked research experience. Cultural Studies of Science Education, 11(4), 959-990.
Van Soom, C., and Donche, V. (2014). Profiling first-year students in STEM programs based on autonomous motivation and academic self-concept and relationship with academic achievement. PloS One, 9(11), e112489. Available at http://dx.doi.org/10.1371/journal.pone.0112489 [November 2016].
Villa, E.Q., Kephart, K., Gates, A.Q., Thiry, H., and Hug, S. (2013). Affinity research groups in practice: Apprenticing students in research. Journal of Engineering Education, 102(3), 444-466.
Villarejo, M., and Barlow, A.E. (2007). Evolution and evaluation of a biology enrichment program for minorities. Journal of Women and Minorities in Science and Engineering, 13(2), 119-144.
Weston, T.J., and Laursen, S.L. (2015). The Undergraduate Research Student Self-Assessment (URSSA): Validation for use in program evaluation. CBE–Life Sciences Education, 14, ar33. Available at http://www.lifescied.org/content/14/3/ar33.long [November 2016].
Woodcock, A., Graziano, W.G., Branch, S.E., Ngambeki, I., and Evangelou, D. (2012). Engineering students’ beliefs about research: Sex differences, personality, and career plans. Journal of Engineering Education, 101(3), 495-511.
Zydney, A.L., Bennett, J.S., Shahid, A., and Bauer, K. (2002). Faculty perspectives regarding the undergraduate research experience in science and engineering. Journal of Engineering Education, 91(3), 291-297.
This page intentionally left blank.