8

Considerations for Design and Implementation of Undergraduate Research Experiences

This report describes an ongoing conversation in the education community that claims that the benefits of undergraduate research experiences (UREs) justify the expansion of such programs. Yet Chapter 7 (Need for Research About UREs) has pointed out many areas where increased research is needed to better understand the impact of UREs and the potential tradeoffs among design choices. This situation created noticeable tension for the committee, as we are charged with working from the evidence base but also want to provide actionable guidance to educators. Given that many schools are moving now to increase their efforts to support undergraduate research, the committee has prepared this chapter to address issues of design and implementation of UREs, drawing on both the currently available evidence base and the expert opinion of the committee. We aim to present a structure for considering relevant aspects of UREs as part of a design and decision making process embedded in the conceptual framework described in Chapter 3.

This chapter is designed to serve as a guide to readers who wish to support the development of UREs on their campus—primarily faculty, URE program designers/directors, and institutional leaders. The committee identifies important questions to ask and issues to address during the process. Keep in mind that URE design and implementation can be a time-consuming process: key players should be provided with adequate time and resources to achieve their program goals, and they should be recognized when that is accomplished. In some situations, however, people must initiate a URE and carry out large amounts of work before getting buy-in from their department or institution. In this case it can be even more crucial that

there is an institutional mechanism to reward their efforts after the fact. The mission, priorities, and resources of the institution will influence many practical decisions of the department and faculty. Many of the questions presented here must be dealt with on an institutionwide basis, whereas others are the purview of departments, the primary concern of the URE designer, or the primary concern of the individual faculty mentor.

Several considerations need to be kept in mind when designing and implementing UREs, whether the intent is to create a new program, refine an existing one, or broaden (scale up) the access to a specific URE. These include the make-up of the student body, the types of programs that can be offered, the envisioned goals and outcomes of the experience, who will implement and who will serve as mentor, and the departmental and institutional constraints that might impact the design and implementation of the experience. Considering the goals of all participants will help ensure that the program can be successful and sustainable with adequate participants and human resources. Understanding the goals of the students will help in designing programs that keep the students engaged and motivated.

As discussed in various places throughout this report, and specifically addressed both within the research agenda in Chapter 7 and in the final chapter detailing the committees’ conclusions and recommendations, there is insufficient causal evidence to develop and support a comprehensive set of guidelines to promote specific best practices or to contrast the effectiveness of different mechanisms and programs. However, based upon the available descriptive evidence, the collective beliefs of the community, and emerging research that supports the utility of UREs in providing unique learning opportunities for students, we provide this chapter as a resource for design, implementation, and evaluation of UREs. In preparing the guidance reported here, the committee draws from best practices that have emerged from education research on the science of learning, published research evidence on UREs, resources and research syntheses by national organizations that support UREs, presentations made during the committee meetings, and the expertise of the committee members.

This chapter begins with some initial considerations to keep in mind when considering the type of URE program that will be institutionally appropriate. It then moves on to discuss goals, resources, implementation (including using the existing evidence and knowledge of how people learn), and improvement (evaluating UREs and resources available). Finally, the chapter concludes with a section that speaks to campus leaders about the importance of campus culture, systemic change, and rewards/incentives.

PRACTICAL QUESTIONS

Many factors need to be considered when trying to determine either the appropriate URE program(s) to implement or whether a new type of URE might be desirable and possible. To facilitate the process, answers to several questions can help to narrow down the potential formats. Some of these questions should be addressed on the departmental or institutional level, to ensure that adequate resources are available to the URE designers and that the tradeoffs that need to be made align with departmental and institutional priorities. The answers to other questions are in the purview of the faculty members guiding the research. The questions provided below are intended to be not an exhaustive list but a starting point.

• What is the overall goal of the program? For example, does it aim to provide research experiences for some or all students in a given STEM major, for students in the beginning courses for the major, or for some other overarching end?
• Is this an expansion of an existing program or a new program?
• How will the new program fit with any current programs? How will it fit within the existing curriculum and major academic requirements?
• What strategies will be used to reach the goals? Do they fit best with an apprentice-style model, a course-based undergraduate research experience (CURE), an internship, etc.? How much active time do the students need to reach the goals? How many hours per week should the students expect to participate? How much total time is needed, and how many weeks will the experience last? Is there already an experience on campus or at a nearby school with the proposed format?
• What are the program costs and how will participants be covered? Will internal and/or external funding be required? Can existing funds be used or repurposed, or will new sources of revenue be needed? If the program is initiated with grant funding, how will it transition to a sustainable mode of operation after the grant period?
• Do faculty members have the resources they need: access to knowledge about designing and assessing UREs and access to necessary financial and logistical resources?
• Is there appropriate space currently available or would modifications be necessary?
• How much faculty time will be needed? Will this require changes to existing responsibilities? How will participating faculty be rewarded or compensated for their time and energy spent on design and over

• sight of the research project? Will time for faculty participation be provided within the normal workload? If extra hours or summer participation will be required, how will that be compensated?

• Who will serve as research mentor(s), and what role will the mentor(s) play? How will the mentor(s) be prepared for that role?
• Who will provide hands-on training to the students, and how will the trainers be prepared for that role?
• Will students be given increased decision making opportunities and responsibility for formulating and designing the content of their research as their experience increases? Are there opportunities for students to take on increasing ownership of the project?
• Is one of the goals to ensure equity and access or to specifically broaden participation? How will students from populations of interest be recruited to the program? Is the recruitment and selection process equitable? Does it promote broadening participation?
• How will participation be documented and participants tracked?
• How will research ethics and standards of research documentation be taught?
• How will the students be rewarded/compensated? If graded, will the grade be pass/fail or a letter grade? How will grades be assigned? If students will be compensated, will they receive a stipend or hourly wage? If a summer program, will room and board be provided?
• How long is the intended research experience? What is the weekly (or monthly) time commitment expected of participants?
• What are the research expectations? Are there steps along the way where expectations must be documented? Do they include keeping a research notebook? Do they include presenting at a conference? How will they be clearly communicated to the students?
• Are there novel questions for students to tackle?
• Are students expected to present or publish on their research?
• Are there plans to help the students gain a sense of belonging?
• Are there opportunities for students to collaborate and discuss their research activities, as well as to reflect on the activities’ wider implications and connections in the field and to broader life issues? Do the students have the resources they need (e.g., access to housing, if needed, and to equipment, library, and mentors)?
• Are there processes in place by which students could file a complaint if they disagree with the decision on who is selected to participate, if they are not provided with the necessary resources to carry out their task, or if they experience discrimination or harassment?
• How will the success of the program be assessed?

Overall, having these basic questions in mind can serve as a guide when identifying programmatic needs that reflect the goals of the various stakeholders. In considering these questions, it might be helpful for URE designers to reflect on the various options discussed in Chapters 2 and 3 about the variety of types of UREs and the many interacting factors that influence them.

INITIAL DESIGN CONSIDERATIONS

For a designer of a URE, the initial steps in the process are to identify the goals that the URE will aim to achieve, recognize the key variables that may influence the URE, identify the types of programs available nationally that may serve as models, identify existing local programs that may be adapted or expanded to meet the goals, and determine what opportunities for innovation exist. Evaluating the programs that are already available can provide models and illuminate gaps. Considering the programs available locally may yield new partners or spark ideas that can be modified for use. Such investigations of the programs offered might point out a type of URE that is not in use but that could be added.

As described in the conceptual framework (see Chapter 3), UREs are affected by an interacting network of players (institutional and departmental policy makers and leaders, faculty, staff, and students) and by an institution’s mission, goals, and resources. These interactions occur within the broader context of national policy (determined by funders, disciplinary societies, and government) that impacts decisions made on campus. Campus decisions on faculty roles, faculty rewards, space, and allocation of resources are of critical importance and directly affect UREs.

Goals and resources must be considered when choosing the type of program so that it will fit the needs of the student population while also working within the constraints of the available support structures (e.g., having space for the program, the necessary human and financial resources). Finding or creating the right program structure that can appropriately balance these various factors can result in a more manageable and sustainable program in which the intended benefits and outcomes are achieved. That is, if the program will not fulfill the needs of the students or cannot be supported institutionally in the long run, then the sustainability of the program will be in question. Table 8-1 provides a simplified view of the landscape of URE program types, illustrating various types to facilitate consideration of options. (See Chapter 2 for a detailed discussion with relevant examples.)

Alignment of a planned URE with a particular program’s various goals and available resources is critical to offering academic experiences that will meet program goals for the students it targets. The types and specifics of UREs offered affect which of the definitional characteristics a student expe-

TABLE 8-1 Overview of the Variability of Attributes of UREs

riences.¹ For example, joining an established research group could channel student work toward a predetermined problem using an already identified approach. A class that challenges students to pick a local environmental issue to investigate could provide many choices for a student to select a novel research question, while another course may have a set research paradigm that all students are expected to follow. Over time, an institution or program may offer more variety in the types of UREs available to their students, and this may enable students to choose UREs particularly tailored to their goals.

There is variability in terms of when the URE is offered—semester, academic year, or summer—as well as in the support systems (human and financial) required. Whereas many programs have the potential to offer students academic credit, summer programs are more likely to need to provide a stipend (and/or other forms of monetary support, such as providing room and board); this is more often available for apprentice-style programs that have financial resources specifically linked to such programs (e.g., external grant funding secured by the faculty mentor, institutional resources, or donor-funded endowments). Bridge and wrap-around programs, which have additional student support structures included such as peer mentoring and tutors, generally require additional financial resources.

Considering the many options of different types of UREs (e.g., apprentice-style, CURE, internship, co-op), it may seem daunting to decide, for a particular program, on the type that best aligns with the relevant stakeholder goals and resources in hand or that might be obtained. Each program will have constraints that will shape the offering and favor some types of UREs over others. If, for example, the goal of a URE is to increase the number of students matriculating into graduate school in STEM fields, then a key component of the program (in addition to experiencing research) may be test preparation and assistance with graduate school applications. Similarly, if a goal is to increase STEM knowledge and literacy, a URE may include not only working alongside a faculty member in a lab, but also assigned readings and periodic workshops featuring presentations on research across STEM disciplines.

Table 8-1 lays out many of the categories and choices for each category that need to be considered in planning and implementing a URE.

Faculty who decide to organize CUREs or expand other research opportunities for undergraduates may need help acquiring the mentoring and managerial skills required to do so effectively (Pfund, 2016). A key characteristic of most CUREs is a “parallel” research problem: one for which the mentor can teach students a common set of experimental approaches and

___________________

¹ The committee’s set of URE definitional characteristics is specified in Conclusion 1 in Chapter 9.

common tools but within which each student has unique responsibilities. For example, in SEA-Phages (see Box 2-7 in Chapter 2) all students isolate a soil phage using a particular host bacterium; the isolated phages are related, but each will be unique, informing an analysis of phage evolution (Hatfull, 2015). Directing a CURE generally requires that faculty move beyond more traditional notions of “teaching” toward a more active mode of promoting student learning via the research framework, using pedagogies that are more aligned with active learning (e.g., shifting to a facilitator of student investigation rather than one who primarily imparts information).

Undergraduate research offices, created either as separate entities or as extensions of a college or university office of teaching and learning, can often provide a centralized resource for faculty, staff, and undergraduates engaged in UREs. In addition to helping undergraduates connect with appropriate experiences, they can facilitate general training (eg., how to keep a research notebook, research ethics), sponsor talks on STEM careers, manage paperwork, arrange summer housing for the undergraduates, or potentially even provide specialized instruction in research (see Box 8-1). At institutions that do not have an undergraduate research office to provide central support to those running or participating in UREs, an effort to create centralized procedures would be worthwhile; a part-time staff position could provide help with some of the needed features. Examples can be found in Appendix B and in the report from a convocation on integrating CUREs into the undergraduate curriculum (National Academies of Sciences, Engineering, and Medicine, 2015). The organizations described in the final section of this chapter may also serve as a source of ideas and

resources for faculty and administrators who are working to start or expand URE programs.

Another option to consider for developing highly effective UREs at many institutions is a “franchising” process. Under such a process, a well-designed URE (a CURE or program-based URE that has been thoroughly evaluated) could be adopted by many institutions. The process could be facilitated by having the initial sites develop tools and an evaluation procedure for other institutions to adopt and adapt. Identifying, encouraging, and funding dissemination of existing programs may accelerate the creation of effective UREs at many institutions and lead to evaluation efforts that can scale to tens or hundreds of institutions, hundreds of faculty, and thousands of students. Several examples of such consortia were discussed in Chapter 2. These consortia show particular promise in enabling institutions that have limited resources to successfully implement and sustain UREs (Blockus, 2016).

THE IMPORTANCE OF INCLUSION, ACCESS, AND EQUITY

Colleges and universities need to consider whether their approaches to offering UREs allow for equity of access. Emphasizing access and equity requires analyses and actions that are student-centered and focused. The Engage to Excel report of the President’s Council of Advisors on Science and Technology (2012) describes many potential benefits of having students engage in some kind of research or other discovery-based experience in STEM and calls on higher education to make research opportunities available to as many students as possible, as early in their undergraduate careers as feasible. However, common practice has been to select the most advanced students (either in terms of length of matriculation, relevant coursework completed, or academic performance as determined by indicators such as grade point average) for preferred access, on the grounds that they will benefit most from such opportunities. Unless the number of research opportunities can grow substantially, such selection decisions likely will exclude many students, particularly those who do not choose to declare a STEM major. Unfortunately, this can include those who intend to become teachers, especially those planning to teach in the elementary or middle grades and who are likely to major in education or English, neither of which is a STEM discipline.

There also is a risk of unfairness if faculty members select students based on those who approach them seeking such opportunities, as ethnic/racial minority students and first generation students often are aware neither of URE opportunities nor of the benefits of a URE (Bangera and Brownell, 2014). Faculty or other mentors also may hold unrecognized, implicit biases that certain types or levels of students are more qualified than others or can

contribute most to the research effort (Moss-Racusin et al., 2012). Both problems can restrict opportunities for disadvantaged students who might benefit the most from such experiences.

It may appear that requiring research and other discovery-based experiences through an apprentice-based program or CURE could address many of these issues of access and equity (Bangera and Brownell, 2014; Dolan, 2016). Accordingly, some colleges and universities are working to make a CURE part of the first-year experience. For example, the First Year Innovation and Research Experience (FIRE) program at the University of Maryland, College Park (modeled on the FRI program at The University of Texas at Austin described in Box 2-9) attempts to lower barriers to research and persistence.² If projects can be limited (for the most part) to hours for which a course or lab is scheduled, then more students who must work to support themselves and their families or who must commute to campus from long distances will be able to participate. However, requiring that most or all students engage in this kind of work presents its own set of problems, as there can be substantial logistical challenges to participating in research. In addition, if students feel that they are being compelled to participate in activities that they neither welcome nor appreciate, then they likely will not do so enthusiastically. Team structure, group work, and the quality of their URE work overall may suffer as a result, thereby diminishing the experience for all students involved in the URE.

Requiring undergraduate research can also present financial challenges. If a required CURE comes with extra fees, it may discourage some students’ from choosing that major. CUREs that add extra fees for participation compared with traditional courses may place an insurmountable burden on some students, essentially blocking their enrollment. On the other hand, if students can be paid a stipend or hourly wage for participating in research with a faculty member, this may alleviate their need to find an off-campus job to cover their expenses and may serve to promote participation in a URE. Asking students to participate in off-campus symposia or meetings of disciplinary societies to present their work may preclude some students from participating, unless their costs for travel are provided by the institution. Too often, these kinds of special costs pose particular burdens for first generation, underrepresented, nontraditional, and socioeconomically disadvantaged students. Intentional recruiting of these subpopulations and dedicated funding sources to provide financial aid can counter many of these obstacles.

On the contrary, making a research experience optional can result in students opting out because they are concerned about the amount of time and effort required for the academic credits gained, and they may worry

___________________

² See http://www.fire.umd.edu/about.html [December 2016].

about how such programs or courses will be graded. Clear communication from the faculty, peer recruiting, and joining a CURE with friends are all conditions that may help ameliorate these challenges. For example, it might be necessary for faculty to talk to students about “failure” being commonplace in research and that in this course their grade will not suffer from an inability to get “the correct answer.”

CONSIDERING THE GOALS OF ALL PARTICIPANTS

The design of UREs should consider the goals of all participants: students, faculty, department, and institution. Knowing who the various stakeholders are and paying attention to their goals and priorities can help shape and direct the design of a new URE or the refinement of an existing program. This is especially important when the stakeholders are at multiple institutions, such as in a National Science Foundation (NSF)-funded science, engineering, or technology center. Moreover, it is crucial to think about how the attributes of specific student populations (e.g., students of color, women, first generation students, community college transfers, commuter versus resident students, full-time versus part-time students, majors in the URE field versus nonmajors) affect the goals those students might have and how those goals will be addressed within the design (Blockus, 2016; Dolan, 2016).

As discussed in the conceptual framework, the committee has grouped the institutional goals for students participating in UREs into three major categories: (1) increasing participation and retention of students in STEM, (2) promoting STEM disciplinary knowledge and practices, and (3) integrating students into STEM culture (see Figure 3-1). This categorization was done to organize the outcomes that have been most frequently measured and documented in the literature. Although these categorical goals may not precisely mirror the motivations driving a particular URE design, they should be considered, along with the goals of the faculty and goals of the students.

Students themselves may not focus on the same goals, described above, that institutions and faculty may have for them. They may be focused on more practical goals, such as the potential for UREs to help them stand out more prominently in the sea of applicants to graduate or professional schools or for future employment. A student may be interested in learning more about a topic or a technique covered in a previous course or in working with a faculty member who taught the course; students may want to add to their resumes or get a strong letter of recommendation. It is important for faculty to share their goals for the students with the URE participants at the start of each experience. In an apprentice-style URE, mentors and students should take time to discuss the student’s goals as well.

Students may choose to participate in a URE because of the topic of the research. For example, a particular URE may provide students with opportunities for community or civic engagement (such as a project related to environmental pollution) or opportunities to explore in depth an issue that has had an impact on the student’s life or the lives of their families, friends, or communities (such as research on a specific disease or illness). Other students may be thinking of careers in a STEM discipline and see UREs as a chance to learn research skills and determine whether they find research interesting enough to want to pursue it further as a career option. Thus, some students might finish a URE with a solidified feeling that research is for them and go on to persist in a STEM degree (or seek a career), whereas others may benefit from the experience itself but might determine that research is not for them. Moreover, some students who never envisioned being a STEM researcher might discover a career path that suits them, although they had not considered it before. Further, as suggested in the conceptual framework, preparation for many career opportunities (perhaps most) can be enhanced by participation in a URE.

RESOURCES

The issue of resources for UREs is complex. Resources needed for research are as varied as the questions that drive the research and the disciplines that set the context for the research opportunity. A comprehensive list of resources needed for all forms of UREs across all STEM disciplines and research questions is beyond the scope of this report, so what follows is an illustrative compendium of resource issues and topics, which may be helpful to consider in the design of UREs. The success of UREs depends on supportive departmental administrators and interested faculty, along with the means to encourage and compensate faculty and to provide facilities so that the faculty have both the time and resources to engage undergraduates in research.

As departments and institutions consider expanding research opportunities for their undergraduates, a primary consideration and concern is cost. Can UREs be expanded by reallocation of current resources, or must new resources be identified and secured? Costs can be estimated based on current institutional budgets of colleges and universities that are currently engaged in providing UREs, as well as from public data on awards supporting such efforts by NSF, the National Institutes of Health, private foundations, and other funders. New costs will depend on the proposed program design (see the list of practical questions above in this chapter) and on what is already incorporated in the instructional budget.

The committee put together a set of questions to gather information about how this challenge is being addressed on a variety of campuses. See

Appendix B for excerpts from some of the institutional responses. These responses reveal considerable ingenuity as campuses move to exploit, repurpose, and conjoin current resources, even as they seek additional resources to expand or strengthen UREs.

Variations in Resources by Institution Type

Resources vary across institutions, but there are some commonalities that the committee observed within types of institutions. By their nature and mission, research-intensive universities include a large number of research-active faculty who potentially are available to design projects and participate in mentoring undergraduate students. Liberal arts schools and community colleges generally have a greater proportion of smaller classes, including smaller introductory classes, such that the transition from a “cook-book” lab course to a more research- or discovery-based lab course may be more easily accomplished within the existing infrastructure.

Institutions with an explicit mission to promote undergraduate research most often have resources already in place (e.g., budget, support personnel, space, equipment) and provide recognition and rewards to departments and faculty for achievement in this mission area. Some four-year colleges pride themselves on having all students engaged in research with a faculty member. For example, The College of New Jersey has reconfigured its entire curriculum to focus on undergraduate research, scholarship, and creative activity (Osborn and Karukstis, 2009). More information on this institution is available in Box 8-2. The culture of an institution with respect to innovation in pedagogy and support for faculty development can influence the extent to which UREs are readily introduced or improved. The physical resources available, including laboratories, field stations, engineering design studios and testing facilities, and the like can have an impact, as can the ability to access resources in the surrounding community (including other parts of a large university campus). In some cases UREs can be designed to take advantage of equipment that can be repurposed from pre-existing teaching laboratories. Faculty may be motivated by a desire to improve instruction, enrich an existing lab experience, or satisfy requirements necessary to receive funding (i.e., requirements aimed at furthering broader objectives of their home institution or funding sources). The intellectual traditions of the STEM field also have an impact. UREs appear to be more common in the life sciences and in geoscience, computer science, chemistry, and engineering than in physics and mathematics. UREs are increasingly more common in the social sciences than they were in the past and are even starting to appear in the humanities.

Some types of colleges and universities (community colleges, historically black colleges and universities, and others) generally expect faculty to

devote most of their time to teaching. Course-based research is more likely to be compatible with such expectations than one-on-one or one-on-few (mentor-to-mentee ratio) apprentice-style experiences.

The institutional and departmental requirements to support course-based research at community colleges will be similar to those noted throughout this report for four-year colleges. Given the increasing pressure to maintain already low tuition costs at the former, they will be under pressure to continue with traditional classroom instruction, which is less labor- and resource-intensive than research-based courses. In addition, it is harder for faculty to find time to develop UREs at institutions where they are required to teach many courses per semester. Faculty at community colleges generally have the heaviest teaching expectations, with little or no expectations or incentives to maintain a research program; they often have limited access to lab or design space and to a comprehensive collection of scientific or engineering journals, as well as few resources to undertake any kind of a research program. These conditions constrain the extent to which UREs can be offered to the approximately 40 percent of U.S. undergraduates who are enrolled in the nation’s community colleges (which generally have high percentages of underrepresented students) for students’ initial science training.³

Financial Costs and Benefits of UREs

The capital resources required for undergraduate research depend on the discipline, type of program, and topic under investigation. Availability of facilities and laboratories, access to field sites, and access to equipment are important considerations. Other financial considerations include staff available for coordination, lab supervision, and mentoring; funds for financial support of students and mentors; and faculty release time for research project development. Local resources, such as community field sites and the availability of business and industry representatives to mentor students, can also be considered as “capital” to support a URE program.

Due to the wide range of potential financial costs and the lack of publicly available information on these costs (Blockus, 2016; Dolan, 2016), the committee is unable to provide even range estimates for the cost of various URE formats. The costs for various components needed for URE programs will vary depending on the specific conditions on a campus and on campus policy on cost accounting. For example, faculty salaries paid for supervision of summer research vary dramatically at different locations. At some colleges, faculty are paid for teaching a summer course if they serve as a research mentor for a minimum number of students, whereas at other colleges such faculty are considered to be conducting summer research that

___________________

³ See http://nces.ed.gov/programs/coe/indicator_cha.asp [February 2017].

is paid, if at all, through a grant they have secured. In the first case, the faculty salary is a URE program cost, whereas in the second case it is not. At many research universities, it is assumed that research-active faculty will absorb undergraduates into their lab year-round with no compensation, while in some cases supply money follows the student. Individual programs and institutions will need to consider their own circumstances, mission, and traditions in determining what sort of support can be provided. Will the potential value-added of providing UREs outweigh the costs in terms of dollars and institutional satisfaction? Programs that keep students on track to graduation have considerable value in maintaining institutional income from tuition, as well as supporting the long-term goals of the students.

Although in many instances funds can be repurposed to support UREs, particularly CUREs (see Box 8-3), institutions often will want to secure additional resources to start up or expand UREs. Funding avenues that can be explored include internal institutional resources and endowments, state-based funding sources, industry grants and partnerships, federal grants, and grants from private foundations. Many institutions have development offices that can provide information and guidance to those seeking funding, and some institutions have development officers who focus on securing funds for the undergraduate research mandate of the institution. Undergraduate research offices, present on many campuses, often post lists of potential funding sources online so that even those at other institutions can benefit from this information. Funding possibilities include federal agencies such as NSF, the National Institutes of Health, the Department of Education, the Department of Defense, and the National Aeronautics and Space Administration. There are also private sources of funding such as the

Howard Hughes Medical Institute and the American Chemical Society. The WebGuru Guide for Undergraduate Research⁴ offers suggestions of possible funding sources, as well as providing information for undergraduates who are considering research. Individual undergraduate research opportunities with the federal government can now be searched in one location at the new website, http://STEMundergrads.science.gov. Many research-intensive universities provide summer research experiences for students from other schools; a strong undergraduate research office can help students identify and apply for such opportunities.

Opportunities for funding may come in various forms, and creative strategies can be used to generate the resources needed for UREs. Sometimes funds focused on other goals or programs can be supplemented to add support for undergraduate research. In other cases, multiple sources of funding can be combined to begin or sustain a program, or pre-existing resources can be repurposed or leveraged within and outside of the institution. For example, NSF’s Advanced Technological Education program explicitly encourages colleges to partner with nonacademic entities in efforts to improve education in science and engineering.⁵ The program’s website suggests the National Network for Manufacturing Innovation as a potential partner; the network was set up with industry, academic, and federal partners to increase U.S. manufacturing competitiveness by promoting a robust and sustainable manufacturing research and development infrastructure.

One area in which opportunities for the low-overhead launch of new UREs would be particularly welcome is multidisciplinary UREs. These can be structured in multiple ways, one example is the VIP Program described in Chapter 2. Multidisciplinary experiences offer a logical way to exploit the most unique aspect of institutions of higher education, which is the presence of experts in many disciplines under one administrative roof and on one physically contiguous campus. A URE is generally much more flexible than a lecture-based class and can attract people who are passionate about some multidisciplinary topic. Enabling low-cost experiments in this area could unleash much creative activity from both faculty and students.

Such complexities related to costs also need to be recognized by organizations that wish to support UREs. Flexible grants that allow institutions to meet and overcome the often unique challenges for their students are likely to produce the greatest benefits. However, careful evaluation of what seems to work most effectively within and across institutions and among different kinds of student populations should be an integral component of any decisions about how to support such initiatives.

___________________

⁴ See http://www.webguru.neu.edu/undergraduate-research/research-funding/possible-funding-sources [February 2017].

⁵ See http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=5464 [February 2017].

Human Resources

Ultimately, the success of a URE is tied to the personnel taking on the various roles required to design, implement, and sustain URE programs. The human resources include faculty advisors, mentors (if not the same as the faculty advisors), and others who provide support related to curricula, logistics, equipment, and supplies. In addition to identifying people who will play a crucial role in the operation of a URE, it is also important to identify experts (on and off campus) who can share knowledge that can support the design and evaluation of the program. These experts might include individuals with expertise in evidence-based teaching practices, curriculum development, learning sciences, and program evaluation, as well as current program directors and scientists with extensive experience supervising such programs. It may be appropriate to consider faculty from other departments or schools and individuals in business and industry with relevant expertise. Consulting or partnering with these experts can allow URE designers to build more easily on the work of others and to learn from the existing experience and evidence that have been gathered.

Those engaged in designing and running UREs can benefit from access to current professional development opportunities. Advisors and mentors participating in and supporting UREs can learn about pedagogy, facilitating group work, mentoring, and assessment, among other topics. As briefly described in Chapter 5, the quality of mentoring can have an impact on students’ persistence in STEM (Johnson, 2002; Johnson and Huwe, 2003; Liang et al., 2002; Nagda et al., 1998; Pfund, 2016). In particular, a bad mentor can lead to a negative experience, which may motivate mentees to leave the program. Thus, professional development, especially for mentoring, can improve student participation and help faculty learn evidence-based practices that can lead to a more successful program.

Professional development is important for all of the key players involved in the URE, not just for faculty. Institutions can provide opportunities for postdoctoral fellows, graduate students, lab technicians, and even teaching assistants to develop their skills as mentors. These programs can occur at campus centers of learning; through participation in disciplinary society meetings, which now frequently hold workshops on these topics; and at related national conferences such as those organized by the Council on Undergraduate Research.

Space, Equipment, and Shared Resources

Implementing or expanding UREs will, by necessity, place competing demands on existing space; on purchase and maintenance of costly instrumentation, supplies, library and computing facilities; and on the personnel

who must be associated with such enterprises. The problem may be exacerbated further in institutions where there is increasing pressure for individual faculty to find external funding to support some or all of their salaries, as well as the instrumentation and supplies that they need for their research programs. Departmental or institutional policies about use or sharing of space and research-grade instruments for both research and teaching are important considerations when seeking to implement or expand various kinds of UREs. As suggested above, revisiting the institution’s stated vision and mission statements may help focus such discussions.

These discussions should include making plans to ensure that undergraduates have access to relevant journals and online resources as well as the necessary space and equipment. If research with students is not already part of the campus culture, identifying and motivating faculty to undertake such efforts can be challenging; doing so not only can involve large investments of time, but also necessitates re-examining current teaching practices.

However, lack of what are assumed to be required resources need not preclude the development of innovative and sometimes unorthodox opportunities for UREs. Such opportunities may include facilities and support from other parts of the campus and through local, state, and national entities, both public and private. Consortia can facilitate sharing of resources across disciplines and departments within the same institution or among different institutions, organizations, and agencies. Consortia that employ research methodologies in common can share curricula and other teaching/learning modules, research and technical data that students collect, and common assessment tools. Some consortia are able to organize scholarly venues for sharing research results as well (Blockus, 2016).⁶ Such shared materials lessen time burdens for individual faculty and provide a larger pool of students to judge efficacy of the particular approach (Lopatto, 2015; National Academies of Sciences, Engineering, and Medicine, 2015, Appendix B).

Many schools have, or have access to, local field stations that can become the focus of a new or expanded research program (National Research Council, 2014). In other cases, students might use the campus or surrounding community itself as the research environment, taking up issues of conservation, efficient resource utilization, etc., which may be priority concerns of these potential partners. For example, the California State University (CSU) system has in place the “Campus as a Living Lab,”⁷ which engages undergraduate students in research by providing funds for faculty to address basic and applied research questions that are essential and unique to individual CSU campuses, such as the energy efficiency of

___________________

⁶ For another example, see the Phages DB website at http://phagesdb.org [December 2016].

⁷ See http://www.calstate.edu/cpdc/sustainability/liv-lab-grant [February 2017].

a given building. Students need to travel only as far as the boundaries of their home campuses to engage in this kind of research-based work. At CSU schools, any cost savings that result from this research are directed back to the program on each campus, to encourage additional research. Similar innovative undergraduate research efforts are being developed through partnerships with campus entities such as dining services and physical plants, as illustrated by the work of Cathy Middlecamp at the University of Wisconsin–Madison (Kober, 2015, p. 47).

Sharing of research-grade instrumentation, often available through the U.S. national laboratories, can enable student investigations. Increasing numbers of these instruments can be operated remotely by faculty and students. In other cases, laboratories are willing to receive and process samples provided through UREs and return the assays or other results to student researchers (Kober, 2015). Sharing and support from local and regional URE networks and/or consortia is a possibility. As characterized in Chapter 2, there are URE programs that involve multiple institutions and leverage the sharing of resources to improve UREs. Numerous examples discussing some of these options are given by Elgin and colleagues (2016), and other examples appear in Appendix B.

DECISIONS ABOUT IMPLEMENTATION

There are many factors to consider when starting up a URE. Instructors and mentors need to consider information from the literature on UREs and use what is known about how people learn. They will need to assist undergraduates to integrate the experiences, activities, mentoring, and assignments they encounter as they participate in UREs so that the students can make connections to their broader experiences and education. Four principles for design are listed in Figure 3-2: (1) make STEM research accessible and relevant, (2) promote autonomy, (3) learn from each other, and (4) make thinking visible. Attention to these principles can enhance student learning.

In addition, the Council on Undergraduate Research (CUR) has outlined several best practices for UREs based on the apprenticeship model. CUR suggests that undergraduate research should be a “normal” part of the undergraduate experience regardless of the type of institution. It identifies changes necessary to include UREs as part of the curriculum and as part of the culture to support curricular reform, including modifications to the incentives and rewards for faculty to engage with undergraduate research. In addition, CUR points to professional development opportunities specifically aimed at improving the pedagogical and mentoring skills of instructional staff in using evidence-based practices as important for a supportive learning culture (Council on Undergraduate Research, 2012).

Learning from Experience and Evidence: How Do UREs Fit into What Is Known About Student Learning?

UREs require students to make connections and to use the research literature to understand and contextualize their research findings. For students to understand the concepts and context for the research they are doing, they need to make sense of new knowledge by connecting it with prior knowledge and experience. To succeed in STEM, students need to learn how to organize their ideas, rather than holding a repertoire of fragmented, sometimes contradictory or disconnected ideas. Knowledge that is organized and coherent is easier to remember because there are multiple links between items that can aid in recall.

Encouraging students to both generate explanations and revise them as they make sense of their research can promote knowledge integration. These activities can set in motion a process of revisiting STEM-specific issues when they arise in new contexts, such as news articles or public lectures. UREs can foster the development of autonomous learners who sort out their existing ideas and integrate them with new ideas to continue to build coherent understanding. By practicing reflection regularly, students can develop the ability to monitor their own progress and to recognize new conflicts and connections as they arise. As this ability develops, students become more likely to use many of the reasoning strategies essential in STEM fields, such as drawing on evidence and forming arguments to reach conclusions.

The process of reflecting and explaining their reasoning can be crucial to student learning gains (Svinicki and McKeachie, 2011). Reflection is common when STEM professionals keep notebooks in which they record results and identify trends. Instructors and mentors can encourage students to maintain notebooks in which they ask students to include reflections about their struggles to conduct their project and the limitations of their work. In CUREs, instructors can include essay questions to instill a practice of reflection, rather than relying on multiple-choice questions. This approach has the advantage of being both part of the instruction and a source of insights into student progress (Lee et al., 2011).

Groups or teams of students working together can establish a community of learners and provide cognitive and social support for each other. Requiring students to be explicit about what they mean and to negotiate any conflicts that arise can foster metacognition. When instructors make their thinking explicit, it helps give students a sense of the process of conjecture, refinement, redesign, and reconceptualization involved in the research enterprise.

Engagement in UREs can enhance student learning over traditional instruction and improve retention of content knowledge (Cortright et al.,

2003; Johnson et al., 1998, 2007). Additional information about how students learn in UREs can be found in the Chapter 4 discussion of research studies and the Chapter 3 presentation of the committee’s conceptual framework for UREs, which is based on research on how students learn.

Assessing Student Outcomes and Evaluating UREs

Proper assessment requires choosing goals and then designing UREs that target those goals through appropriate content and processes. Assessments should be designed so that they measure the extent to which a program’s goals have been reached. A discussion of choosing goals and assessments can be found in Shortlidge and Brownell (2016). If, for example, the goal of a URE is increased matriculation into graduate programs, then a key component of the program (in addition to experiencing research) may be test preparation and coaching on graduate school applications. Measurement would need to track students over time to learn of their experiences with further education. Similarly, if a goal is to increase STEM knowledge and literacy, a URE may include not only working alongside one or more faculty mentors in a lab, but also additional assigned readings and periodic workshops featuring presentations on concepts and research across STEM disciplines. Measurement might include concept inventories and tests of disciplinary content. Overall, alignment of a planned URE with the various goals and available resources of the institution is a key strategy in offering academic experiences that succeed.

Faculty need to consider up front what type of evaluation will be completed, who will design the assessments, and how to ensure that the measurements are appropriate and informative. Information on evaluation and assessment of UREs can be found in numerous publications, including the following reports: Knowing What Students Know (National Research Council, 2001), Reaching Students (Kober, 2015), and Vision and Change in Undergraduate Biology Education (American Association for the Advancement of Science, 2011).

Another important aspect to consider when designing a URE is whether or not there is a specific intent to contribute to the extant literature on the efficacy of programs of undergraduate research. Although some level of evaluating the URE program is beneficial in all cases, to ensure that there is alignment between the objectives of the experience and the measurable outcomes, some programs are designed to address a specific research question about UREs (e.g., “Does the use of teamwork/collaboration in apprentice-style UREs lead to increases in the communication skills of students?”). In these instances, special considerations must be made during the design of the URE so that the type and quality of the evidence collected will be useful for drawing conclusions. (For a description of evidence type, see Chapters

1 and 7.) For programs designed to evaluate a particular outcome, it is important to identify the pre and post assessments that will be administered and to determine whether the measurements have been validated. In all cases, the local Institutional Review Board must be consulted and appropriate human subjects protections put in place before the assessment begins.

While it is clearly desirable for the design of new types of UREs to be well grounded in education and social science research, asking or requiring every new type of URE to be based upon or informed by education research before it can begin operation or receive funding could stifle creativity. Circumstances may be such that a short-term opportunity or collaboration makes it possible for faculty to quickly develop and test a new type of URE within a discipline, across two or more disciplines, or even across multiple institutions. If the experiment shows promising results, then the effort should be evaluated to understand how and why. After that, the approach can be tested for sustainability, transferability to other disciplines, and scalability.

NATIONAL ORGANIZATIONS THAT SUPPORT URES

This process of improvement can benefit from participation in collaborations and networks with others engaged in similar efforts. Sharing human, financial, and scientific or technical resources can strengthen the broad implementation of effective, high-quality, and more cost-efficient UREs. Strategically designed networks of faculty, institutions, regionally and nationally coordinated URE initiatives, professional societies, and funders can facilitate the exchange of evidence and experience related to UREs. These networks can help provide a venue for considering the policy context and larger implications of increasing the number, size, and scope of UREs. Such networks also could provide a more robust infrastructure to improve the sustainability and expansion of URE opportunities.

It may especially behoove community colleges, as well as geographically isolated and underresourced institutions, to engage in partnerships in order to expand opportunities for more undergraduates to participate in diverse UREs (see, for example, discussions in National Academies of Sciences, Engineering, and Medicine, 2015, and in Elgin et al., 2016). Faculty at community colleges and other institutions focused on teaching may be able to share pedagogical innovations with colleagues involved in these partnerships. Existing networks and consortia of faculty involved with UREs can serve as resources for those new to URE design or implementation (Blockus, 2016); for examples, see the text boxes in Chapter 2.

There are several organizations that focus directly on undergraduate research and cut across disciplines. CUR and the National Conference on Undergraduate Research promote and advocate for all types of UREs,

across all disciplines in STEM and in the humanities.^8,9 CUR has developed an extensive description of Characteristics of Excellence for Undergraduate Research and a related web supplement with specifics on using these characteristics to assess undergraduate research. The Community College Undergraduate Research Initiative provides resources to 38 institutional partners; these resources include introductory workshops and start-up supplies, as well as faculty development opportunities.

Multiple groups focus on increasing opportunities for historically underrepresented students. The Annual Biomedical Research Conference for Minority Students¹⁰ and the Society for Advancing Chicanos/Hispanics and Native Americans in Science¹¹ both sponsor opportunities and provide venues for underrepresented students to present the results of their scientific research and to network with each other, the scientists who mentor them, and other scientists who attend these gatherings. The National Action Council for Minorities in Engineering¹² performs a comparable role for underrepresented students in that discipline. The American Society for Microbiology’s capstone program provides funding to undergraduates from underrepresented minority groups to enhance their ability to present their research.¹³

Societies of STEM research professionals traditionally have served as a platform for leaders and members from their respective STEM fields and subspecialties to present their research, discuss challenges, and scout opportunities in their field. These organizations provide opportunities for professional development and networking among members at regional and national levels. Many disciplinary society meetings invite undergraduate researchers to present their research during poster sessions or flash talks. The opportunity for undergraduates to communicate their research to a broader audience and engage with others aligns with many design characteristics of UREs (see Chapter 3). In addition to providing their meetings as platforms for undergraduate researchers to connect with peers, network with leaders of the field, and learn about other types of research, some disciplinary societies also are playing active roles to support the development and/or refinement of undergraduate teaching materials within their subject domains.

Although some societies have staff, standing committees, and policy

___________________

⁸ See http://www.cur.org [February 2017].

⁹ See http://www.cur.org/ncur_2015 [February 2017].

¹⁰ See http://www.abrcms.org [February 2017].

¹¹ See http://sacnas.org/about [February 2017].

¹² See http://www.nacme.org [February 2017].

¹³ See http://www.asm.org/index.php/component/content/article/25-education/students/142asm-undergraduate-research-capstone-program-ur-capstone-2016?highlight=YToxOntpOjA7czo4OiJjYXBzdG9uZSI7fQ== [February 2017].

statements that focus on educational topics pertaining to preparing the next generation of STEM professionals, relatively few focus directly on the role of UREs in undergraduate education and how their society may influence the discussions, implementation, and expansion of such programs. Professional societies can act to support undergraduate research in many ways. For instance, many societies fund travel grants for undergraduates to attend professional conferences. Some societies engage in undergraduate research on a deeper level. The Committee on the Undergraduate Program in Mathematics of the Mathematics Association of America, for example, prepares and disseminates a curriculum guide that includes a chapter on Undergraduate Research in Mathematics.¹⁴ The chapter provides guidance on building successful programs, mentoring, and communicating results. This association is also responsible for PICMath, a program to prepare mathematical sciences students for industrial careers by engaging them in research problems that come directly from industry.¹⁵

Other types of national groups have focused specifically on UREs. Some of these are discipline-specific, such as On the Cutting Edge, a program managed by the National Association of Geoscience Teachers that has held workshops for faculty on how to engage undergraduates in geosciences research. This association hosts a detailed website with many examples of UREs, as well as resources for learning about pedagogy and practice related to undergraduate research.¹⁶ The Partnership for Undergraduate Life Science Education, which grew out of the report Vision and Change in Undergraduate Biology: A Call to Action (American Association for the Advancement of Science, 2011), consists of a network of biology faculty who work to improve undergraduate biology. This group has prepared a rubric to evaluate the progress of change, one section of which focuses on activities beyond the classroom—mainly undergraduate students participation in research.¹⁷ Also in biology, CURENet is an organization whose stated mission is “a network of people and programs that are creating CUREs in biology as a means of helping students understand core concepts in biology; develop core scientific competencies; and become active, contributing members of the scientific community.”¹⁸

___________________

¹⁴ See http://www.maa.org/sites/default/files/pdf/CUPM/pdf/CUPMguide_print.pdf [December 2016].

¹⁵ See http://www.maa.org/pic-math [December 2016].

¹⁶ See http://serc.carleton.edu/NAGTWorkshops/undergraduate_research/index.html [December 2016].

¹⁷ See http://api.ning.com/files/KFu*MfW7V8MYZfU7LNGdOnG4MNryzUgUpC2IxdtUmucnB4QNCdLaOwWGoMoULSeKw8hF9jiFdh75tlzuv1nqtfCuM11hNPp3/PULSERubricsPacketv2_0_FINALVERSION.pdf [December 2016].

¹⁸ See the CURENet website at http://curenet.cns.utexas.edu [February 2017].

CAMPUS CULTURE AND SYSTEMIC CHANGE

As institutional leaders consider the role of undergraduate research on their campus, they must consider how UREs fit into their institution’s existing mission and culture. Faculty engagement in developing UREs requires significant time and effort and is not likely to be undertaken widely unless departmental and institutional reward systems recognize and reward faculty for the time required to initiate and implement UREs. Decisions to allocate limited funds to move courses, departments, and at times entire programs toward different outcomes may be required. For example, for some institutions it might be a good fit to have CUREs become more widespread and integral components of the departmental curricula. These types of changes will interact in multiple ways with the recognition and incentive systems and professional cultures to which individual faculty, departments, and interdisciplinary programs are accustomed. Changes to the systems and institutional culture might include policies for hiring, promotion, tenure, annual performance reviews, and compensation, along with potential changes in the institutional teaching/research balance. Changes in any or all of these areas can offer new pathways and incentives toward making UREs an integral component of a department’s or institution’s educational mission.

Regardless of institution type, focus of the research effort, and resources available, by emphasizing a student-centered approach, departments or institutions can increase their likelihood of success in improving existing UREs or in expanding the number and diversity of such learning opportunities to the greatest number of students possible. Campuses that cultivate environments that support continuous refinement of teaching programs, based on evidence of student learning and other measures of success, are more likely to be successful in cultivating and sustaining URE programs (for an example, see Box 8-2, above, on The College of New Jersey). Faculty and others who develop and implement such activities need support to be able to embed meaningful assessments into the design of their programs, to undertake the work involved with evaluating their courses or other types of UREs, and to analyze evidence to make decisions about URE design. Where they are available, centers for learning and teaching can provide guidance to URE developers on topics such as pedagogy and assessment. They can also be good venues for faculty to meet colleagues from other schools, departments, and disciplines for sharing education-related experiences and expertise.

To help projects for studying the mechanisms of UREs move forward more smoothly, partnerships can be formed that combine URE developers from the natural sciences and engineering with those engaged in disciplinary-based education research or with colleagues in the social sciences or schools of education who have appropriate expertise in design-

ing experiments involving human subjects. Such partnerships should also include representatives of the campus Institutional Review Board. In addition, intercampus connections such as those between community colleges or other resource-limited institutions and research-intensive universities can improve the prospects for faculty in the former types of institutions to gain access to instrumentation and other resources, share student-generated research data and common assessments, collaborate with colleagues who are undertaking similar programs, and allow both faculty and students to benefit from interactions across more diverse student populations.

An equally important component of such efforts is recognition by departmental and institutional leaders that, as with any scientific research agenda, not all efforts to develop UREs will succeed, at least initially. Pedagogical efforts are more likely to succeed if they are encouraged and supported by academic leaders. Such support is particularly relevant and important for any untenured faculty member who chooses to take the risks associated with URE innovation. This can be done by acknowledging up front the potential for failure and establishing policies and procedures to accommodate initial failures, while simultaneously instilling expectations and pathways for continued improvements and success over time. Such proactive, supportive efforts will likely catalyze many kinds of innovations in the types of UREs that become available in a department or on a campus because they convey the important message that innovation is encouraged and risks will be managed. Similarly, policies must take into account the challenges that arise when efforts are made to scale up a pilot program or adapt a program begun at another institution.

Demonstrating that the leadership of an institution values UREs enough to engage the faculty and other stakeholders in discussions about changing reward systems to account positively for excellence in this realm also can be highly motivating to those who are, or wish to become, involved with such efforts. Allowing quality involvement with undergraduate research to have a role in decisions about tenure, promotion, or continuation of long-term employment contracts sends a powerful message. Restructuring reward systems in this fashion also may benefit the campus more broadly by broadcasting to the larger campus community (including prospective students who may be attracted to enroll and currently matriculated students who may remain because of such policies) about including such practices as an integral component of the institution’s mission.

SUMMARY

This chapter provides many ideas that can be used by those designing or running UREs today. The information presented here is not grounded in the research literature as are other sections of the report; instead it builds

on the knowledge and expertise of the committee and those they have heard about via their information gathering for this study and through their professional networks. The great variation in the types of UREs that can be offered and the groups of students who can participate mean that there are multiple factors to consider in choosing and designing a program. Goals and resources must be carefully considered when choosing the type(s) of URE to use on a given campus and when making decisions about how to implement, assess, and improve UREs. The culture of the campus and the incentives operating on faculty are key considerations, as are the interests and goals of the students. Every campus has a variety of resources that can be reconfigured and repurposed to support UREs, starting with current teaching laboratory facilities and budgets. Creative uses of the local site as the laboratory, exploiting online resources, and working with consortia can open up additional possibilities.

REFERENCES

Flaherty, C. (2014). Faculty work, student success: How The College of New Jersey reimagined what professors can do. Available: www.insidehighered.com/news/2014/10/16/how-college-new-jersey-rethought-faculty-work-student-success-mind [February 2017].

Hatfull, G. (2015). Innovations in undergraduate science education: Going viral. Journal of Virology, 89(16), 8,111-8,113.

Johnson, D.W., Johnson, R.T., and Smith, K.A. (1998). Cooperative learning returns to college: What evidence is there that it works? Change, 30, 26-35.

Johnson, D.W., Johnson, R.T., and Smith, K.A. (2007). The state of cooperative learning in postsecondary and professional settings. Educational Psychology Review, 19(1), 15-29.

Johnson, W. (2002). The intentional mentor: Strategies and guidelines for the practice of mentoring. Professional Psychology: Research and Practice, 33, 89-96.

Johnson, W.B., and Huwe, J.M. (2003). Getting Mentored in Graduate School. Washington, DC: American Psychological Association.

Kober, N. (2015). Reaching Students: What Research Says about Effective Instruction in Undergraduate Science and Engineering. Board on Science Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

Lee, Jr., J.M., Contreras, F., McGuire, K.M., Flores-Ragade, A., Rawls, A., Edwards, K., and Menson, R. (2011). The College Completion Agenda: 2011 Progress Report. New York: College Board Advocacy and Policy Center.

Liang, B., Tracy, A.J., Taylor, C.A., and Williams, L.M. (2002). Mentoring college-age women: A relational approach. American Journal of Community Psychology, 30(2), 271-288.

Lopatto, D. (2015). The Consortium as Experiment. Paper commissioned for the Committee for Convocation on Integrating Discovery-Based Research into the Undergraduate Curriculum. Division on Earth and Life Studies. Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

Moss-Racusin, C.A., Dovidio, J.F., Brescoll, V.L., Graham, M.J., and Handelsman, J. (2012). Science faculty’s subtle gender biases favor male students. Proceedings of the National Academy of Sciences, 109(41), 16,474-16,479.

Nagda, B.A, Gregerman, S.R., Jonides, J., von Hippel, W., and Lerner, J.S. (1998). Undergraduate student-faculty research partnerships affect student retention. Review of Higher Education, 22, 55-72. Available: http://scholar.harvard.edu/files/jenniferlerner/files/nagda_1998_paper.pdf [February 2017].

National Academies of Sciences, Engineering, and Medicine. (2015). Integrating Discovery-Based Research into the Undergraduate Curriculum: Report of a Convocation. Washington, DC: The National Academies Press.

National Research Council. (2001). Knowing What Students Know: The Science and Design of Educational Assessment. J. Pellegrino, N. Chudowsky, and R. Glaser (Eds.). Committee on the Foundations of Assessment, Board on Testing and Assessment, Center for Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

National Research Council. (2014). Convergence: Facilitating Transdisciplinary Integration of Life Sciences, Physical Sciences, Engineering, and Beyond. Committee on Key Challenge Areas for Convergence and Health, Board on Life Sciences, Division on Earth and Life Studies. Washington, DC: The National Academies Press.

Osborn, J.M., and K.K. Karukstis. 2009. The benefits of undergraduate research, scholarship, and creative activity. Pages 41-53 in M. Boyd and J. Wesemann (Eds.), Broadening Participation in Undergraduate Research: Fostering Excellence and Enhancing the Impact. Washington, DC: Council on Undergraduate Research.

Pfund, C. (2016). Studying the Role and Impact of Mentoring on Undergraduate Research Experiences. Paper commissioned for the Committee on Strengthening Research Experiences for Undergraduate STEM Students. Board on Science Education, Division of Behavioral and Social Sciences and Education. Board on Life Sciences, Division of Earth and Life Studies. National Academies of Sciences, Engineering, and Medicine. Available: http://nas.edu/STEM_Undergraduate_Research_Mentoring.

President’s Council of Advisors in Science and Technology. (2012). Engage to Excel: Producing One Million Additional College Graduates with Degrees in STEM. Washington, DC: Executive Office of the President. Available: http://files.eric.ed.gov/fulltext/ED541511.pdf [February 2017].

Shortlidge, E., and Brownell, S. (2016). How to assess your CURE: A practical guide for instructors of course-based undergraduate research experiences. Journal of Microbiology and Biology Education, 17(3), 399-408.

Svinicki, M., and McKeachie, W.J. (2011). McKeachie’s Teaching Tips: Strategies, Research, and Theory for College and University Teachers. Belmont, CA: Wadsworth, Cengage Learning.