National Academies Press: OpenBook
« Previous: Summary
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 11
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 12
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 13
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 14
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 15
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 16
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 17
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 18
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 19
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 20
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 21
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 22
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 23
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 24
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 25
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 26
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 27
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 28
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 29
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 30
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 31
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2017. Indicators for Monitoring Undergraduate STEM Education. Washington, DC: The National Academies Press. doi: 10.17226/24943.
×
Page 32

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

PREPUBLICATION COPY, UNCORRECTED PROOFS 1 Introduction Science and technology are engines of U.S. economic growth and international competitiveness in the global 21st century economy. Leading economists (e.g., Solow, 1957; Minkiw, 2003; Romer, 1990), policy makers, and the public all agree that technological innovation fueled by scientific research is the primary mechanism for sustained economic growth (Xie and Killewald, 2012). As the nation continues to recover from the 2008 economic recession, the science, technology, engineering, and mathematics (STEM) fields are critical drivers for the health of the economy. Hence, a robust, skilled STEM workforce is important for the nation (National Academy of Sciences, National Academy of Engineering, and Institute of Medicine, 2007, 2010). Because undergraduate STEM education plays a central role in developing the STEM workforce, and also contributes to a strong general education for all students, improving the quality of undergraduate STEM education is a national imperative. Some recent trends raise concerns about the health of the nation’s STEM workforce (see Xie and Killewald, 2012). First, scientists’ earnings (adjusted for inflation) have stagnated since the 1960s and have declined relative to those of other high-status, high-education professions, such as law and business, which could discourage individuals from entering or staying in science careers. Second, it has become more difficult for recent science doctorates to obtain any academic position, and the available academic positions are weighted toward more postdoctoral appointments and fewer faculty positions, which could discourage young people from pursuing academic research. Third, U.S. science faces increasing foreign competition as the share of global research conducted in other countries is increasing. These trends could lead to gradual erosion of U.S. dominance in science and a slowdown in the economic growth fueled by technological innovation. To strengthen the nation’s research and technology enterprise in the face of these trends, the President’s Council of Advisors on Science and Technology (PCAST) (2012) recommended producing 1 million additional college graduates with degrees in STEM over the following decade. Recognizing that many students with an interest in and aptitude for STEM, especially females and underrepresented minorities, are not completing degrees in these fields (see National Research Council, 2011; National Academies of Science, Engineering, and Medicine, 2016), the PCAST report called for widespread implementation of strategies to engage, motivate, and retain diverse students in STEM. Such strategies are beginning to emerge from a growing body of relevant research, but they have not yet been widely implemented (see National Research Council, 2012a; National Academies of Sciences, Engineering, and Medicine, 2016a). Many initiatives to improve the quality of undergraduate STEM education are now under way. Some focus on the national level, others involve multi-institution collaborations, and others take place on individual campuses. For example, the interagency Committee on STEM Education of the National Science and Technology Council (2013) developed a STEM Education 5-year strategic plan that identified improving the experience of undergraduate students as a priority area for federal investment. Within this broad goal, the strategic plan identified four priority areas: (1) promoting evidence-based instructional practices; (2) improving STEM experiences in community colleges; (3) expanding undergraduate research experiences; and () advancing success in the key gateway of introductory mathematics. Other initiatives include the 1-1

PREPUBLICATION COPY, UNCORRECTED PROOFS undergraduate STEM initiative of the Association of American Universities;1 a workshop and sourcebook on undergraduate STEM reform of the Coalition for Reform in Undergraduate Education;2 and the Partnership for Undergraduate Life Sciences Education, or PULSE.3 At present, policy makers and the public do not know whether these various federal, state, and local initiatives are accomplishing their stated goals and achieving nationwide improvement in undergraduate STEM education. This is partly due to a lack of high-quality, national data on undergraduate STEM teaching and learning. A recent study of barriers and opportunities for 2-year and 4-year STEM degrees (National Academies of Sciences, Engineering, and Medicine, 2016a) highlighted the mismatch between currently available datasets and the realities of student trajectories. Although students today often transfer across institutions, stop in and out of STEM programs, and attend classes part-time, existing surveys rarely capture these trends. The study committee concluded that existing data collection systems (national, state, and institutional) were often not structured to gather the information needed to understand the quality of undergraduate education. Anticipating these challenges, the President’s Council of Advisors on Science and Technology (2012) recommended that the National Academies of Science, Engineering, and Medicine (National Academies) develop metrics to evaluate undergraduate STEM education. In response, the National Science Foundation charged the National Academies to conduct a consensus study to identify objectives for improving undergraduate STEM education and to outline a framework and set of indicators to document the status and quality of undergraduate STEM education at the national level over multiple years: see Box 1-1 for the full study charge. INTERPRETING THE STUDY CHARGE As it began its work, the committee identified a number of important issues in the study charge and within the broader social, political, and historical context that led to this study. In constructing a shared understanding of these issues, the committee was able to calibrate its interpretations of key terms and phrases in the charge as well as other terms and phrases that emerged over the course of its deliberations. Throughout this chapter, the committee identifies the issues emerging in its interpretation of the study charge. For each one, it shares definitions of terms and, as appropriate, discusses the context in order to help ground the committee’s proposed indicator system. Underlying this work is the committee’s vision for undergraduate STEM education. Vision In developing a conceptual framework and indicators to monitor improvement in undergraduate STEM education, the committee envisioned what such improvement would look like. In this vision, students—from all walks of life and with all types of experiences and backgrounds—would be well prepared to help address global, societal, economic and 1 See https://stemedhub.org/groups/aau/about [July 2017]. 2 See https://www.aacu.org/pkal/sourcebook [July 2017]. 3 See http://www.pulsecommunity.org/ [July 2017]. 1-2

PREPUBLICATION COPY, UNCORRECTED PROOFS technological challenges. Students would have the STEM background to become successful in the careers of today as well as those of tomorrow as U.S. society continues to become increasingly diverse, global, and interconnected. Among these well-prepared graduates, some would become professional scientists and engineers, conducting research and developing new technologies to support sustained economic growth. At the same time, the committee envisions that all students, not merely those who pursue STEM degrees and careers, would have both access and exposure to high-quality STEM education, to support the development of STEM literacy, a relatively new concept. The committee’s adoption of this concept was informed by a recent report (National Academies of Science, Engineering, and Medicine, 2016b) that identified several aspects of science literacy: (1) the understanding of scientific practices, such as the formulation and testing of hypotheses; (2) content knowledge, including concepts and vocabulary; and (3) understanding of science as a social process, for example, the role of peer review. In keeping with these expanding definitions of science literacy, the committee recognized that “engaging in science—whether using knowledge or creating it—necessitates some level of familiarity with the enterprise and practice of science” (National Academies of Science, Engineering, and Medicine, 2016b, p. 11). Students’ ability to use STEM knowledge (for example, as citizens) or create STEM knowledge, then, will require an analogous familiarity with the enterprises and practices of science and technology and engineering and mathematics. The application of the term “literacy” to these disciplines “signifies something like ‘knowledge, skills, and fluency’ within [these] particular domain[s]” (National Academies of Sciences, Engineering, and Medicine, 2016b, p. 17). In the committee’s vision, STEM literacy, along with lifelong interest in STEM, is important for all graduates, regardless of their field of study, both in the workplace and outside of work. In the workplace, STEM knowledge and skills, such as those encompassed within the concept of STEM literacy, are useful across a range of occupations beyond those of scientists and engineers (Carnevale, Smith, and Melton, 2011; National Science Foundation, 2014). Outside of work, people can draw on their science literacy for making decisions that could involve science (e.g., decisions about personal health or voting on environmental issues), although such knowledge is only one of many factors contributing to their decision making (National Academies of Science, Engineering, and Medicine, 2016b). Given that access to high- quality science learning experiences can facilitate and support the development of science knowledge bases, the committee expects that analogous exposure to high-quality STEM learning experiences in undergraduate education will facilitate and support the development of STEM knowledge bases that graduates may choose to draw on when making important life decisions. However, the committee did not reach consensus on exactly what level of exposure (e.g., completion of a certain number or type of courses or learning experiences outside the classroom) would support STEM literacy. For some students, STEM literacy might require exposure to adult basic education, adult literacy, or vocational programs offered in the community. However, measures of such programs would fall outside of the committee’s charge to develop indicators of the status and quality of undergraduate STEM education. The committee envisions that exposure to STEM concepts and processes can help individuals to make sense of the world around them, enabling the skills and dispositions needed to participate actively in a democracy. Because of its commitment to these values, the committee sees the goals for undergraduate education presented later in this chapter as working in service of ensuring STEM literacy for all undergraduates, not just those majoring in STEM fields. 1-3

PREPUBLICATION COPY, UNCORRECTED PROOFS Given the increasingly diverse and global nature of U.S. society, the committee envisions that STEM education will embrace approaches that increase representation of diverse populations in STEM careers. All undergraduate institutions will provide equitable STEM educational practices, both inside and outside the classroom (curricular and co-curricular), ensuring that all students have the opportunities and support they need to reach their potential. In addition, faculty members, staff, and administrators will have the knowledge, skills, and understanding of evidence-based teaching and learning methods to deliver a 21st century, inclusive STEM curriculum and co-curriculum, and students will have clear pathways into and through STEM programs of learning. A Focus on the National Level Given its charge, the committee’s work focused on national-level indicators. As noted above, many initiatives are currently under way to monitor and improve undergraduate STEM Education. However, these initiatives tend to gather detailed, local data that are appropriate for local feedback and improvement; they are not appropriate for representing STEM education phenomena broadly and nationally across 2-year and 4-year institutions. For example, the PULSE vision and change rubrics are designed to evaluate life science departments’ progress toward specified reform principles (American Association for the Advancement of Science, 2011). Department-level leaders (current or former department chairs or deans) voluntarily use the rubrics for self-study and improvement in terms of 66 different criteria across five areas: curriculum alignment, assessment, faculty practice/faculty support, infrastructure, and climate for change. Analyzing rubric data from a sample of life sciences departments,4 Brancaccio-Taras and colleagues (2016) concluded that the rubrics constitute a valid and reliable instrument for evaluating departmental change across different institution types. The authors also concluded that their analysis of rubric data from the 26 institutions that responded to all five groups of rubrics provided “baseline knowledge and insights about the state of the adoption of the recommendations of the Vision and Change report” (Brancaccio-Taras et al., 2016, p. 11). (The authors are currently working to gather data from a larger number of departments.) Although the committee views the currently available baseline data as a valuable snapshot of quality improvement in some life sciences department, these data are not nationally representative, do not include other disciplines, and do not constitute national indicators of improvement in undergraduate STEM. The national-level data needed for the committee’s proposed indicators differ from the PULSE rubrics or other fine-grained data designed to guide local improvement efforts. In K-12 education, for example, teacher observation instruments can be designed to provide detailed data that provide very specific feedback to teachers for improving their practices. However, because schools and districts seek instruments that can provide a picture of teachers’ work across multiple content areas and grade levels, they often design more “course-grained” observation instruments to provide more global data (Hill and Grossman, 2013). One result is that three- fourths of the 15,000 teachers responding to a recent survey indicated that their most recent 4 The authors invited all members of the PULSE community (which includes 2- and 4-year colleges, regional comprehensive universities and research universities) to submit their rubric data. The respondents provided varying amounts of data: 26 institutions provided rubric data across all five areas, 57 provided data on curriculum alignment, 35 on assessment, 49 on faculty practice/faculty support, 28 on infrastructure, and 32 on climate for change. 1-4

PREPUBLICATION COPY, UNCORRECTED PROOFS evaluations failed to identify areas for improvement (Weisberg et al., 2009, cited in Hill and Grossman, 2013). The granularity of data matters, and the global data needed for national indicators may not be useful for informing improvement by individual instructors, STEM departments or programs (Wilson and Anagnostopolous, in press). Equity, Diversity, and Inclusion Given the national need for a robust supply of STEM professionals for technological innovation and sustained economic growth, another important dimension of this study is the underrepresentation of certain groups in STEM, involving issues of equity, diversity, and inclusion. Equity refers to the fair distribution of opportunities to participate and succeed in education for all students. Diversity focuses on the differences among individuals, including demographic differences such as gender, race, ethnicity, and country of origin. Inclusion refers to the processes through which all students are made to feel welcome and are treated as motivated learners. The committee views equity as a central element of quality in undergraduate STEM education and considers measurement of equity as essential to measuring and improving quality. From an accountability perspective, equity in undergraduate STEM education is the achievement of proportional representation of all demographic groups in terms of access, retention, degree completion, and participation in enriching STEM educational programs, experiences, and activities that prepare students to enter the STEM workforce. The committee uses the word demographics broadly to capture the full spectrum of diversity in the population. Such diversity includes, but is not limited to, diversity in socioeconomic level, gender identity, race and ethnicity, religion, first-generation in college, marital and parental status, veteran status, disability status, and age. The National Science Foundation (2017) has identified certain demographic groups as “underrepresented” because their representation in STEM education and employment is not proportional to their representation in the national population. These underrepresented groups include persons with disabilities, women in some STEM disciplines, and three racial and ethnic minority groups: Hispanics, blacks, and American Indians. Goals and Objectives The committee defines a goal as an intended outcome representing improvement in undergraduate STEM education. It is stated in general terms and covers a long time frame. Because achieving long-term goals involves the work of multiple stakeholders at different levels in the undergraduate education system (e.g., classroom, institution, state higher education system), the committee’s goals are stated as action verbs but do not identify a specific actor as the subject. The committee defines objectives are more specific and measurable steps toward achieving goals. Measures and Indicators The committee defines a measure as a value that is quantified against a standard at a specified time. An indicator is a specific type of measure that provides evidence that a certain condition exists or that specifies how well certain results or objectives have or have not been 1-5

PREPUBLICATION COPY, UNCORRECTED PROOFS achieved (Brizius and Campbell, 1991). It goes beyond raw statistics or data to provide easily understandable information that can be used to guide educational policy and practice (National Research Council, 2014). After considering various definitions of an educational indicator (e.g., Oakes, 1986), the committee adopted Planty and Carlson’s (2010) conceptualization of an educational indicator and an indicator system. In this definition, an educational indicator has three key characteristics. First, it attempts to represent the status of a specific condition or phenomenon. For example, it may measure student achievement, dropout rates, crime in schools, or another aspect of education. Second, it typically is quantitative (Planty and Carlson, 2010, p. 4): Indicators are created from data (i.e., observations collected in numerical form) and presented in the form of numbers and statistics. Statistics are numerical facts, but by themselves are not indicators . . . Indicators combine statistics with purpose, meaning, and context to provide useful information about a condition or phenomenon of interest. Third, an indicator has a temporal component, meaning that it might not only indicate the status of a condition or phenomenon at a given time, but also can represent change in the condition over time. An educational indicator may be either a single statistic or a composite statistic. Single- statistic indicators measure a specific condition of an education system (e.g., an institution’s student enrollment and number of Pell grant recipients). Composite indicators combine single statistics to depict a relationship between two or more aspects of the education system; examples include student-faculty ratio and student readiness (Planty and Carlson, 2010). Because a lone indicator rarely provides useful information about complex conditions or phenomena, indicator systems are designed to generate more comprehensive and, therefore, more useful information about conditions. More than just a collection of indicator statistics, an educational indicator system measures the system’s inputs, processes, and outputs. In education, such inputs may include fiscal and material resources, instructor quality, and student background. Processes may include instructional quality and institutional context and structure, while outputs may include student achievement, participation, and attitudes and aspirations (Shavelson et al, 1989; Odden, 1990, pp. 24-25). A high-quality indicator system not only measures these individual components, but also suggests how they work together to produce an overall effect. In the following chapters of this report, the committee proposes an educational indicator system reflecting this definition. The system is based on a conceptual framework that views undergraduate STEM education as a system and considers how the components (inputs, processes, the environment) work together to produce an effect on the desired student outcomes. The committee emphasizes that its proposed indicators should be viewed in concert with one another to provide insight into the overall quality of undergraduate STEM education (see National Research Council, 2014). Though individual indicators can provide discrete markers of progress toward improvement in undergraduate STEM, the committee also recognizes that telling an “end-to-end story” requires the creation of a coherent framework demonstrating how relationships across indicators facilitate achieving the identified objectives. This report provides such a framework, identifying overarching and mutually reinforcing goals that are aligned with more specific objectives for improving undergraduate STEM education, along with a set of indicators to monitor progress toward the objectives. 1-6

PREPUBLICATION COPY, UNCORRECTED PROOFS Undergraduate STEM Education For the purposes of this report, the committee defines undergraduate STEM education as undergraduate education in the natural and social sciences, technology, engineering, and mathematics. This definition follows the National Science Foundation (2016) definition of the STEM fields, which includes the social sciences. In keeping with the committee’s charge to develop indicators for 2-year and 4-year STEM, the definition encompasses programs of study leading to bachelor’s degrees, associate’s degrees, and certificates at all types of public, private, for-profit, and non-profit institutions of higher education, in both brick and mortar and online environments: see Box 1-4. This definition also includes workforce development programs that prepare students for “middle skill” jobs, defined as those jobs that require less educational preparation than a bachelor’s degree, but some education or training beyond a high school degree (Holzer and Lerman, 2007; Rothwell, 2013). Finally, the committee’s definition includes introductory STEM courses that students may take as part of general education requirements, regardless of their major field of study. Evidence-Based STEM Educational Practices and Programs In the committee’s view, improving the quality of undergraduate STEM education will require wider use of “evidence-based STEM educational practices and programs.” A growing body of research (see National Research Council, 2012 and National Academies of Sciences, Engineering, and Medicine, 2016a) has begun to identify effective teaching practices and co- curricular programs that support students’ mastery of STEM concepts and skills and their retention in STEM programs. Based on this research, the committee defines evidence-based STEM educational practices and programs as those meeting at least one of the following criteria:  the preponderance of published literature suggests that it will be effective across settings or in the specific local setting, or  the practice is built explicitly from accepted theories of teaching and learning and is faithful to best practices of implementation, or  locally collected, valid, and reliable evidence, based on a sound methodological research approach, suggests that the practice is effective. MEASURING COLLEGE QUALITY IN AN ERA OF ACCOUNTABILITY The committee defines improvement as progress toward the committee’s vision, goals, and objectives for undergraduate STEM education (see Chapter 2). To define quality, the committee adapted a definition of higher education quality from Matsudaira (2015), which, in turn, reflects earlier work on quality improvement in health care delivery (Institute of Medicine, 2001). Specifically, the committee defines quality in higher education as the degree to which exposure to STEM educational offerings increases the likelihood of desired educational outcomes. This definition focuses on the causal effect that exposure to some STEM educational experience (e.g., the physics program at college A; an innovative remedial mathematics course at college B) has on advancing valued outcomes (Matsudaira, 2015; see also Matchett et al., 2016). 1-7

PREPUBLICATION COPY, UNCORRECTED PROOFS In this case, the valued outcomes of undergraduate STEM include mastery of STEM concepts and skills and attainment of STEM credentials. In developing this definition of quality, the committee considered the larger social and political context. Responding to soaring college costs, high attrition rates, and rising student debt, parents, employers, policy makers and taxpayers are asking questions about the quality of higher education: Are students learning? Are graduates earning? What is the value of higher education? Each of these questions is complex, and the committee acknowledges that they cannot be answered with single indicators. Nevertheless, the goals, objectives, and indicators presented later in this report contribute to answering these questions. Employment Outcomes In response to growing calls for accountability, one of the most widely used methods for measuring the “value” of a college or university is to assemble and analyze data on graduates’ earnings. However, research has demonstrated that both graduation rates and postgraduation earnings vary widely, depending on the type and selectivity of the institution and the characteristics of incoming students (National Academies of Sciences, Engineering, and Medicine, 2016a; Matsudaira, 2015). Although economists are beginning to develop methods to adjust graduates’ earnings to account for the characteristics of incoming students, these methods are not yet fully developed, and further research would be needed to develop uniform quality measures (Matsudaira, 2015). In addition, graduates’ earning are influenced by labor market demand, and the wage premium for STEM graduates varies by time, place, and field in ways that are characteristics of a market economy. Furthermore, many STEM majors enter occupations that are not traditionally considered part of the STEM workforce (National Science Foundation, 2016), but their STEM knowledge may indeed contribute to their earnings (Carnevale et al., 2011): it is not practical to precisely identify these workers and measure this contribution. Moreover, individuals who have received STEM education may possess unobserved attributes that make them incomparable to their peers without STEM education. For all of these reasons, some experts and leaders in higher education agree that postgraduation earnings alone are not a suitable measure of institutional quality (e.g., Matchett et al., 2016). Nevertheless, many states have already implemented performance-based funding systems that reward institutions on the basis of the average earnings of graduates Another, related method for measuring an institution’s “value” focuses on the extent to which graduates find jobs related to their chosen fields of study. This method, however, does not fully address the measurement challenges described above. U.S. colleges and universities intentionally offer a variety of majors to meet students’ varying interests and demands. Some STEM majors (such as engineering and nursing) are more closely tied to future occupations than others (such as social science and biology). Thus, across STEM fields, there is a large variation in the flow of students from STEM majors to STEM occupations (Xie and Killewald 2012), and STEM education can be a good preparation for non-STEM careers. Labor market demand, external to higher education, influences whether STEM graduates find any type of job, a job in their specific STEM discipline, or a job in another STEM discipline. A graduate’s employment prospects are also influenced by the geographical location and movement of companies and academic research organizations, changes in domination of certain industries and technologies, and shifting demand for different skills as new technologies emerge and others become obsolete. 1-8

PREPUBLICATION COPY, UNCORRECTED PROOFS We know that rates of employment and unemployment vary by discipline: see Box 1-2. In 2013, most STEM professionals (93.3 percent) were employed in their field of study, with only 6.7 percent reporting that they worked in a job outside the field of their highest degree because a job in their field was not available (National Science Foundation, 2016). Within this average, however, the rate of working outside one’s field of study varied by discipline, with higher rates in the social, life, and physical sciences and lower rates in engineering and computer and mathematical sciences (see Box 1-2). In the committee’s view, care should be taken in interpreting these different job placement rates. They may reflect differences in employers’ skill demands, the intended purpose of the students’ majors, or students’ selections of fields. Thus, one should not attribute the observed differences in job placement rates simply to differences in the overall national quality of the undergraduate education programs in each discipline. Even when STEM majors enter occupations outside the STEM workforce, the knowledge and skills they developed in their undergraduate STEM programs contribute to the national economy (see “Vision,” above). In light of these complexities, the committee does not propose any overarching goals, more specific objectives, or indicators related to placement in STEM jobs. The STEM Workforce The committee defines the STEM workforce to include science and engineering occupations (currently about 5.7 million people) and science and engineering-related occupations (currently about 7.4 million people; National Science Foundation, 2016). These two groups of occupations have been carefully defined and studied by the National Science Foundation (NSF) (2014, 2015, and 2016). In this definition (National Science Foundation, 2016), the science and engineering occupations include computer and mathematical scientists; biological, agricultural, and environmental life scientists; physical scientists (e.g., physicists, chemists, geoscientists); social scientists (e.g., psychologists, economists, sociologists); engineers; and postsecondary teachers in science and engineering fields. The science- and engineering-related occupations include health care workers (e.g., physicians, audiologists, nurses); science and engineering managers (e.g., engineering managers, natural and social science managers); science and engineering precollege teachers (e.g., K-12 science teachers); technologists and technicians in science and engineering; and other science and engineering-related occupations (e.g., actuaries, architects). The committee notes that, although the concept of a “STEM workforce” is widely used and has been referenced in law, there is no consensus on how it is defined. Various reports use different definitions, leading to divergent and sometimes conflicting conclusions about the size and other characteristics of the STEM workforce: see Box 1-2. Furthermore, the STEM workforce is heterogeneous; it is composed of many different “sub-workforces” that can be characterized by field of degree, occupational field, the education level required, or some combination of these elements. Several approaches to defining and measuring the STEM workforce or the science and engineering workforce have been proposed in recent years. One approach simply counts any job held by an individual with at least a bachelor’s degree in science or engineering as part of the science and engineering workforce; with this approach, the workforce totals 19.5 million people (National Science Foundation, 2014). In another approach, based on surveys of college graduates about their job requirements, 16.5 million people indicated that their position required 1-9

PREPUBLICATION COPY, UNCORRECTED PROOFS a bachelor’s level degree of science and engineering knowledge (National Science Board, 2014). In yet another approach, Rothwell (2013) analyzed data on skill and knowledge requirements from the Occupational Information Network (O*NET) national database and found that 26 million jobs required significant STEM expertise. There is little agreement across these various approaches. Given this lack of consensus, the committee followed the National Science Foundation’s approach, the first one above, defining the STEM workforce to include science and engineering occupations (currently about 5.7 million people) and science- and engineering- related occupations (currently about 7.4 million people; National Science Foundation, 2016). Learning Outcomes In the Chapter 2, the committee identifies increasing students’ mastery of STEM concepts and skills as one of three overarching goals for improving the quality of undergraduate STEM education. However, the committee does not propose any indicators that would directly measure student learning because of the complexities discussed here. There is no simple way to address questions about whether students are acquiring the STEM concepts, skills and abilities that will serve them for their lives after college, for several reasons. First, expectations for the holder of an associate’s degree or certificate are different than those for the holder of a baccalaureate degree. Second, faculty, employers, professional societies, accreditation agencies, testing companies, and curriculum committees all have different answers about the ideal and acceptable levels of proficiency, and, more fundamentally, about what concepts and skills should be measured for proficiency. These groups have launched a variety of efforts to define proficiency, some of which focus on core knowledge and skills for all 2-year and 4-year graduates, across all fields of study (e.g., Association of American Colleges and Universities, 2007; Lumina Foundation, 2015), while others focus on specific disciplines (e.g., Arum et al., 2016). Leaders in life sciences education, for example, have identified core concepts, competencies, and disciplinary practices for “biological literacy” in undergraduate biology (Brewer and Smith, 2011). Third, the STEM disciplines are characterized by rapid discoveries and the ongoing development of new knowledge and skills. Within and across these disciplines, new sub- disciplines and interdisciplinary fields are continually being created, bringing differing views about the core knowledge and skills that define successful learning. Fourth, in U.S. higher education, there have never been national tests, graduation standards, or uniform STEM curricula. These would be incompatible with the tradition of state and system-level autonomy in public higher education and with the diversity of public, private nonprofit, and private for-profit institutions that provide undergraduate STEM education. And fifth, some learning outcomes are ways of seeing problems, analyzing them, and solving them, with appropriate tools and with collaborations among diverse group (see, e.g., Association of American Colleges & Universities, 2007). These outcomes are not knowledge about specific content areas, nor can they be easily translated to a national-level measure. Because of these complications and practical difficulties, the committee does not propose any indicators that would directly measure student learning. However, the committee does target increased acquisition of STEM concepts and skills as an overarching goal for improving undergraduate STEM education (see Chapters 2 and 3). In the future, more detailed, automated, proficiency exams and fine-grained records of accomplishment may be available. At that time, it will be important to revisit the conceptual 1-10

PREPUBLICATION COPY, UNCORRECTED PROOFS framework and indicators proposed in this report. The committee envisions that its recommended indicator system will undergo continuous improvement and updating (see Chapter 7). Goals of the Indicator System In light of the pressures for accountability and the complexity of measuring quality described above, the committee stresses that the primary goal of the indicator system is to allow federal agencies to monitor the status and quality of undergraduate STEM education over time (based on data aggregated from individual institutions). For such monitoring, the proposed indicator system can use data from nationally representative samples of institutions and students (see Chapters 6 and 7); in contrast, a system designed for accountability or ranking would require data from the entire universe of 2- and 4-year institutions. The committee envisions that institutional data collected by federal agencies to inform the national indicators could also be accessible to, and used by, individual institutions, state higher education systems, or consortia of institutions to monitor their own programs over time for the purpose of improvement. Although such accessibility might also allow institutions, consortia, states, or individuals to compare, and rate institutions with the goal of holding them accountable, that is not the intended purpose of the indicator system. Rather, the committee expects that the indicator system will be used by the National Science Foundation and other federal agencies to monitor nationwide progress toward improving the quality of STEM undergraduate education. The committee also anticipates that the interagency committee on STEM education will use the indicator system as it works to advance the objectives for undergraduate STEM education identified in the federal STEM education 5-year strategic plan of the National Science and Technology Council. STUDY APPROACH AND ORGANIZATION OF THE REPORT To address its charge, the committee met seven times over the course of the study. The meetings were organized to allow the committee to consider the testimony of expert presenters, as well as privately deliberate the weight of existing evidence. In Phase I, the committee gathered and reviewed a wide catalog of literature on discipline-based education research, change strategies for improving the quality of undergraduate STEM. It also obtained information on existing systems for monitoring the quality of undergraduate education, both generally and in STEM specifically by inviting outside experts to speak to the committee and convening a workshop in February, 2016 (see Appendix A). Drawing on these sources of evidence and its own expert judgment, the committee developed a draft report with a conceptual framework of goals and objectives for undergraduate STEM, releasing it for public comment in August, 2016. In addition to soliciting comments and feedback online, the committee convened a public workshop in October, 2016 to obtain further input (see Appendix A for a summary of comments received and committee responses). In Phase II of the study, the committee met four times in closed session to deliberate on the public comments, revise the framework of goals and objectives, and develop indicators of progress toward the objectives. Based on the vision presented above, and its deliberations about the preliminary conceptual framework created in Phase I, the committee identified three overarching goals for STEM education: 1) Increase Students’ Mastery of STEM Concepts and Skills by Engaging 1-11

PREPUBLICATION COPY, UNCORRECTED PROOFS them in Evidence-based STEM Educational Practices; 2) Strive for Equity, Diversity, and Inclusion; and 3) Ensure Adequate Numbers of STEM Professionals. These three goals are discussed in greater detail in Chapter 2. In Phase II, the committee also reviewed additional literature as it deliberated on the proposed indicators and developed conclusions and recommendations for research and data collection to develop the indicator system. Throughout the study process, committee members drafted sections of text, which were shared, reviewed, edited, and revised across members of the entire committee. The report is organized around the major tasks outlined in the committee’s charge. Chapter 2 presents the conceptual framework for the indicator system; Chapters 3, 4, and 5 discuss the committee’s three goals for improvement in undergraduate STEM, along with objectives and indicators to measure progress toward those goals. Chapter 6 reviews existing monitoring systems and data sources related to undergraduate STEM education, and Chapter 7 discusses alternative approaches to implementing the indicator system. 1-12

PREPUBLICATION COPY, UNCORRECTED PROOFS BOX 1-1 Study Charge An ad hoc committee will conduct a study to identify objectives for improving undergraduate STEM education and to outline a framework and set of indicators intended to document the status and quality of undergraduate STEM education at the national level over multiple years. The committee's work will progress in 2 phases, with an interim report after Phase I and a final report at the conclusion of Phase II. The committee will use the strategic objectives of the federal STEM education strategic plan as a starting point, but will also consider whether additional objectives need to be tracked in order to determine the status of undergraduate STEM education over time. At NSF's request, the study will focus in particular on the first two years of undergraduate education. In Phase I, the committee will:  identify objectives for improving undergraduate STEM education at both community colleges and 4-year institutions building from the objectives for higher education outlined in the federal strategic plan to coordinate federal investments in STEM education and emphasizing the first 2 years of undergraduate education;  review existing systems for monitoring undergraduate STEM education; and  develop a conceptual framework for the indicator system. In Phase II of the study, the committee will:  develop a set of indicators that are linked to the objectives identified in Phase I;  identify existing and additional measures needed for tracking progress toward the objectives identified in Phase I;  discuss the feasibility of including such measures in existing programs of data collection;  identify additional research that would be needed to fully develop the indicators needed to track progress toward the objectives developed in Phase I; and  make recommendations regarding the roles of various federal and state institutions in supporting the needed research and data collection for an evaluation of progress. 1-13

PREPUBLICATION COPY, UNCORRECTED PROOFS BOX 1-2 Debates about Supply and Demand for STEM Professionals Since the 1950s, scientists, engineers, employers, and policy makers have periodically raised alarms about the possibility of impending shortfalls in the supply of STEM professionals. One of the most visible reports, Rising Above the Gathering Storm (National Research Council, 2007), conveyed deep concern about the future supply of U.S. scientists and engineers at a time when other nations are rapidly advancing in science and innovation. Lowell and Salzman (2007) disputed these arguments, finding that the U.S. supply of graduates in science and engineering was not only large, but also considered among the best in the world. Based on their analysis of available data, they also argued that the U.S. education system was producing far more science and engineering graduates than needed for the available job openings. Studies by economists have also found little evidence of a market shortage of scientists (see, e.g., Butz et al., 2003). More recently, Xie and Killewald (2012) conducted a detailed analysis of multiple datasets, finding little evidence of either an oversupply or a shortage of U.S. scientists and engineers. They found that over the past four decades the number of graduates in the biological and physical sciences, mathematics, and engineering had grown, although at a slow rate. Contrary to the claims of an oversupply, most of these graduates, especially bachelor’s and master’s degree recipients, found jobs related to their training in these fields.5 However, the real earnings of basic scientists generally declined over the same period, challenging the claims of a market shortage of basic scientists (a shortage should have resulted in rising wages). For more than two decades, the unemployment rate in STEM occupations has been considerably lower than the general unemployment rate (National Science Foundation, 2016), a reflection of sustained demand for STEM professionals. In 2013, a survey of STEM professionals with at least a 4-year degree in a STEM field found that most (96.2 percent) were currently employed (National Science Foundation, 2016). Furthermore, studies have found that individuals who earned long-term certificates6 in STEM-intensive fields, such as health, nursing, and transportation, gained positive economic returns on their educational investments in the form of higher wages and lower unemployment (Dadger and Weiss, 2012; Stevens, Kurlaender, and Grosz, 2014). These recent data support Xie and Killewald’s (2012) view that the supply of and demand for STEM professionals overall is roughly in balance, with neither an oversupply nor a shortage. However, demand for STEM graduates is not monolithic, varying in response to market forces and business cycles. For example, unemployment among STEM professionals with at least a 4-year degree spiked in the recessions of the early and late 2000s, and in 2014 the unemployment rate remained higher, at 3.4 percent, than the pre-recession rate of 2.2 percent in 2006 (National Science Foundation, 2016). In addition, the rates of both unemployment and employment within one’s field of study vary across the different STEM disciplines. In a 2013 survey, STEM professionals with at least a 4-year degree were asked if they were working outside the field of their highest degree because a job in that field was not available. On average across all disciplines, 6.7 percent of graduates reported working outside their field of study, but the rate varied in different disciplines (National Science Foundation, 2016): see Table 1B-1 5 In their analysis, Xie and Killewald (2012) did not include graduates in the social sciences. They note that social science graduates were less likely to be employed in jobs related to their training than were their peers in the biological and physical sciences, mathematics, and engineering. 1-14

PREPUBLICATION COPY, UNCORRECTED PROOFS TABLE 1B-1 Rates of Unemployment and Working Outside One’s Field of Study, 2013 Discipline Percentage Unemployed Percentage Working Outside Field of Study Life Sciences 3.3 9.4 Computer and Mathematical 3.1 4.1 Sciences Physical Sciences 4.5 8.3 Social Sciences 3.3 11.8 Engineering 2.5 4.6 All STEM Disciplines 3.9 6.7 SOURCE: National Science Foundation (2016) Appendix Table 3-8 and Table 3-12. 1-15

PREPUBLICATION COPY, UNCORRECTED PROOFS BOX 1-3 The Potential of Online Education The committee’s study was conducted at a time of rapid growth in distance and online higher education, fueled by many of the same communications technologies that enable the globalization of STEM. Some experts envision that inexpensive, widely accessible online courses and programs will lower the costs and improve the quality of undergraduate education. For example, according to Bonvillian and Singer (2013, p. 23): The idea has been growing that universities will change dramatically, and perhaps largely fade away, under the spread of online education increasingly enabled by improvements in broadband Internet access and new mobile devices. Recent years have also seen advances in the science of learning that are enabling society and researchers to look at new education approaches. The accumulating evidence challenges the model that has long dominated higher education: the sage on the stage; that is, the lecture. To date, however, there is little evidence that purely online education is as effective for supporting student learning of STEM concepts and processes as face-to-face education. Distance education or distance learning is defined as the education of students independent of physical presence in a traditional classroom or campus setting (Maeroff, 2003). Over the past two decades, distance education (i.e., online courses and online degree programs) has increased in representation within the undergraduate education landscape (Radford, 2011; Ginder and Stearns, 2014). This growth has been significantly influenced not only by the exponential rise in the development and use of digital communication tools, but also by four more specific factors: (1) to meet students’ demands for flexibility, (2) to widen access to disadvantaged students, (3) to increase course availability, and (4) to increase student enrollment (Parsad and Lewis, 2008). Another attractive aspect of distance education is students’ ability to take courses across state lines without paying out-of-state tuition, which is generally higher than in-state tuition: in 2012, 41percent of undergraduates and 55 percent of graduate students participating in distance education were enrolled in institutions outside of their state of residence (Ginder and Stearns, 2014). Undergraduate enrollment in distance education courses varies by field of study. In 2008, students studying computer science made up the highest share (27%) of all participants in distance education classes (Radford, 2011). By contrast, engineering students (16%) and natural science, mathematics, and agriculture students combined (14%) constituted the smallest shares among all participants (Radford, 2011). The most recent data from 2012 showed that traditional classroom instruction was dominant for all STEM disciplines except computer science (Snyder, de Brey, and Dillow, 2016). Thus, although the frequency of technology-based course instruction is becoming comparable to face-to-face instruction in some undergraduate STEM fields, the extent to which it will be adopted across all disciplines is unknown. Entire educational programs are now available through distance education. As of 2007, 32 percent of all 2-year and 4-year institutions were offering college-level degree or certificate programs designed to be completed entirely through distance education (Parsad and Lewis, 2008); these programs relied most heavily on asynchronous (not simultaneous or in real-time) Internet-based communications technology. However, research to date suggests that 1-16

PREPUBLICATION COPY, UNCORRECTED PROOFS asynchronous online communication, by itself, has limited effectiveness for supporting STEM learning. A review of 13 studies that that assessed learning outcomes in undergraduate biology suggests that combining asynchronous online and synchronous face-to-face communication is a key factor for effective learning (Biel and Brame, 2016). Across the 13 studies, 9 found no significant difference in learning between the students in face to face and online instruction, 2 found that students receiving face-to-face outperformed those receiving online instruction, and 2 found that online students performed better than those in face-to-face classrooms. Biel and Brame (2016) concluded that well-designed online biology courses can be at least as effective for student learning as face-to-face courses and called for further research to illuminate the specific course elements and structures that can maximize students’ learning of biology skills and concepts. Separately, Bowen and colleagues (2012) compared the effectiveness of hybrid (including both face-to-face and online interactions) and traditional formats for a statistics course offered at six public university campuses. Using a randomized controlled study design, they found no statistically significant differences in learning outcomes between students in the traditional and hybrid-format sections. One type of purely online education, massive open online courses (MOOCs), has grown rapidly since 2012, but students frequently drop out of these courses, possibly due to the lack of synchronous communication. Completion rates in MOOCs have generally been below 10 percent, and only five MOOCs have been recognized as credit-worthy by the American Council on Education (Kolowich and Newman, 2013; Ho et al., 2014). 1-17

PREPUBLICATION COPY, UNCORRECTED PROOFS REFERENCES American Association for the Advancement of Science. (2011). Vision and Change in Undergraduate Biology Education: A Call to Action. Washington, DC: Author. Available: http://visionandchange.org/finalreport [July 2017]. Arum, R., Roksa, J., and Cook, A. (2016). Improving Quality in American Higher Education: Learning Outcomes and Assessments for the 21st Century. Hoboken, N.J.: John Wiley & Sons, Inc. Association of American Colleges & Universities. (2007). College Learning for the New Global Century. Washington, DC: Author. Available: https://www.aacu.org/sites/default/files/files/LEAP/GlobalCentury_final.pdf [July 2017]. Biel, R., and Brame, C. (2016). Traditional Versus Online Biology Courses: Connecting Course Design and Student Learning in an Online Setting. Journal of Microbiology & Biology Education 17:417-422. Bonvillian, W.B. and Singer, S.R. (2013). The online challenge to higher education. Issues in Science and Technology, 29 (4). Available: http://issues.org/29-4/the-online-challenge- to-higher-education/ [October 2017]. Bowen, W.G., Chingos, M.M., Lack, K.A. and Nygren, T.I. (2012). Interactive Learning Online at Public Universities: Evidence from Randomized Trials. Ithaka S+R, 2012. Available: http://www.sr.ithaka.org/wp-content/uploads/2015/08/sr-ithaka-interactive-learning- online-at-public-universities.pdf [October 2017]. Brancaccio-Taras, L., Pape-Lindstrom, P., Peteroy-Kelly, M., Aguirre, K., Awong-Taylor, J., Balser, R., Cahill, M.J., Frey, R.G., Jack, R., Kelrick, M., Marley, K., Miller, K.G., Osgood, M., Romano, S., Uzman, J.A., and Zhao, J. (2016). The PULSE vision & change rubrics, version 1.0: A valid and equitable tool to measure transformation of life sciences departments at all institution types. CBE-Life Sciences Education, 15(4), ar 60. Available: http://www.lifescied.org/content/15/4/ar60.full [March 2017]. Brewer, C.A., and Smith, D. (2011). Vision and Change in Undergraduate Biology Education: A Call to Action. Final Report of a National Conference organized by the American Association for the Advancement of Science. Washington, DC: American Association for the Advancement of Science. Available: http://visionandchange.org/files/2011/03/Revised-Vision-and-Change-Final-Report.pdf [May 2017]. Brizius, J.A., and Campbell, M.D. (1991). Getting results: A guide for government accountability. Washington, DC: Council of Governors’ Policy Advisors Butz, W.P., Bloom, G.A., Gross, M.E., Kelly, T.K., Kofner, A., and Rippen, H.E. (2003). Is There a Shortage of Scientists and Engineers? How Would We Know? RAND Science and Technology Issue Paper. Available: https://www.rand.org/content/dam/rand/pubs/issue_papers/2005/IP241.pdf [August 2017]. Carnevale, A.P., Smith, N., and Melton, M. (2011). STEM: Science, Technology, Engineering, and Mathematics. Washington, DC: Georgetown University Center on Education and the Workforce. Available: https://cew.georgetown.edu/wp- content/uploads/2014/11/stem-complete.pdf [July 2017]. Dadgar, M., and Weiss, M.J. (2012). Labor Market Returns to Sub-Baccalaureate Credentials: How Much Does a Community College Degree or Certificate Pay? CCRC Working 1-18

PREPUBLICATION COPY, UNCORRECTED PROOFS Paper No. 45. New York: Columbia University, Teachers College, Community College Research Center. Ginder, S. and Stearns, C. (2014). Web Tables: Enrollment in Distance Education Courses by State: Fall 2012. NCES 2014-023. Washington, D.C.: U.S. Department of Education, National Center for Education Statistics. Available: https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2014023 [August, 2017]. Hill, H. and Grossman, P. (2013). Learning from teacher evaluation: Challenges and opportunities. Harvard Educational Review, 82(1), 123-141. Ho, A.D., Reich, J., Nesterko, S., Seaton, D.T., Mullaney, T., Waldo, J., and Chuang, I. (2014). HarvardX and MITx: The first year of open online courses (HarvardX and MITx Working Paper No. 1). Available: https://harvardx.harvard.edu/multiple-course-report [August, 2017]. Holzer, H.J., and Lerman, R.I. (2007). America’s Forgotten Middle Skill Jobs: Education and Training Requirements in the Next Decade and Beyond. Available: http://www.urban.org/sites/default/files/publication/31566/411633-America-s-Forgotten- Middle-Skill-Jobs.PDF [March 2017]. Institute of Medicine. (2001). Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, D.C.: National Academies Press. Kolowich, S., and Newman, J. (2013). The professors behind the MOOC hype.” The Chronicle of Higher Education. 18 Mar. 2013. Available: http://www.chronicle.com/article/The- Professors-Behind-the-MOOC/137905 [August, 2017]. Lowell, B.L., and Salzman, H. (2007). Into the Eye of the Storm: Assessing the Evidence on Science and Engineering Education, Quality, and Workforce Demand. Washington, DC: The Urban Institute. Available: http://www.urban.org/sites/default/files/publication/46796/411562-Into-the-Eye-of-the- Storm.PDF [August 2017]. Lumina Foundation. (2015). The Degree Qualifications Profile: A Learning-Centered Framework for What College Graduates Should Know and Be Able to Do to Earn the Associate’s, Bachelor’s or Master’s Degree. Indianapolis, IN: Lumina Foundation. Available: https://www.luminafoundation.org/files/resources/dqp.pdf [November 2015]. Matchett, K., Dahlberg, M., and Rudin, T. (2016). Quality in the Undergraduate Experience: What Is It? How Is It Measured? Who Decides? Summary of a Workshop. Washington, DC: The National Academies Press. Available: http://www.nap.edu/catalog/23514/quality-in-the-undergraduate-experience-what-is-it- how-is [July 2016]. Matsudaira, J. (2015). Defining and Measuring Quality in Higher Education. Paper commissioned for the Board on Higher Education and the Workforce Meeting on Quality in Higher education, December 14-15. Available: http://sites.nationalacademies.org/cs/groups/pgasite/documents/webpage/pga_170937.pdf [July 2017]. Mankiw, N.G. (2003). Principles of Microeconomics. 3rd Edition. Boston, MA: South-Western College. National Academies of Sciences, Engineering, and Medicine. (2016a). Barriers and Opportunities for 2-Year and 4-Year STEM Degrees: Systemic Change to Support Students’ Diverse Pathways. Committee on Barriers and Opportunities in Completing 2- Year and 4-Year STEM Degrees. S. Malcom and M. Feder (Eds.) Board on Science 1-19

PREPUBLICATION COPY, UNCORRECTED PROOFS Education, Division of Behavioral and Social Sciences and Education and Board on Higher Education and the Workforce, Policy and Global Affairs.. Washington, DC: The National Academies Press. Available: http://www.nap.edu/catalog/21739/barriers-and- opportunities-for-2-year-and-4-year-stem-degrees [March 2016]. National Academies of Sciences, Engineering, and Medicine (2016b). Science Literacy: Concepts, Contexts, and Consequences. Washington, D.C.: National Academies Press. Available: https://www.nap.edu/catalog/23595/science-literacy-concepts-contexts-and- consequences [August, 2017]. National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. (2007). Rising above the Gathering Storm: Energizing and Employing America for a Brighter Future. Committee on Prospering in the Global Economy of the 21st Century: An Agenda for American Science and Technology and Committee on Science, Engineering, and Public Policy. Washington, DC: The National Academies Press. Available: http://www.nap.edu/catalog/11463/rising-above-the-gathering-storm- energizing-and-employing-america-for [March 2016]. National Academy of Science, National Academy of Engineering, and Institute of Medicine. 2010. Rising above the Gathering Storm, Revisited: Rapidly Approaching Category 5. Washington, DC: National Academies Press. Available: https://www.nap.edu/catalog/11463/rising-above-the-gathering-storm-energizing-and- employing-america-for [August, 2017]. National Research Council. (2011). Expanding Underrepresented Minority Participation: America’s Science and Technology Talent at the Crossroads. Washington, D.C.: National Academies Press. Available: https://www.nap.edu/catalog/12984/expanding- underrepresented-minority-participation-americas-science-and-technology-talent-at [August, 2017]. National Research Council. (2012). Discipline-Based Education Research: Understanding and Improving Learning in Undergraduate Science and Engineering. Committee on the Status, Contributions, and Future Directions of Discipline-Based Education Research. S.R. Singer, N.R. Nielsen, and H.A. Schweingruber (Eds.). Board on Science Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press. Available: http://www.nap.edu/catalog/13362/discipline- based-education-research-understanding-and-improving-learning-in-undergraduate [March 2016]. National Research Council. (2014). Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, D.C.: National Academies Press. Available: https://www.nap.edu/catalog/18606/capturing-change-in-science- technology-and-innovation-improving-indicators-to [August, 2017]. National Science Foundation. (2014). Science and Engineering Indicators 2014. Arlington, VA: Author. Available: https://www.nsf.gov/statistics/seind14/ [July 2014]. National Science Foundation (2015). Revisiting the STEM Workforce: A Companion to Science and Engineering Indicators 2014. Arlington, VA: Author. Available: http://www.nsf.gov/pubs/2015/nsb201510/nsb201510.pdf [March 2016]. National Science Foundation. (2016). Science and Engineering Indicators 2016. Arlington, VA: Author. Available: https://www.nsf.gov/statistics/2016/nsb20161/#/ [July 2017]. National Science Foundation, National Center for Science and Engineering Statistics. 2017. Women, Minorities, and Persons with Disabilities in Science and Engineering: 1-20

PREPUBLICATION COPY, UNCORRECTED PROOFS 2017. Special Report NSF 17-310. Arlington, VA. Available: https://www.nsf.gov/statistics/2017/nsf17310/ [August, 2017]. National Science and Technology Council (2013). Federal STEM Education 5-Year Strategic Plan. Available: https://www.whitehouse.gov/sites/default/files/microsites/ostp/stem_stratplan_2013.pdf [March 2016]. Oakes, J. (1986). Educational Indicators: A Guide for Policymakers. Santa Monica, CA: Center for Policy Research in Education. Odden, A. (1990). Educational indicators in the United States: The need for analysis. Educational Researcher, 19(5), 24–29. Parsad, B. and Lewis, L. (2008). Distance Education at Degree-Granting Postsecondary Institutions: 2006–07 (NCES 2009–044). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. Washington, DC. Available: https://nces.ed.gov/pubs2009/2009044.pdf [August, 2017]. Planty, M., and Carlson, D. (2010). Understanding Education Indicators: A Practical Primer for Research and Policy. New York: Teachers College Press. President’s Council of Advisors on Science and Technology (2012). Engage to Excel: Producing One Million Additional College Graduates with Degrees in Science, Technology, Engineering and Mathematics. Available: https://www.whitehouse.gov/sites/default/files/microsites/ostp/pcast-engage-to-excel- final_feb.pdf [March 2016]. Radford, A.W. (2011). Stats in Brief: Learning at a distance: Undergraduate enrollment in distance education courses and degree programs. (NCES 2012-154). Washington, D.C.: National Center for Education Statistics, U.S. Department of Education. Available: https://nces.ed.gov/pubs2012/2012154.pdf [August, 2017]. Romer, Paul M. 1990. “Endogenous Technological Change.” Journal of Political Economy 98:S5, S71. Rothwell, J. (2013). The Hidden STEM Economy. Metropolitan Policy Program at Brookings Institute. Available: http://www.brookings.edu/~/media/research/files/reports/2013/06/10-stem-economy- rothwell/thehiddenstemeconomy610.pdf [July 2017]. Shavelson, R.J., McDonnell, L.M., and Oakes, J. (1989). Indicators for Monitoring Mathematics and Science Education. Santa Monica, CA: RAND Corporation. Available: http://www.rand.org/pubs/reports/R3742.html [July 2017]. Snyder, T.D., de Brey, C., and Dillow, S.A. (2016). Digest of Education Statistics 2014. (NCES 2016-006). Washington, D.C.: National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. Available: https://nces.ed.gov/pubs2016/2016006.pdf [August, 2017]. Solow, Robert M. 1957. “Technical Change and the Aggregate Production Function.” The Review of Economics and Statistics 39:312-20. Stevens, A.H., Kurlaender, M., and Grosz, M. (2015). Career Technical Education and Labor Market Outcomes: Evidence from California Community Colleges. (NBER Working Paper No. 21137). National Bureau of Economic Research. Available: U.S. Department of Education. http://www.nber.org/papers/w21137 [August, 2017]. 1-21

PREPUBLICATION COPY, UNCORRECTED PROOFS Weisberg, D., Sexton, S., Mulhern, J., and Keeling, D. (2009). The widget effect: Our national failure to acknowledge and act on differences in teacher effectiveness. Brooklyn, NY: The New Teacher Project. Wilson, S.M., and Anagnostopolous, D. (in press). The seen and the foreseen: Will unintended consequences thwart efforts to (re)build trust in teacher preparation? Journal of Teacher Education. Xie, Y., and Killewald, A.A. (2012). Is American Science in Decline? Cambridge, MA: Harvard University Press. 1-22

Next: 2 Conceptual Framework for the Indicator System »
Indicators for Monitoring Undergraduate STEM Education Get This Book
×
Buy Paperback | $55.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Science, technology, engineering and mathematics (STEM) professionals generate a stream of scientific discoveries and technological innovations that fuel job creation and national economic growth. Ensuring a robust supply of these professionals is critical for sustaining growth and creating jobs growth at a time of intense global competition. Undergraduate STEM education prepares the STEM professionals of today and those of tomorrow, while also helping all students develop knowledge and skills they can draw on in a variety of occupations and as individual citizens. However, many capable students intending to major in STEM later switch to another field or drop out of higher education altogether, partly because of documented weaknesses in STEM teaching, learning and student supports. Improving undergraduate STEM education to address these weaknesses is a national imperative.

Many initiatives are now underway to improve the quality of undergraduate STEM teaching and learning. Some focus on the national level, others involve multi-institution collaborations, and others take place on individual campuses. At present, however, policymakers and the public do not know whether these various initiatives are accomplishing their goals and leading to nationwide improvement in undergraduate STEM education.

Indicators for Monitoring Undergraduate STEM Education outlines a framework and a set of indicators that document the status and quality of undergraduate STEM education at the national level over multiple years. It also indicates areas where additional research is needed in order to develop appropriate measures. This publication will be valuable to government agencies that make investments in higher education, institutions of higher education, private funders of higher education programs, and industry stakeholders. It will also be of interest to researchers who study higher education.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!