STEM Integration in K-12 Education: Status, Prospects, and an Agenda for Research (2014)

6

Findings, Recommendations, and Research Agenda

There is considerable concern among policymakers, educators, employers, and others about improving K–12 STEM education in the United States and in raising the number and quality of students who are both interested in and prepared to enter STEM and related professions. Historically, most efforts to improve STEM education at the pre-college level have focused on the individual subjects—particularly science and mathematics—rather than on how or whether they can or should be connected in ways that might improve student thinking, learning, engagement, motivation, or persistence.

Several converging forces have elevated the importance of understanding the potential value—as well as the limitations and challenges—of integrated STEM education. One development is the small but growing presence of engineering in K–12 classrooms¹ and out-of-school settings. Because engineering by its nature draws on ideas and practices from the other three STEM disciplines, it is has been seen by some as a natural focus for integration efforts. The recent publication of the Next Generation Science Standards

_______________________

¹ Two of the most widespread programs are Engineering is Elementary (EiE) and Project Lead the Way (PLTW). EiE estimates that its curriculum has reached 4.1 million students and has been used by 52,000 teachers (Christine Cunningham, Museum of Science, Boston, personal communication, August 1, 2013). PLTW estimates that 5,500 schools offer at least one of its programs each year, enrolling between 400,000 and 500,000 students annually (Jennifer Cahill, PLTW, personal communication, August 7, 2013).

(NGSS; Achieve 2013), which suggest that certain science concepts be learned in the context of engineering design, is a significant illustration of the belief that concepts and practices from different STEM disciplines can be learned in concert. This appealing but still somewhat intuitive notion, however, is not yet strongly supported by findings from research, as indicated by the committee’s review of the literature.

A key driver of the NGSS was the desire to present science to students in ways that more closely represent how scientists experience it: as a practice requiring application of knowledge from multiple disciplines. This speaks to a second factor driving interest in integrated STEM education: concern about how to better prepare US students to enter the workplace, whether immediately after high school or following postsecondary coursework. An increasing share of jobs across a range of economic sectors, not just in science and engineering, is likely to require some background in the STEM subjects (Carnevale et al. 2011). In addition, employers have made clear their need for workers who can flexibly apply knowledge to solve practical problems (AACU 2013). Development of problem-solving expertise is neither the goal nor an assured outcome of every integrated STEM education initiative, but the committee’s review of programs finds problem solving² to be a common element of many integrated approaches to STEM learning.

Finally, some of the impetus for integrated STEM education is undoubtedly driven by dissatisfaction with traditional approaches to science and mathematics education in the United States. Although decades of education reform have brought significant changes to curricula, standards, and professional development in these subjects, much science and mathematics teaching still emphasizes rote skills and memorization; relatively few K–12 students express interest in pursuing these subjects in college or as a career; and the performance of US students on international comparative assessments is below what many feel is adequate, given how expertise in these subjects helps fuel the nation’s innovation engine. Might integrated STEM education be part of the solution to the country’s math and science education woes?

There are many more questions than answers. Research on integrated STEM education is just emerging as a distinct topic. As noted, few data convincingly correlate integrated STEM education with student outcomes. Additionally, much of the research that has been conducted does not distin-

_______________________

² The committee uses “problem solving” here as it is described in the cognitive science literature. For additional background, see Mayer 1992; Newell and Simon 1972; and Polya 1973.

guish between the different possible curriculum approaches and pedagogical methods for enabling integration or between the different cognitive mechanisms students may use to construct meaning from integrated learning experiences.

Taking into account these various limitations, this chapter presents the committee’s findings and recommendations in four areas: research on integrated STEM education; outcomes of integrated STEM education; the nature of integrated STEM education; and the design and implementation of integrated STEM education. In some cases, the recommendations are directed to researchers, in others to those who design, implement, or assess integrated STEM education.

In the chapter’s last three sections, we pose important questions for researchers in education and the learning sciences. Taken together, the questions constitute a research agenda for advancing understanding and the effective design and implementation of integrated STEM education in the United States. Addressing some of the questions will benefit from, if not require, the participation of K–12 educators who are engaged in efforts to integrate the STEM subjects.

FINDINGS AND RECOMMENDATIONS RELATED TO RESEARCH ON INTEGRATED STEM EDUCATION

In the majority of studies of curricula and programs in integrated STEM education, whether in formal or out-of-/after-school settings, the educational interventions are poorly described. Evaluation studies are of little value if it is not possible to tell what was done to improve students’ understanding, skills, or attitudes. Lack of detailed descriptions of interventions and of experimental methods makes it difficult to have confidence in reported outcomes or to identify the essential ingredients of effective integrated STEM education. In addition, many of the studies used research designs or outcome measures that did not appear optimal for addressing the questions posed. Last, most studies did not have control groups, making it impossible to disentangle program effects from student selection effects.

RECOMMENDATION 1: In future studies of integrated STEM education, researchers need to document the curriculum, program, or other intervention in greater detail, with particular attention to the nature of the integration and how it was supported. When reporting on outcomes,

researchers should be explicit about the nature of the integration, the types of scaffolds and instructional designs used, and the type of evidence collected to demonstrate whether the goals of the intervention were achieved. Specific learning mechanisms should be articulated and supporting evidence provided for them.

Across studies of integrated STEM education, there is often inconsistent use of language, failure to define terms, and lack of a theoretical framework for understanding integrated STEM education. One goal of this report is to provide a common vocabulary and a starting point for a theoretical framework. Generally recognized theoretical perspectives can be a powerful tool for helping build a community of researchers, program designers, and practitioners who are working toward a shared understanding. Chapter 2’s descriptive framework, which suggests four high-level dimensions of integrated STEM education, provides a vocabulary that practitioners and researchers can use as a basis for mutual understanding; the implications of the research reviewed by the committee presented in Chapter 4 can be a foundation for implementation efforts as well as a basis for further research.

RECOMMENDATION 2: Researchers, program designers, and practitioners focused on integrated STEM education, and the professional organizations that represent them, need to develop a common language to describe their work. This report can serve as a starting point.

The research literature reviewed by the committee shows that studies of integrated STEM education vary greatly in design and methodology. In some cases, the study design was not suited to addressing the questions posed in the research. A recent report by education staff at the Institute for Education Sciences at the Department of Education and the National Science Foundation clarifies the categories of education research and provides basic guidance about their purposes, justifications, design features, and expected outcomes (Box 6-1). Importantly, the document notes that, although the study types are presented in a linear sequence, the reality of education research is considerably more complex and often involves multiple feedback loops between and among the categories. The two agencies intend to use the document to establish uniform expectations for proposals submitted in response to particular program announcements, solicitations, or other funding opportunities. Research teams engaged in efforts to understand integrated STEM education could benefit greatly by attending to the guid-

ance presented in this framework. Similarly, the composition of the teams, which will vary according to the category of research, can be informed by the considerations discussed in the agencies’ report.

FINDINGS AND RECOMMENDATIONS RELATED TO THE OUTCOMES OF INTEGRATED STEM EDUCATION

Advocates of integrated STEM education claim that integrated approaches can produce improvements across a range of outcomes, including learn-

ing and achievement, interest, identity, and persistence in STEM fields. Yet research on these outcomes is uneven and lacks consistency in terms of definitions and variables. Studies in formal settings tend to emphasize learning outcomes, for which measures are narrowly focused on improved conceptual knowledge or achievement. In contrast, studies in after-/out-of-school settings tend to give more emphasis to outcomes related to development of interest and identity. In both settings, there is some attention to persistence in STEM, but few studies follow students over multiple years. Thus it is unclear why and how integration might offer better support for developing certain conceptual knowledge and skills. And while there is a theoretical basis for the conjecture that integrated instruction can promote stronger engagement and longer-term interest in STEM subjects, there is a lack of empirical support for this conjecture.

To determine what forms of integration are most effective, more attention needs to be directed to the types of outcome measures collected. For example, if a particular integrated STEM program focuses on science and engineering, the researchers should include separate measures of learning for each. Similarly, the differences between programs may be apparent not simply in measures of basic knowledge, such as recall of normative ideas or very contextualized problem solutions, but also in measures of deep, connected conceptual understanding and transfer. Too few studies measure retention or transfer of learning. Finally, hypothesized benefits of long-term engagement in STEM and participation in the STEM pipeline are often cited as the rationale for integrated STEM instruction. But persistent changes in attitudes, the development of STEM identities, and subsequent course taking are rarely measured in evaluations of such programs. The emerging domain of discipline-based education research is attending to many of these issues (NRC 2012a).

RECOMMENDATION 3: Study outcomes should be identified from the outset based on clearly articulated hypotheses about the mechanisms by which integrated STEM education supports learning, thinking, interest, identity, and persistence. Measures should be selected or developed based on these outcomes.

Learning, Reasoning, and Achievement

In studies on integrated STEM education, learning outcomes are often measured using scores on standardized tests, though some studies include tests of knowledge specifically tied to the intervention. For outcomes that are more

typically measured, such as gains in content knowledge, it is important to be sure that the measure fits the expected outcomes; achievement tests are often used, but a more refined measure directly linked to the particular experience may be more appropriate. Outcomes tested might relate to students’ ability to make connections between disciplines or to use concepts or skills learned in the context of one discipline in the context of a different discipline. Such outcomes are likely very difficult to measure but more reflective of the deeper learning many experts believe is vital for college and career readiness (NRC 2012b).

Interest, Identity, and Persistence

Understanding the impact of integrated STEM education on learners’ attitudes about the disciplines is important and will help determine whether integrated STEM environments can be more interesting and motivating for students than settings in which there is no integration. In its review of the literature, the committee found hints that integrated STEM education may positively influence STEM interest and identity, and that this effect may be particularly strong for populations that have historically struggled in STEM classes and are underrepresented in STEM programs in higher education and STEM professions.

As noted in Chapter 3, identity generally refers to who one is or wants to be, as well as to how one is recognized by others—as a particular kind of person with particular interests, expertise, and ways of being in particular social contexts. Preliminary evidence indicates that problem- and inquiry-based work better position youth in expert roles, especially when students can define the content and direction of the research. Integrated experiences also appear to offer a wider range of knowledge, experience, and ways of knowing that might be valued by schools and employers and that are integral to identity development. However, more research is needed to determine if this is the case across different populations and contexts.

All but two of the studies on identity that the committee reviewed were qualitative in nature, and most were ethnographic. Ethnography is a clear choice for documenting local settings in rich detail with in-depth focus on individuals, resources, tools, actions, and interactions. Most of the studies took place over several months (some were shorter term, and several longer). Few of the studies seriously considered race or class as central analytic lenses, even if the students involved were of different races and ethnicities. The few

longer-term studies and small sample sizes limit the ability to understand patterns of identity development and the mechanisms that lead to productive patterns.

RECOMMENDATION 4: Research on integrated STEM education that is focused on interest and identity should include more longitudinal studies, use multiple methods, including design experiments, and address diversity and equity.

An underlying assumption of the focus on interest and identity is that students with greater interest in STEM and who identify with STEM will be more likely to seek and persist in STEM-related experiences, not only through traditional or interdisciplinary career paths but also by using their STEM knowledge and skills in other professions and pursuits. This has typically been measured by tracking course taking after an integrated STEM experience or through self-reported aspirations. In the case of integrated STEM education programs in after- and out-of-school settings, most of the studies that measured persistence did not include a sufficient control or comparison group of students who did not participate in the program to enable inferences about impact. Because students in after-/out-of-school programs can choose whether to participate, it is important to design studies that support inferences about the role of the integrated STEM programs in persistence in STEM.

Box 6-2 provides examples of research questions related to the outcomes of integrated STEM education.

FINDINGS RELATED TO THE NATURE OF INTEGRATED STEM EDUCATION

Our analysis of the research and examination of specific programs led to important findings about three aspects of integrated STEM education: the interplay between integrated and disciplinary learning; the cognitive pluses and minuses associated with connection making; and the role that context seems to play in supporting integrated STEM learning.

One reasonable expectation of integrated STEM education is that it encourages the learner to make new and useful connections between or among STEM disciplines. These connections may be exhibited as improvements in student performance, learning and transfer, and interest and moti-

vation. Some research suggests that integration can support learning because basic qualities of cognition favor connecting concepts and representations, so they are associated with other knowledge and grounded in familiar experiences and prior knowledge. Yet the literature is also full of examples suggesting that integration requires considerable cognitive resources. Thus, in some cases activities that integrate multiple disciplines may actually impede comprehension and learning because of the large mental processing demands associated with split attention—dividing one’s attention between multiple sources of information presented in noncomplementary forms, in different settings, or at different times.

Our review of the research suggests that, to benefit from integrated STEM education experiences, students need to be competent with discipline-specific representations and able to translate between them, exhibiting what some scholars refer to as “representational fluency.” Participation in shared practices, such as modeling in engineering, science, and mathematics,

may support such fluency. But because the practice of modeling differs in these disciplines, it is possible that shared practices might actually muddle important distinctions. The level of disciplinary competency may be fairly low in younger students and still allow meaningful integrated STEM experiences; higher levels of content knowledge become increasingly important as students move into high school and tackle more challenging problems. To help students both build and use disciplinary knowledge and skill in integrated settings, it may be necessary to strategically incorporate discipline-specific learning opportunities into integrated experiences (see, for example, Burghardt and Hacker 2008; Lehrer and Schauble 2004).

The use of problem solving as a context and pedagogical approach for integrating concepts and practices from multiple disciplines is a feature of many integrated STEM education programs. One implication of this finding is that practices such as engineering design and science inquiry, and instructional approaches like problem- and project-based learning, may offer special opportunities to support STEM integration when sufficient and intentional instructional support is provided. Some problem situations aim for authenticity, but there are also contrived problems that may support student learning, and some authentic problems will be too complex to carry out in the classroom. Working on complex, authentic problems, which almost always calls upon multiple disciplines, has the potential to support both short-term learning and longer-term application or transfer to new contexts. However, such outcomes are not a given and depend on a number of factors related to the design and implementation of the learning experience, as well as the teacher’s ability to successfully support student problem-solving efforts.

Box 6-3 provides examples of research questions related to the nature of integrated STEM education.

FINDINGS AND RECOMMENDATIONS RELATED TO THE DESIGN AND IMPLEMENTATION OF INTEGRATED STEM EDUCATION

Those who design and implement integrated STEM education experiences will need to attend to a number of interrelated factors, if they hope to influence student learning, interest, motivation, and persistence in STEM subjects. A starting point can be the committee’s effort in Chapter 4 to spell out the implications of the research for creating or modifying existing STEM education programs to include elements of disciplinary integration. The

potential benefits and challenges of making connections across the STEM subjects suggest the importance of a measured, strategic approach to implementing integrated STEM education that accounts for the inherent tradeoffs in cognition and learning.

Beyond this, being clear up front about what an educational intervention is expected to achieve—goal setting—is critical. As noted, we found that the goals of integrated STEM education in formal settings are often focused on achievement and preparation for future academic study. In after-/out-of-school settings, the goals of integrated STEM education tend to focus on

promoting awareness and interest in the STEM disciplines more than on academic achievement and preparation for future study or careers. The majority of integrated STEM education programs reviewed by the committee did not explicitly tie goals to hypotheses about the design of the intervention. This makes it very difficult to determine whether integration is helping to achieve the particular goal.

RECOMMENDATION 5: Designers of integrated STEM education initiatives need to be explicit about the goals they aim to achieve and design the integrated STEM experience purposefully to achieve these goals. They also need to better articulate their hypotheses about why and how a particular integrated STEM experience will lead to particular outcomes and how those outcomes should be measured.

As noted, there are few studies that specify and test the mechanisms by which integrated STEM experiences support learning within and across the disciplines. In this report we have begun to describe potential mechanisms based on existing research on learning in integrated settings and more broadly. One example of such a mechanism is to look for core concepts and practices that recur in the STEM subjects. Such recurring concepts and practices can be elaborated, extended, or otherwise transformed by exploring the different senses of the “same” idea or practice across multiple STEM disciplines. Designing educational initiatives to take advantage of these recurring ideas and practices is challenging. Many of the practices, such as developing and using models, and crosscutting concepts, such as patterns and systems, identified in A Framework for K–12 Science Education (NRC 2012c) may be good candidates.

It is clear from the research that STEM connections that may appear obvious to teachers, curriculum developers, and disciplinary experts often are not obvious to novice learners. At times, teachers themselves may not apprehend the connections. For either reason, integration of STEM concepts and transfer of learning to new contexts may not be spontaneously made by students and cannot be assumed to take place simply because certain concepts and practices are introduced at the same time or place. Put another way, deep, lasting, and transferable learning through integrative experiences will rarely be automatic for most students and some teachers.

RECOMMENDATION 6: Designers of integrated STEM education initiatives need to build in opportunities that make STEM connections

explicit to students and educators (e.g., through appropriate scaffolding and sufficient opportunities to engage in activities that address connected ideas).

As previously noted, neither the committee nor this report is suggesting that integrated STEM education take the place of high-quality education focused on the individual STEM subjects. Indeed, if anything, integrated STEM education reinforces the need for students to hone their disciplinary expertise.

RECOMMENDATION 7: Designers of integrated STEM experiences need to attend to the learning goals and learning progressions in the individual STEM subjects so as not to inadvertently undermine student learning in those subjects.

It seems clear that implementing integrated STEM experiences in school and after-/out-of-school settings will often require educator expertise beyond that required to teach any of the STEM disciplines alone. This finding has implications for the education and ongoing support of those charged with delivering integrated STEM instruction.

RECOMMENDATION 8: Programs that prepare people to deliver integrated STEM instruction need to provide experiences that help these educators identify and make explicit to their students connections among the disciplines. These educators will also need opportunities and training to work collaboratively with their colleagues, and in some cases administrators or curriculum coordinators will need to play a role in creating these opportunities. Finally, some forms of professional development may need to be designed as partnerships among educators, STEM professionals, and researchers.

A growing number of K–12 schools self-identify as “STEM” schools, and some proportion of these schools are or claim to be delivering integrated STEM education. For schools attempting integrated STEM education as well as for those, such as private funders, with a desire to support such efforts, this report can provide a useful guide for assessing the nature and degree of integration present. The committee’s effort to map one school’s efforts against the descriptive framework for integrated STEM education (see Table 2-1, p. 48) is an example of how this might be done.

Assessment

The design and implementation of any educational intervention will benefit by having methods or tools for assessing outcomes. Without a way of determining how student understanding of STEM concepts and facility with STEM practices are changing, it will be impossible to modify the design or implementation in ways that improve chances for success. In the case of integrated STEM education, we have found no evidence that researchers, curriculum developers, or practitioners are measuring outcomes from integrated STEM experiences in reliable, valid ways. The National Assessment of Educational Progress will field an assessment of technology and engineering literacy in a sample of US 8th graders in 2014 (NAEP 2013). While the test will not probe connections among the STEM subjects per se, it will measure performance on problem-solving activities. Lessons learned from this assessment may be useful to future efforts to develop assessments for integrated STEM education. In addition, as the NGSS begin to be adopted and implemented by the states, pressure for new assessments, including those that measure facets of STEM integration, may increase. The National Research Council has developed a framework for assessment of K–12 science proficiency based on NGSS that addresses issues of integration.³

RECOMMENDATION 9: Organizations with expertise in assessment research and development should create assessments appropriate to measuring the various learning and affective outcomes of integrated STEM education. This work should involve not only the modification of existing tools and techniques but also exploration of novel approaches. Federal agencies with a major role in supporting STEM education in the United States, such as the Department of Education and the National Science Foundation, should consider supporting these efforts.

In order for any significant and lasting change to take hold within the K–12 education system, decades of research and experience suggest the importance of aligning key aspects of the improvement process. Thus, for example, both the design of an educational intervention and its implementation should reflect the goals and objectives established by the developers (e.g., Krajcik et al. 2008). Furthermore, what is learned during implementation and data gathered on the outcomes of the intervention should inform an iterative process of con-

_______________________

³ Information on the project is available at www8.nationalacademies.org/cp/projectview.aspx?key=49464.

tinuous improvement (see, for example, NRC 2003). The design, implementation, and even the original goals may need to be modified to reflect experience and optimize the desired outcomes.These ideas can be represented as a series of feedback loops (Figure 6-1).

The challenge of creating alignment and assuring productive feedback among the major elements of education change is not trivial. As the committee notes in its review of the research on STEM education in Chapter 3 and repeats in this chapter, many well-meaning efforts to develop integrated STEM education programs are either unclear about goals or do not collect outcomes data that allow one to determine if the stated goals have been met. In addition, in many cases, it is not possible to determine if the program as

FIGURE 6-1 Iterative model of educational change.

designed, what is sometimes called the intended curriculum, is the same as what is implemented, the enacted curriculum.

RECOMMENDATION 10: To allow for continuous and meaningful improvement, designers of integrated STEM education initiatives, those charged with implementing such efforts, and organizations that fund the interventions should explicitly ground their efforts in an iterative model of educational improvement.

Box 6-4 provides examples of research questions related to the design and implementation of integrated STEM education.

Two recent developments in US K–12 STEM education provide special opportunities for researchers to investigate some of these questions. One, as noted above, is publication of the NGSS, which is spurring new curriculum, assessments, and educator supports, some of which will focus on how science and engineering concepts and practices are connected. The second is the implementation of restructured advanced placement (AP) science courses by the College Board, developed in response to recommendations from the National Research Council (2002).The new AP biology course was introduced in the 2012-13 school year (College Board 2011a) and the new chemistry course in 2013-14 (College Board 2011b). The revised physics course will begin in the 2014-15 school year, and the environmental science course sometime after 2015. Although each of these courses has a disciplinary emphasis, they also aim to build student competencies to connect with other subjects, particularly mathematics.

FINAL THOUGHTS

In addressing its charge, the committee has carefully reviewed the available research on integrated STEM education and related research and examined a selected set of curricula, schools, professional development efforts, and other relevant initiatives. As this final chapter suggests, there is much more that can and should be learned about the outcomes, nature, and design and implementation of integrated STEM education. This should not discourage those designing, implementing, or studying integrated STEM education programs. On the contrary, our findings, recommendations, and research agenda strongly suggest the potential of some forms of integrated STEM education to make a positive difference in student learning, interest, and other valued outcomes.

In order to achieve this potential, the energy, creativity, and resources of researchers, practitioners, and concerned funders must now be directed at generating more thoughtful, high-quality, and evidence-based work. Given the inherent complexities, it will not be a surprise to find that designing, implementing, and documenting effective integrated STEM education is both time consuming and expensive. Despite these very real challenges, the possibility of adding new tools to the STEM education toolbox is tantalizing.

REFERENCES

AACU (Association of American Colleges and Universities). 2013. It takes more than a major: Employer priorities for college learning and student success. Available at www.aacu.org/leap/documents/2013_EmployerSurvey.pdf (retrieved January 21, 2014).

Achieve, Inc. 2013. Next Generation Science Standards. Available at www.nextgenscience.org/next-generation-science-standards (retrieved May 31, 2013).

Burghardt, D., and M. Hacker. 2008. Work in progress: Math infusion in a middle school engineering/technology class. 38th ASEE/IEEE Frontiers in Education Conference, October 22–25, 2008, Saratoga Springs, NY. Available at http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=4720457 (retrieved May 2, 2013).

Carnevale, A.P., N. Smith, and M. Melton. 2011. STEM. Georgetown University Center on Education and the Workforce. Available at www9.georgetown.edu/grad/gppi/hpi/cew/pdfs/stem-complete.pdf. (retrieved May 26, 2013).

College Board. 2011a. AP Biology Curriculum Framework 2012-2013. New York. The College Board. Available at http://media.collegeboard.com/digitalServices/pdf/ap/10b_2727_AP_Biology_CF_WEB_110128.pdf (retrieved November 11, 2013).

College Board. 2011b. AP Chemistry Curriculum Framework 2013-2014. New York: The College Board. Available at http://media.collegeboard.com/digitalServices/pdf/ap/11_3461_AP_CF_Chemistry_WEB_110930.pdf (retrieved November 11, 2013).

IES (Institute for Educational Sciences)/NSF (National Science Foundation). 2013. Common guidelines for education research and development—A report from the Institute of Education Sciences, U.S. Department of Education and the National Science Foundation. August 2013. Available at http://ies.ed.gov/pdf/CommonGuidelines.pdf (retrieved October 30, 2013).

Krajcik, J., K.L. McNeill, and B. Reiser. 2008. Learning-goals-driven design model: Developing curriculum materials that align with national standards and incorporate project-based pedagogy. Science Education 92(1):1–32.

Lehrer, R., and L. Schauble. 2004. Modeling natural variation through distribution. American Educational Research Journal 41(3):635–679.

Mayer, R.E. 1992. Thinking, problem solving, cognition. New York: W.H. Freeman.

NAEP (National Assessment of Educational Progress. 2013. NAEP Technology and Engineering Literacy Assessment (TEL). Available at http://nces.ed.gov/nationsreportcard/tel/ (retrieved June 11, 2013).

Newell, A., and H.A. Simon. 1972. Human problem solving. Englewood Cliffs, NJ: Prentice-Hall.

Polya, G. 1973. How to solve it: A new aspect of mathematical method. Princeton, NJ: Princeton University Press.

NRC (National Research Council). 2002. Learning and understanding: Improving advanced study of mathematics and science in U.S. high schools. J.P. Gollub, M.W. Bertenthal, J.B. Labov, and P.C. Curtis, Eds. Available at www.nap.edu/catalog.php?record_id=10129 (retrieved November 11, 2013).

NRC. 2003. Strategic Education Research Partnership. Available at www.nap.edu/catalog.php?record_id=10670 (retrieved January 16, 2013).

NRC. 2012a. Discipline-based education research: Understanding and improving learning in undergraduate science and engineering. Committee on the Status, Contributions, and Future Directions of Discipline-Based Education Research, Board on Science Education, Division of Behavioral and Social Sciences and Education. S.R. Singer, N.R. Nielsen, and H.A. Schweingruber, Eds. Available at www.nap.edu/catalog.php?record_id=13362 (retrieved May 30, 2013).

NRC. 2012b. Education for life and work: Developing transferable knowledge and skills in the 21st century. Committee on Defining Deeper Learning and 21st Century Skills. Available at www.nap.edu/catalog.php?record_id=13398 (retrieved May 31, 2013).

NRC. 2012c. A Framework for K–12 Science Education. Available at www.nap.edu/catalog.php?record_id=13165 (retrieved January 14, 2014).