Because education in the United States is decentralized, it is likely that schools, districts, and states will implement the recommendations in Successful K-12 STEM Education in varied ways and to varying degrees. And although some data that could provide indications of progress in meeting the recommendations of the 2011 report are currently collected at the state and national levels, these data are not collected or analyzed expressly for that purpose, nor is there any attempt to coordinate data collection. As a result, educators, researchers, and policy makers lack a clear picture of the extent to which the nation as a whole is implementing the recommended improvements to STEM education, or making progress toward them.
To address this lack, and in response to a request from Congress, the National Science Foundation (NSF) requested that the National Research Council’s Board on Science Education and Board on Testing and Assessment convene an ad hoc committee (see Box 1 for the formal charge to the committee). This report of the Committee on the Evaluation Framework for Successful K-12 STEM Education presents the committee’s proposed indicators and framework for creating a monitoring and reporting system with those indicators at its core.
The committee’s charge addresses only the recommendations in Successful K-12 STEM Education. However, those recommendations were developed to support improvements that would enable progress toward the broader goals of U.S. STEM education (see Figure 1). Thus, the committee decided that it is important for this proposed monitoring and reporting system to include information related to the goals for STEM education. By including indicators of conditions that lead to improved performance of students and the education system alongside key performance measures that can be tracked over time to evaluate whether the United States is making progress toward its STEM education goals, the committee’s proposed monitoring and reporting system has the potential to support better decision making and contribute to the desired improvements (Walpole and Noeth, 2002).
Reflecting the emphasis in Successful K-12 STEM Education, this report primarily addresses science and mathematics education rather than all of the STEM disciplines. Because the research base that underpins Successful K-12 STEM Education was the strongest for mathematics and science, the recommendations in that report relate mostly to those subjects, with some of the recommendations addressing only science. Research in technology and engineering education is less mature because those subjects are not as commonly taught in K-12 education (National Research Council, 2011, p. 2). However, engineering and technology are included in A Framework for K-12 Science Education (National Research Council, 2012), which is forming the basis for the Next Generation Science Standards (being developed by Achieve, Inc.). Thus, any of the committee’s proposed indicators that relate to the 2012 report include engineering and technology. In addition, the scheduled 2014 National Assessment of Educational Progress (NAEP) technology
BOX 1
Charge to the Committee
An ad hoc committee will conduct a fast-track, targeted consensus study to identify methods for tracking and evaluating the implementation of the improvements for K-12 education recommended in the 2011 NRC report, Successful K-12 STEM Education. At the school and district levels, these recommended improvements include adequate instructional time and resources for science, coherent standards and curriculum, greater teacher capacity, and supportive school conditions that have been identified in the research. At the state and national levels, these recommended improvements include greater attention to science, including assessments for science, investment in support for STEM teachers, and increased support for research programs that can identify instructional practices that improve student outcomes in STEM.
The committee was charged with authoring a short report that includes a plan for evaluating progress toward the 2011 report’s recommendations. The short report is to provide guidance to enable NSF and Congress to consider how to support a full-scale evaluation in the future. To that end, the report will
• identify existing and additional measures needed for tracking the improvements recommended in 2011,
• discuss the feasibility of including such measures in existing programs of data collection and linking them to existing and improved measures of science learning outcomes,
• identify additional research that would be needed to inform a full-scale evaluation of progress toward the 2011 report’s recommendations, and
• make new recommendations regarding the roles of various federal and state institutions in supporting the needed research and data collection for an evaluation of progress.
and engineering framework is increasing the emphasis on these subjects by measuring students’ literacy in them. As these subjects become more central to K-12 education in the future, they can be more fully incorporated into the proposed monitoring system, with additional research to identify the most important aspects to measure.
Study Approach and Sources of Evidence
The committee worked through an iterative process of gathering information, deliberating, identifying gaps and questions, gathering further information to fill those gaps, and further deliberations. The recommendations in Successful K-12 STEM Education (National Research Council, 2011) were the starting point for our deliberations: we took the previous committee’s work and its recommendations as given and did not debate their merit. To identify indicators of progress that would be important to measure for each recommendation, we also drew on a long series of research reports related to K-12 science and mathematics instructional practices (National Research Council, 1999, 2001, 2006, 2007, 2009, 2012; National Academy of Engineering, 2010); school conditions, leadership, and teacher capacity building (e.g., Bryk et al., 2010; Gamoran et al., 2003; McLaughlin and Talbert, 2006); learning science in informal environments (National Research Council, 2009); and general principles of monitoring and evaluation (Shadish, Cook, and Campbell, 2001; U.S. General Accounting Office, 2003). Finally, we consulted relevant experts at agencies that manage large-scale data sets (e.g., National Center for Education Statistics, National Science Foundation) and other organizations (e.g., Council of Chief State School Officers) about the availability of certain types of data.
The rest of this report identifies key indicators that could provide evidence of progress toward the recommendations in Successful K-12 STEM Education. We set four criteria to guide the selection of indicators. Indicators must
• be shown by research to be related to a given recommendation,
• generate information that clearly enhances understanding of progress toward the recommendation for all student groups,
• have the potential to be used for continuous improvement, and
• be feasible to measure on a large scale.
Some of the proposed indicators require new kinds of data collection, and all would benefit from additional research and conceptual development to enhance understanding of what to measure. We included indicators as long as they met the above criteria, but we did not undertake a systematic analysis of the costs involved in measuring the proposed indicators. Instead, we drew on the committee’s expertise with federal research and data collection systems and on consultations with relevant experts to determine whether collecting data on the proposed indicators would generally be feasible.
The monitoring and reporting system proposed in this report is not the first of its kind. In the late 1980s similar indicator systems were developed for K-12 science and mathematics education (National Research Council, 1988; Shavelson, McDonnell, and Oakes, 1989), and the committee consulted those reports. The indicators in those reports were comprehensive, were
carefully developed, and had the potential to generate valuable information about the condition of mathematics and science education in the nation’s schools and how individual components of the education system interact to affect the system as a whole (National Research Council, 1988; Shavelson, McDonnell, and Oakes, 1989). Although they were never fully implemented, these indicator systems provided a base for our work.
As we note at the beginning of this report, the committee’s proposed indicators are being offered at a time of promising changes to what is taught in K-12 science and mathematics, how students are assessed in those subjects, and how teachers are prepared to meet those changing demands. Although the proposed indicators overlap to some degree with previously developed indicators, they reflect current national priorities and recent research on teaching and learning in STEM. They also reflect the continuing interest of Congress and NSF in putting the recommendations of Successful K-12 STEM Education into action. By proposing a comparatively small set of focused, specific, and actionable indicators that are driven by research on impact, the committee hopes that the current political will and momentum will be harnessed to build and sustain a valuable monitoring and reporting system.