INDICATORS FOR MEASURING IMPROVEMENTS TO THE U.S. K-12 STEM EDUCATION SYSTEM

In this section, we identify and describe a set of indicators that could provide evidence of progress toward each recommendation of Successful K-12 STEM Education. Although the indicators are linked to specific recommendations in that report, taken together, they address several key elements of successful K-12 education in STEM: access to quality learning opportunities, educators with high capacity to teach in their disciplines, and supportive policies and funding initiatives. Thus, the committee’s intent is for these indicators to form the core of a national program to monitor the health of the education system in STEM. It is not the committee’s aim for these indicators to become part of a new accountability system for K-12 education in STEM. Rather, the goal is for the National Center for Education Statistics (NCES) and NSF to generate information that enables education leaders, researchers, and policy makers to understand and improve state and local education systems.

We propose indicators on five topics related to recommendations for school districts in the 2011 report:

•     multiple models of STEM-focused schools,

•     adequate instructional time and resources for science in grades K-5,

•     high-quality curricula,

•     the capacity of K-12 teachers, and

•     professional development for instructional leaders.

We also propose indicators on four topics related to recommendations for state and national policy makers in the 2011 report:

•     elevated status for science;

•     effective systems of assessment;

•     federal and state support for STEM teachers; and

•     research to enhance understanding of STEM schools, practices, and outcomes.

For each indicator, we discuss available and potentially available data, with “potentially available” defined as data that could be collected by modifying existing data collection systems (see the Appendix for information about surveys that are the potential sources of data). We also discuss data and research needs for each proposed indicator.

The proposed indicator system relies heavily on the NCES Schools and Staffing Survey (SASS).2 Making all of the proposed modifications to SASS might have negative effects on response rates if

___________________

2When this report was written, the most recently available SASS was 2007-2008.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 12
Indicators for Measuring Improvements to The U.S. K-12 STEM Education System I n this section, we identify and describe a set of indicators that could provide evidence of progress toward each recommendation of Successful K-12 STEM Education. Although the indi- cators are linked to specific recommendations in that report, taken together, they address several key elements of successful K-12 education in STEM: access to quality learning oppor- tunities, educators with high capacity to teach in their disciplines, and supportive policies and funding initiatives. Thus, the committee’s intent is for these indicators to form the core of a national program to monitor the health of the education system in STEM. It is not the commit- tee’s aim for these indicators to become part of a new accountability system for K-12 education in STEM. Rather, the goal is for the National Center for Education Statistics (NCES) and NSF to generate information that enables education leaders, researchers, and policy makers to understand and improve state and local education systems. We propose indicators on five topics related to recommendations for school districts in the 2011 report: • multiple models of STEM-focused schools, • adequate instructional time and resources for science in grades K-5, • high-quality curricula, • the capacity of K-12 teachers, and • professional development for instructional leaders. We also propose indicators on four topics related to recommendations for state and national policy makers in the 2011 report: • elevated status for science; • effective systems of assessment; • federal and state support for STEM teachers; and • research to enhance understanding of STEM schools, practices, and outcomes. For each indicator, we discuss available and potentially available data, with “potentially available” defined as data that could be collected by modifying existing data collection systems (see the Appendix for information about surveys that are the potential sources of data). We also discuss data and research needs for each proposed indicator. The proposed indicator system relies heavily on the NCES Schools and Staffing Survey (SASS).2 Making all of the proposed modifications to SASS might have negative effects on response rates if 2When this report was written, the most recently available SASS was 2007-2008. 12 R02309 Monitoring K–12 STEM Ed-PRF3.indd 12 3/13/13 2:05 PM

OCR for page 12
Monitoring Progress Toward Successful K-12 STEM Education the survey becomes too onerous for respondents. Thus, any modifications would ideally be under- taken as part of efforts to systematically streamline SASS. More broadly, placing the burden of this monitoring system on the shoulders of existing national surveys will require strategic decisions about the frequency, subject matter, and question rotations of those surveys. Developing new kinds of data collection for some of the indicators might help to alleviate this problem. For example, background questionnaires for students, teachers, and schools that could be administered with assessments related to the Common Core State Standards for Mathematics (and even- tually, A Framework for K-12 Science Education3) would be a valuable new data collection mechanism for several of the proposed indicators. Because student assessments aligned with the Common Core State Standards for Mathematics are currently under development, an exceptional opportunity exists to develop accompanying background surveys that directly measure the key elements of these reforms. Such surveys could become the primary data collection vehicle for several of the proposed indicators. They would be especially valuable because they would be regularly administered across the majority of states and could be coupled with student achievement data. Although the proposed indicators do not specifically mention different student populations, the aim of equitable access to resources and learning opportunities for all students is central to the goals for education in STEM. Because disparities in access to high-quality learning opportunities, instructional materials, and teachers contribute to achievement gaps among students from different racial, ethnic, language, and socioeconomic groups, tracking patterns in access to those resources is an essential component of Indicators 1-8. Thus, the committee’s intent is for data on Indicators 1-8 to be collected and analyzed in a manner that provides an understanding of variation among different student populations and socioeconomic contexts. Multiple Models of STEM-Focused Schools As noted in Successful K-12 STEM Education, high-quality education in the STEM disciplines can take place in diverse public school settings, including STEM-focused schools with selective admis- sion policies, STEM-focused schools with inclusive admission policies, STEM-focused career and technical education programs, and comprehensive public schools (see National Research Council, 2011, for a more complete description of the school types). Because these schools often have dif- ferent goals and pursue different strategies to meet those goals, and because the evidence is not sufficient to recommend one type of school over another, Successful K-12 STEM Education recom- mended that “districts seeking to improve STEM outcomes beyond comprehensive schools should consider all three models of STEM-focused schools” (National Research Council, 2011, p. 27). Although variation exists within and across these categories, they share an emphasis on the STEM 3Because the Next Generation Science Standards were under development at the time of the report, the committee used the basis for those standards—A Framework for K-12 Science Education—to inform this report. 13 R02309 Monitoring K–12 STEM Ed-PRF3.indd 13 3/13/13 2:05 PM

OCR for page 12
Monitoring Progress Toward Successful K-12 STEM Education disciplines, and on providing access to courses and experiences that will prepare students to be scientifically literate and perhaps pursue careers or further study in the STEM disciplines after high school (National Research Council, 2011). Key Indicator to Monitor As a first step toward measuring progress toward this recommendation, the committee proposes collecting descriptive information to quantify the availability of STEM-focused schools and pro- grams. The indicator that we propose below is a measure only of quantity—and thus, the degree of access to STEM-related learning experiences. To support decisions about the types of schools or programs in which districts should invest, eventually it will be necessary to collect data that address the quality of these schools and programs. Indicator 1. Number of, and enrollment in, different types of STEM schools and programs in each district. This indicator is intended to measure the extent to which all students have the opportunity to pursue some kind of focused experience in STEM as part of their K-12 education, which is particu- larly important for students in areas with limited resources. The indicator should include selective STEM schools, inclusive STEM schools, STEM-focused career and technical education schools or programs, and STEM-focused programs in comprehensive schools, as defined in Successful K-12 STEM Education. Public charter and noncharter schools of these types should be included in the monitoring system. To qualify as a specialized STEM experience, a school or program would need to provide all of its students with a range and depth of STEM learning experiences that exceed state require- ments. Developing the criteria for each type of STEM-focused school or program might involve an expert meeting with individuals who have experience and expertise in implementing and studying such programs. Available and potentially available data. Currently, the primary way to count STEM- focused schools is by searching databases of school names for the words science, technology, engineering, or mathematics. This method does not provide a full or accurate count of STEM- focused schools, in part because there is no uniform definition of what constitutes a “STEM- focused” school or program. Surveys conducted by NCES (e.g., the SASS, the High School Longitudinal Study, and the National Education Longitudinal Study) include yes/no questions about whether a school has a special pro- gram emphasis, but do not differentiate among different themes. The SASS also contains questions about career and technical education schools and programs, but they, too, are not differentiated to 14 R02309 Monitoring K–12 STEM Ed-PRF3.indd 14 3/13/13 2:05 PM

OCR for page 12
Monitoring Progress Toward Successful K-12 STEM Education focus on the STEM disciplines. Additional questions could be added to the SASS or to the National Civil Rights Data Collection as a way of identifying various types of STEM-focused schools and pro- grams. However, these data sets do not contain a census of all schools or use a common definition of “STEM-focused.” To provide a census, this information could be added to the NCES Common Core of Data. Regardless of the survey used, measuring this indicator would involve creating definitions of the elements that characterize a STEM-focused school and would require districts and states to consent to using those common definitions in their reporting. Data and research needs. Research is needed to define the criteria for STEM-focused schools and programs and use them as the framework for developing survey items for administration to school principals. Surveys are needed that elicit principals’ reports of the requirements for entering their STEM-focused school or program, to ascertain the extent to which the program targets those who have already demonstrated STEM talent. These surveys also should capture the geographic areas from which the school draws its students so that analysts can derive estimates of the extent to which different student subgroups do or do not have equal access to the opportunities. Further research also is needed on the essential characteristics of effective STEM schools or programs, leading to the development of indicators that assess whether schools or programs within schools have these characteristics. Adequate Instructional Time and Resources for Science in Grades K-5 The recommendation in Successful K-12 STEM Education that “districts should devote adequate time and resources to science education in grades K-5” (National Research Council, 2011, p. 27) was proposed to mitigate an unintended consequence of the federal accountability system in education: that instructional time for science in elementary school has been reduced to devote additional time to reading and mathematics (Center on Education Policy, 2007; Dorph et al., 2011). Reducing the time devoted to science in the elementary grades is of special concern because some research suggests that “life experiences before 8th grade and in elementary school may have an important impact on future career plans,” which requires “close attention to children’s early exposure to sci- ence” (Tai et al., 2006, p. 1144). Key Indicators to Monitor To the extent that early experiences in science are valuable in preparing students for future science learning and careers, reducing the exposure to science in elementary school is particularly prob- lematic for students who do not have access to science learning opportunities in their homes and communities (National Research Council, 2007). For these students, limiting early science learning opportunities leaves them unprepared for science courses in middle and high school (Hartry et al., 2012), which can exacerbate future inequities in interest, course-taking, and achievement, in STEM. 15 R02309 Monitoring K–12 STEM Ed-PRF3.indd 15 3/13/13 2:05 PM

OCR for page 12
Monitoring Progress Toward Successful K-12 STEM Education The committee proposes two indicators related to instructional time. First, as a proxy for the value that is placed on science, it is essential to measure the number of instructional minutes allocated to science. It also is important to consider the characteristics of that time. For example, implementing instructional approaches that afford students opportunities to engage in the practices of science requires more time than the 30-45 minute session that elementary schools typically allocate to a science lesson. Teachers who have the flexibility to consider time on a weekly basis may create a larger block of time to allow students to plan and carry out investigations, construct explanations, and engage in building an argument from evidence (Dorph et al., 2011). Second, when measuring time devoted to science, it is also important to include opportuni- ties that schools provide for students to engage in science learning both in and beyond formal class time (OECD, 2012). These opportunities vary widely and can include field trips to local science-rich institutions, as well as after-school programs, science camps, clubs, and competitions. Opportunities for science learning in and outside the classroom are particularly important for students who do not have access to science learning opportunities beyond the school (National Research Council, 2009).4 The committee did not attempt to define what was meant by “adequate” time and resources, or to measure the quality of instructional time. Rather, Indicators 2 and 3 are intended to provide ongo- ing measures of the amount of time and the kinds of opportunities that are available for science learning in the elementary grades. Determining adequacy would be considerably more difficult and might entail analyses of the quality of instructional time and the relationship between the average amount of time devoted to science instruction and the level of demand of a state’s science standards. It also might entail additional research on any learning loss from time not spent on other subjects and on the relative effectiveness of different organizational structures for instructional time. Indicator 2. Time allocated to teach science in grades K-5. This indicator simply measures teachers’ estimates of the amount of time that they devote to teach- ing science. Time should include the number of instructional minutes per week, as well as the dif- ferent configurations elementary teachers use to implement those instructional minutes. Available and potentially available data. The SASS teacher questionnaires include ques- tions about time allocated to general subjects such as science and mathematics, as does the NAEP grade 4 teacher survey. Although NAEP offers the opportunity to link survey results with student achievement data, the response options may require modification to accurately assess time use. These limitations notwithstanding, estimates from the existing SASS and NAEP surveys would 4Although the original recommendation addressed instructional time and resources for science in grades K-5, Indicators 2 and 3 only address time. Indicator 4 addresses instructional resources. 16 R02309 Monitoring K–12 STEM Ed-PRF3.indd 16 3/13/13 2:05 PM

OCR for page 12
Monitoring Progress Toward Successful K-12 STEM Education allow for state-level indicators or for comparisons of schools that serve different demographic populations. The surveys could be amended to include questions about the extent to which instruc- tional time that is devoted to other subjects includes science content. Data and research needs. Although data are available for this indicator, existing measures rely solely on teacher self-reported data. Additional research is needed to assess the reliability of these indicators, for example by comparing teachers’ self-reports of time usage with data from classroom observations and teacher logs. Additional research also might be needed to determine how to measure time devoted to science when science is taught in the context of other subjects, perhaps building on the work of Dorph et al. (2011) and Hartry et al. (2012). Indicator 3. Science-related learning opportunities in elemen- tary schools. This indicator is intended to reflect the full range of science learning opportunities that elemen- tary schools provide for students. It should include a focused examination of in-school but non- classroom science learning experiences, together with out-of-school opportunities that schools and districts intentionally provide to enhance their science offerings for all students. The latter may include science centers, museums, zoos, or STEM-related businesses and may depend on the access that different communities have to such science-rich institutions and resources. Available and potentially available data. The NCES High School Longitudinal Study includes questions about high school students’ participation in science, engineering, and math- ematics competitions, museums, clubs, and extracurricular activities. Similar items are under con- sideration for the kindergarten cohort study. If those questions are asked in the earlier grades, they could be modified to indicate whether these opportunities are offered through the school. The OECD’s Programme for International Student Assessment (PISA) is developing teacher question- naires for the 2015 science-focused assessment that also could be modified or adapted. A set of relevant questions also could be added to the SASS teacher survey or the NAEP science teacher survey, although the latter would provide information only for grade 4. Focused, Rigorous, and Sequenced Curricula Typical science and mathematics curricula in the United States have been criticized as being frag- mented and containing too much material for students to be able to build understanding over time (Valverde et al., 2002; Schmidt et al., 2001; Schmidt, McKnight, and Raizen, 1996). In response to this concern, Successful K-12 STEM Education recommended that “districts should ensure that their STEM curricula are focused on the most important topics in each discipline, are rigorous, and are articulated as a sequence of topics and performances” (National Research Council, 2011, p. 27). 17 R02309 Monitoring K–12 STEM Ed-PRF3.indd 17 3/13/13 2:05 PM

OCR for page 12
Monitoring Progress Toward Successful K-12 STEM Education The Common Core State Standards for Mathematics (National Governors Association and Council of Chief State School Officers, 2010) and A Framework for K-12 Science Education (National Research Council, 2012) were designed to address these concerns. The Common Core State Standards for Mathematics have been adopted by 45 states, and A Framework for K-12 Science Education is forming the basis for the Next Generation Science Standards that are currently under development by Achieve, Inc. As significant drivers of K-12 mathematics and science education, these documents provided the context for this committee’s work; however, the following indicators are framed in such a way that relevant data also can be collected in states that do not adopt the standards. The quality of the standards and the effects of adopting them have not yet been fully evaluated. As more research becomes available about these important issues, the proposed indicators that are linked to the standards will need to be revisited and refined. Key Indicators to Monitor In the strictest sense, measuring progress toward this recommendation of Successful K-12 STEM Education would entail determining how many districts had adopted curricula that embody the Common Core State Standards for Mathematics and A Framework for K-12 Science Education or were solidly grounded in current research on teaching and learning in science and mathematics. Although it would be useful to know which curricula are being used and the extent to which they embody research on learning, simply adopting focused, rigorous, and coherent curricula is not sufficient to improve instruction and student outcomes. Thus, the committee also proposes an indicator related to teachers’ reports of how those curricula are being implemented. Indicators that are related to curriculum may include engineering, as well as career and technical education. Career and technical education is a potentially important pathway to prepare students for STEM-related careers, including those in the information technology, computer science, and health fields (Silverberg et al., 2004; National Research Center for Career and Technical Education, 2010). A limited amount of evidence suggests that career and technical education, “assumed to motivate learning through real-life applications, does not have to be in conflict with academic achievement” (National Research Council, 2011, p. 13), as long as the curricula integrate rigorous academic content with occupational training (Stone, Alfeld, and Pearson, 2008). As this report was being written, curricular materials that embody the Common Core State Standards for Mathematics were not in widespread use, nor were those that embody A Framework for K-12 Science Education.5 Thus, Indicators 4 and 5 cannot be monitored until it is clear what materials have been 5Because the Next Generation Science Standards had not been published at the time of this report, the committee used A Framework for K-12 Science Education (National Research Council, 2012) to develop Indicators 4, 5, and 12. These indicators can be tracked in relation to the Next Generation Science Standards when they are published. 18 R02309 Monitoring K–12 STEM Ed-PRF3.indd 18 3/13/13 2:05 PM

OCR for page 12
Monitoring Progress Toward Successful K-12 STEM Education developed, which districts have adopted particular curricular materials and assessments, and how they are using those resources. Nonetheless, it is advisable to develop the indicators before those materials are available so their use can be monitored from the outset. Indicator 4. Adoption of instructional materials in grades K-12 that embody the Common Core State Standards for Mathematics and A Framework for K-12 Science Education. This indicator would provide descriptive information about which districts have adopted instruc- tional materials that embody the Common Core State Standards for Mathematics and A Framework for K-12 Science Education or that have been shown by research to improve student achievement and profi- ciency with the practices of science or mathematics. The committee proposes a two-tiered data collection for this indicator. The first tier includes determining which curricula districts and schools have adopted for science, mathematics, engi- neering, and career and technical education. The second tier involves analysis by an independent entity of the extent to which the most widely used curricula include the practices of science and mathematics, as specified in the Common Core State Standards for Mathematics and A Framework for K-12 Science Education. Available and potentially available data. No data are currently collected that would pro- vide information on a large scale about this indicator. For general K-12 mathematics, science, and engineering education, some questions might be added to the SASS for teachers to ascertain which instructional materials—main and supplemental—they use. Questions about district-level adop- tion of curricula could be added to a district-level survey such as the National Civil Rights Data Collection. These questions also could be incorporated into questionnaires that might accom- pany the assessments that are eventually developed in conjunction with mathematics and science standards. Regarding the alignment of career and technical education materials to standards, the National Center for Research on Career and Technical Education, and some individual states, are undertaking efforts to map career and technical education curricula to the Common Core State Standards for Mathematics. Data and research needs. Considerable research and development work is needed to create a transparent, scientifically neutral set of criteria for determining the degree to which the most wide- ly used instructional materials embody the standards or have been shown by research to improve student achievement and proficiency with the practices of science, mathematics, or engineering. Ideally, the reviews of curricular materials would be available as the materials were produced so that districts could use the reviews to inform their purchasing decisions. 19 R02309 Monitoring K–12 STEM Ed-PRF3.indd 19 3/13/13 2:05 PM

OCR for page 12
Monitoring Progress Toward Successful K-12 STEM Education Research to develop these criteria could build on current efforts to map career and technical education curricula to standards, as well as previous efforts by the American Association for the Advancement of Science (2005) in science and Gueudet, Pepin, and Trouche (2012) and Schmidt et al. (1997) in mathematics. Most of these efforts have addressed content; less research has been done to analyze the extent to which curricula address the practices of science, math- ematics, and engineering (e.g., building an argument from evidence, constructing explanations, and designing solutions). Much work remains to define these practices and determine how they might look across different grades and subject areas. Such work may include pilot studies to develop coding schemes for assessing curricula. Indicator 5. Classroom coverage of content and practices in the Common Core State Standards for Mathematics and A Framework for K-12 Science Education. The opportunity students have to learn content and practices is a critical indicator that has been shown in numerous studies to be related to achievement and distributed inequitably across different populations of students (Schmidt and Maier, 2009; Schmidt and McKnight, 2012). Content coverage (or opportunity to learn) is defined in three ways: (1) the extent of coverage, (2) the amount of time devoted to content, and (3) the order of coverage. Because it is central to academic achievement, coverage provides an intermediate indicator related to the quality of schooling. In fact, in many countries, content coverage is one of the important judgments usu- ally made by inspectors. Here we propose it as a statistical indicator based on teacher responses to survey items. Available and potentially available data. Regarding nationally collected data, the SASS includes questions about time allocated to general subjects such as science and mathematics. The survey could be amended to add questions about specific topics taught in each subject, linked to the core ideas and the mathematical, scientific, and engineering practices presented in the Common Core State Standards for Mathematics (National Governors Association and Council of Chief State School Officers, 2010) and A Framework for K-12 Science Education (National Research Council, 2012). However, doing so likely would increase respondent burden, and it might require subsampling mathematics and science teachers in particular grades. The existing NAEP science teacher survey includes some items that map well to the practices of science that are described in A Framework for K-12 Science Education. Data from these questions could be used immediately to provide baseline measurement. These surveys could also be modified and expanded to provide more thorough and comprehensive measures of classroom coverage of the content and practices of science and engineering. Appropriate questions about classroom coverage 20 R02309 Monitoring K–12 STEM Ed-PRF3.indd 20 3/13/13 2:05 PM

OCR for page 12
Monitoring Progress Toward Successful K-12 STEM Education of science and mathematics content and practices also could be added to future surveys that are developed in conjunction with new assessments in science and mathematics. Several other large-scale efforts have examined classroom coverage of science and mathematics content and have used other measures to validate teachers’ self-reported data. For example, the Surveys of Enacted Curriculum, which are used by hundreds of schools across the country, examine the content and cognitive demand of K-12 mathematics, science, and English language arts cur- riculum and could serve as a baseline for examining instructional change over time (Porter, Polikoff, and Smithson, 2009; Porter et al., 2007; Smithson and Blank, 2006). Although those surveys were not designed to be specifically tied to current standards documents, they could be adapted for that purpose. A different, large-scale effort to measure classroom coverage in Ohio and Michigan (Schmidt and McKnight, 2012) has used a Web-based survey of classroom coverage that was based on the international instruments developed in the Trends in International Mathematics and Science Study (TIMSS) and modified to reflect the Common Core State Standards for Mathematics. Data and research needs. Current efforts to measure classroom coverage are more developed for mathematics than for science, engineering, or career and technical education. To date, these efforts have focused more on measuring content than on the standards for mathematical practice (e.g., modeling with mathematics, reasoning abstractly and quantitatively) or the practices of sci- ence and engineering (e.g., analyzing and interpreting data, engaging in argument from evidence). Considerable research and development efforts are needed to develop measures for the coverage of mathematical practices, and for coverage of content and practices in engineering, science, and career and technical education. Such efforts might draw on previous and ongoing surveys (e.g., Porter, Polikoff, and Smithson, 2009; Schmidt and McKnight, 2012; Dorph et al., 2011; and Smith et al., 2002), while addressing well-documented concerns about the validity of teacher self-report data (Hudson, McMahon, and Overstreet, 2002). Enhanced Capacity of K-12 Teachers Teaching in ways that inspire students and deepen their understanding of science and mathemat- ics content and practices requires teachers to have content knowledge and expertise in teaching that content (National Research Council, 2010). Successful K-12 STEM Education recommended that “districts need to enhance the capacity of K-12 teachers” (National Research Council, 2011, p. 27), in part because many current teachers, including those who are teaching out of their field of exper- tise, are underprepared for the demands of STEM teaching. However, professional development in science and mathematics, when available, is often short, fragmented, not designed to address the specific needs of individual teachers, and therefore ineffective (National Research Council, 2011, pp. 20-21). Developing the requisite knowledge and teaching strategies will require profound changes to current systems for supporting teachers’ learning across their careers. 21 R02309 Monitoring K–12 STEM Ed-PRF3.indd 21 3/13/13 2:05 PM

OCR for page 12
Monitoring Progress Toward Successful K-12 STEM Education Key Indicators to Monitor Indicators to measure progress toward this recommendation of Successful K-12 STEM Education include baseline information about teachers’ content knowledge and knowledge of how to effec- tively teach science or mathematics (pedagogical content knowledge) and information about par- ticipation in high-quality professional development activities. Indicator 6. Teachers’ science and mathematics content knowledge for teaching. Content knowledge and knowledge of effective teaching and learning strategies are important components of teaching effectively. Teachers must integrate content and pedagogy in ways that reflect an understanding of “how students’ learning develops in [a given] field, the kinds of miscon- ceptions students may develop, and strategies for addressing students’ evolving needs” (National Research Council, 2010, p. 73). Thus, this indicator is intended to measure the depth of teachers’ understanding of the content they teach, as well as the knowledge needed for effective teaching (e.g., common misunderstandings that students have about a topic). Available and potentially available data. Teachers’ degrees and the courses they take in col- lege often are used as proxies for science or mathematics content knowledge. Other measures are needed, however, because research has shown that these indicators do not consistently predict stu- dent achievement at the secondary level (Wilson, Floden, and Ferrini-Mundy, 2001). In contrast, teachers’ own reports of their capacity to teach certain topics have been reliably shown to indicate content knowledge, as long as “stakes” are not attached to teachers’ responses (PROM/SE, 2006). Questions about teachers’ perceived capacity to teach certain topics in mathematics appear on the TIMSS surveys for grades 4 and 8. These questions could be amended so that they embody the key tenets of the Common Core State Standards for Mathematics and A Framework for K-12 Science Education (National Research Council, 2012). Questions that are linked to the core ideas in those documents also could be added to teacher questionnaires in the longitudinal studies conducted by NCES. Alternatively, such questions could be added to the SASS, or for grade 8 teachers on NAEP. Questions about teach- ers’ self-reported abilities to teach particular content areas also could be added to future surveys that are developed in conjunction with new assessments in mathematics and science. Whatever mecha- nism is used to collect these data, it will be important to ensure the confidentiality of responses so that teachers do not fear negative consequences for candid self-evaluations. Regarding more direct measures of teachers’ knowledge of mathematics and science for teach- ing, the Teacher Education and Development Study has measured the level of mathematics and related teaching knowledge that teachers acquire in their preparation programs in 18 countries (Tatto et al., 2008). In addition, the Learning Mathematics for Teaching Project has developed 22 R02309 Monitoring K–12 STEM Ed-PRF3.indd 22 3/13/13 2:05 PM

OCR for page 12
Monitoring Progress Toward Successful K-12 STEM Education surveys that measure teachers’ knowledge for teaching mathematics and it has used those surveys to understand how teachers acquire mathematical knowledge for teaching, and how that knowl- edge relates to students’ achievement in mathematics (Hill, Schilling, and Ball, 2004; Hill and Ball, 2004; Hill, Rowan, and Ball, 2005). Work to develop similar instruments for science is under way (e.g., Smith and Taylor, 2010), but faces challenges such as the need to measure content knowledge for multiple subject areas. Data and research needs. Although there is widespread agreement that teacher content knowl- edge for teaching is essential, research on this issue is at an early stage. Expanding the emerging research on direct measures of teacher knowledge in STEM fields is needed, as are measures of teachers’ abilities to integrate content knowledge with understanding of student thinking. Because these efforts are more mature for mathematics, greater investments will be needed to develop simi- lar measures for science and engineering. Indicator 7. Teachers’ participation in STEM-specific profes- sional development activities. This indicator is designed to measure teachers’ participation in high-quality, research-based professional development in science and mathematics. Although the research is not conclusive, there is emerging consensus that high-quality professional development (a) focuses on developing teachers’ capabilities and knowledge to teach content and subject matter, (b) addresses teachers’ classroom work and the problems they encounter in their school settings, and (c) provides multiple and sustained opportunities for teacher learning over a substantial time interval (National Research Council, 2011, p. 21). Because the challenges of teaching science and mathematics and of provid- ing quality professional development differ by grade level and subject matter, progress toward this indicator should be tracked along those dimensions. A desirable goal is for the content of a professional development program in STEM to be rooted in practices that have been found to be effective in studies that use valid scientific designs such as experiments or quasi-experiments. However, a review of 1,343 studies of professional develop- ment revealed that only 9 of them had the types of designs—randomized control trials or quasi- experimental designs—that allow causal inferences about the effectiveness of the professional development strategies they examined (Yoon et al., 2007). To assess changes in teachers’ instruc- tional practice as well as changes in students’ learning, these kinds of studies should be contrasted with “customer satisfaction” reports that often are compiled about whether the audience “liked” the professional development activity or the speaker. Although such satisfaction reports might be appropriate for monitoring whether users believed a given professional development activity was useful for them, they do not convey information about whether the activity was designed around evidence-based practices. 23 R02309 Monitoring K–12 STEM Ed-PRF3.indd 23 3/13/13 2:05 PM

OCR for page 12
Monitoring Progress Toward Successful K-12 STEM Education Available and potentially available data. Participation in professional development will be measured in the 2013 OECD Teaching and Learning International Survey, in which the United States will participate. In addition, the SASS asks whether teachers have participated in any pro- fessional development specific to and concentrating on the content of the subject(s) they teach. Mathematics and science teachers can be identified in this manner, but the surveys do not provide information about mathematics- or science-specific professional development for teachers who do not designate a single subject as their main teaching assignment (e.g., elementary school teach- ers). Although it would be possible to restrict the sample from the 2007-2008 SASS to secondary mathematics and science teachers and describe their responses to questions about whether they participated in professional development activities specific to the subjects they teach, the resulting data set would leave the field uninformed about the science professional development received by elementary teachers. The 2007-2008 SASS asked questions about the duration and perceived value of professional development, and the 2003-2004 SASS asked questions related to professional development that involves peers and shared planning time. Data from the 2003-2004 survey could be compiled to provide some insights into the nature of professional development for mathematics and science teachers, and the relevant questions from that survey could be reused in future versions of the SASS. Survey questions from the evaluation of the Eisenhower Professional Development Program (Garet et al., 2001), ongoing surveys of professional development for mathematics and science teachers (Banilower et al., 2006), and California surveys of science education (Dorph et al., 2011; Hartry et al., 2012) also could be adapted for this purpose. Relevant questions could be added to new questionnaires that might be developed in conjunction with mathematics and science assess- ments, which would offer the opportunity to link student achievement data to teachers’ participa- tion in professional development. Data and research needs. Although knowledge is accumulating on professional development (Desimone, 2009; Hochberg and Desimone, 2010), additional research is needed to determine the characteristics of effective professional development for science and mathematics teachers (e.g., Garet et al., 2011), and potential obstacles to engaging in professional development. Such research could evaluate the quality of existing professional development activities, the range of opportuni- ties that might lead to improvements in teachers’ practice, and the extent to which they are aligned to instructional policies such as the Common Core State Standards. Because research that links professional development activities to student outcomes is especially sparse, additional research in this area would be particularly valuable. A sharper focus on understanding the prevalence of evidence-based practices in professional development also would bolster this indicator; however, no system currently exists for doing so. 24 R02309 Monitoring K–12 STEM Ed-PRF3.indd 24 3/13/13 2:05 PM

OCR for page 12
Monitoring Progress Toward Successful K-12 STEM Education Professional Development for Instructional Leaders Although it is necessary to have qualified, capable teachers, research has shown that school con- text matters just as much as teachers’ qualifications (DeAngelis and Presley, 2011; McLaughlin and Talbert, 2006). Longitudinal research in Chicago elementary schools identified five common elements shared by elementary schools that improve reading and mathematics scores (Bryk et al., 2010): (1) school leadership as the driver for change, (2) professional capacity of school staff, (3) strong ties with parents and the community, (4) a student-centered learning climate, and (5) instructional guidance for teachers. Schools that are strong in these areas are much more likely to improve student learning than schools that are not strong in these areas, and these supports have been associated with improved learning even in neighborhoods of extreme hardship and poverty. The available evidence does not indicate whether these supports also improve science achieve- ment, and whether the same supports are important in middle and high schools, but it is clear that strong school leadership is vital to create school conditions and cultures that support successful education in all subjects (Bryk and Driscoll, 1988; Stein and Nelson, 2003). Thus, Successful K-12 STEM Education recommended that “districts should provide instructional leaders with profes- sional development that helps them to create the school conditions that appear to support student achievement” (National Research Council, 2001, p. 27). Key Indicator to Monitor The recommendation from Successful K-12 STEM Education was intended to ensure that school lead- ers receive supports to create the school conditions identified above. The committee proposes one indicator to provide descriptive information about the extent to which school principals participate in high-quality, research-based professional development to help them create those conditions. It does not measure the prevalence of the school conditions that support learning, which would require another set of indicators and additional research to develop those indicators. Indicator 8. Instructional leaders’ participation in professional development on creating conditions that support STEM learning. Professional development for leaders should be defined broadly, and could include activities such as coaching and time to discuss work with peers. Available and potentially available data. The SASS asks about participation in general pro- fessional development for school principals, but it does not ask about the characteristics or quality of that professional development or about professional development related to instructional leader- ship in specific subjects. Data are needed to provide information on program focus, duration, and 25 R02309 Monitoring K–12 STEM Ed-PRF3.indd 25 3/13/13 2:05 PM

OCR for page 12
Monitoring Progress Toward Successful K-12 STEM Education the presence of features that research has shown to be effective. The SASS could be modified to address specific subjects and the issue of quality of professional development for school leaders. Relevant questions on the 2003-2004 and 2007-2008 SASS for teachers about the nature, dura- tion, and perceived value of professional development also could be adapted for principals. A set of similar questions about school leaders’ exposure to professional development also could be added to the science section of the NAEP school questionnaire for administrators, or to questionnaires that might be developed in conjunction with assessments related to the Common Core State Standards for Mathematics and A Framework for K-12 Science Education. Data and research needs. Additional research would be needed to identify what effective instructional leadership in mathematics and science might look like (e.g., observation and feed- back on instructional practices, observing teachers during collaborative activities, supporting the work of math and science coaches) and then to evaluate the quality of various professional development opportunities that promote effective leadership, similar to work by Cobb et al. (in press). Data on the alignment of the focus of the professional development with STEM disci- plines also would be needed. Elevated Status for Science The recommendation from Successful K-12 STEM Education to “elevate science to the same level of importance as reading and mathematics” (National Research Council, 2011, p. 28) was developed in response to an unintended consequence of the emphasis on mathematics and reading created by the current federal accountability system, noted above. The committee’s proposed indicators to address this recommendation are directly related to the indicators on devoting more time and resources to science instruction and are designed to encourage state and federal policy makers to remedy the current imbalances. Key Indicators to Monitor Funding levels, accountability structures, and legislative mandates provide the most meaningful indicators of the value policy makers place on science education. Indicator 9. Inclusion of science in federal and state account- ability systems. For each state, this indicator would measure whether science is assessed, whether science is includ- ed in the state accountability system, and whether science counts as much as reading and math- ematics in that accountability system. Similarly, at the federal level, the indicator would capture whether or not the reauthorized Elementary and Secondary Education Act includes accountability provisions for science, and if so, in what grades. 26 R02309 Monitoring K–12 STEM Ed-PRF3.indd 26 3/13/13 2:05 PM

OCR for page 12
Monitoring Progress Toward Successful K-12 STEM Education Available and potentially available data. The Council of Chief State School Officers (CCSSO) has conducted surveys of states’ accountability and assessment policies. Analyses of the survey results have reported whether science is assessed, but not whether science is part of the accountability system (Stillman and Blank, 2008). State applications for flexibility under the No Child Left Behind Act do indicate whether the states that have applied include science in their accountability systems: to date, less than half of them do. Data and research needs. If the CCSSO survey cannot be modified to include questions about whether science is part of a state’s accountability system, another organization that routinely conducts surveys of state educational policies, such as Education Week or the National Governors Association, might conduct a 50-state survey of the role of science in state accountability systems. Indicator 10. Inclusion of science in major federal K-12 edu- cation initiatives. Indicators of a federal commitment to science education could include the level of federal funding for the Next Generation Science Standards assessment relative to the Common Core State Standards for English Language Arts and Mathematics; the inclusion of science in major federal incentive programs such as Race to the Top; whether funding in support of science education is included in the reauthorization of the Elementary and Secondary Education Act; and the frequency of science assessment in NAEP, compared with reading and mathematics. Available and potentially available data. Information about these indicators could be gath- ered by analyzing publicly available budget data; federal solicitations for the largest grant programs and other competitive funding; and eventually, language in the reauthorized Elementary and Secondary Education Act. Information about the frequency of NAEP assessments is well known and readily available from the NCES. Indicator 11. State and district staff dedicated to supporting science instruction. This indicator is intended to measure the human resources that are available for science relative to mathematics and reading for all grade levels. It should include the various types of human resources that are devoted to these subjects at the district and state levels, such as curriculum specialists, coaches, and teacher specialists. For example, as research identifies the expertise needed for high- quality teaching of specific content domains, elementary schools are increasingly moving away from the traditional model of one elementary teacher who teaches all four content areas and begin- ning to use specialists (e.g., Association of Mathematics Teacher Educators, 2010). These special- ists are often district staff who serve multiple schools and take on a variety of roles, including 27 R02309 Monitoring K–12 STEM Ed-PRF3.indd 27 3/13/13 2:05 PM

OCR for page 12
Monitoring Progress Toward Successful K-12 STEM Education teaching students, managing materials, planning with the classroom teachers, conducting profes- sional development, and offering demonstration lessons (Century, Rudnick, and Freeman, 2008). Available and potentially available data. No data are currently collected that provide com- plete information about this indicator on a large scale. The SASS includes questions about the percentage of schools with science and mathematics coaches and science specialists. The SASS district survey, the CCSSO 50-state survey, or Education Week’s Quality Counts survey could be used to collect district and state data, respectively. The Common Core of Data state survey includes questions about instructional coordinators and supervisors; these questions could be revised to be specific to reading, mathematics, and science. Relevant questions from a survey of science educa- tion in California (Dorph et al., 2011) also might be adapted for use on a national scale. Data and research needs. Additional research is needed to learn whether elementary students’ science learning opportunities are of high quality and whether their science learning is enhanced when they receive science instruction from a science specialist rather than a generalist teacher. Research on the most effective use of content-focused coaches (e.g., modeling lessons, coaching indi- vidual teachers, providing professional development to groups of teachers) also would be productive. Effective Systems of Assessment Indicator 10 is proposed to suggest that states and the federal government can help elevate science to the same level of importance as reading and mathematics by including more science assessment in accountability systems. Yet, as with reading and mathematics assessments, it is crucial that sci- ence assessments support, rather than undercut, effective science instruction. Thus, in addition to developing quality assessments to accompany the Common Core State Standards for Mathematics, it is necessary to develop science assessments that reflect current research on teaching and learning and that emphasize the practices of science before including science in accountability systems. Indeed, Successful K-12 STEM Education recommended that “states and national organizations should develop effective systems of assessment that … emphasize science practices rather than mere factual recall” (National Research Council, 2011, p. 28). Key Indicator to Monitor The committee proposes one indicator related to state assessment systems. Collecting data for this indicator involves analyzing the mathematics and science assessment systems that states adopt in the coming years, to determine the extent to which they embody the Common Core State Standards for Mathematics and the vision for science education in A Framework for K-12 Science Education (National Research Council, 2012) or the extent to which those assessment systems otherwise emphasize mathematical and science practices in addition to concept mastery. In this way, it will be possible to identify states with assessment systems that support effective instruction. 28 R02309 Monitoring K–12 STEM Ed-PRF3.indd 28 3/13/13 2:05 PM

OCR for page 12
Monitoring Progress Toward Successful K-12 STEM Education We include mathematics in this indicator because the concern about quality assessments is not limited to science. If this indicator is to measure assessments that accompany new standards in mathematics and science, no data could be collected on it until the next generation of assessments has been developed and states have adopted them. Indicator 12. States’ use of assessments that measure the core concepts and practices of science and mathematics disciplines. The committee proposes a two-tiered data collection for this indicator—similar to that proposed for Indicator 4—that involves determining which assessments states use and then analyzing the most commonly used assessments for their consistency with standards documents. Available and potentially available data. The CCSSO has conducted a 50-state survey of state assessment polices, and their report includes the most commonly used assessments in science (Stillman and Blank, 2008). The survey should be administered again when states adopt assessments related to the Common Core State Standards for Mathematics and A Framework for K-12 Science Education. However, such a survey would only provide information on the first tier of data collec- tion: which assessments states use. Data and research needs. If the CCSSO 50-state survey cannot be readministered, similar surveys of state departments of education, analyses of state education websites, and other means could yield information about what assessments are being used by each state, as well as accom- modations that are offered students. Other research would need to be conducted on the qualities of the assessments, including their rigor, reliability, and validity. Efforts to develop a procedure for analyzing the degree to which assessments embody the standards or emphasize the prac- tices of science could build on similar efforts to evaluate curricular materials by the American Association for the Advancement of Science (2005); Schmidt et al. (1997); and Gueudet, Pepin, and Trouche (2012). Federal and State Support for STEM Teachers Successful K-12 STEM Education recommended that “National and state policy makers should invest in a coherent, focused, and sustained set of supports for STEM teachers to help them teach in effec- tive ways” (National Research Council, 2011, p. 28). Although districts bear most of the respon- sibility for providing professional development for teachers, federal and state agencies provide the bulk of funding for these activities. Most states and districts do not develop and implement strategic plans to bolster science and mathematics teaching, which poses a significant hurdle to supporting teachers’ continued growth and development throughout their careers (Borko, 2004; Wilson, Rozelle, and Mikeska, 2011). Many districts lack a mechanism for coordinating a focused 29 R02309 Monitoring K–12 STEM Ed-PRF3.indd 29 3/13/13 2:05 PM

OCR for page 12
Monitoring Progress Toward Successful K-12 STEM Education portfolio of professional development that is aligned with instructional reforms. Some schools send single volunteers to a professional development program, with the understanding that a teacher who returns will share what she has learned with her colleagues. Other schools have professional development offered by the developers of new textbooks or instructional materials that they have adopted. In yet other schools, teachers participate in professional development through NSF proj- ects at local universities or educational organizations. A coherent strategy for investing in high-quality learning opportunities for science and mathemat- ics teachers would represent a promising step toward enhancing and maintaining their capacity to teach (Cobb and Smith, 2008). Such a strategy might include crafting teacher induction programs that are aligned with later professional development so that teachers can deepen their understand- ing of STEM disciplines, STEM teaching, and how their students understand STEM concepts. It also might involve identifying the core concepts that teachers need to understand to be able to teach to the Common Core State Standards and the core ideas, practices, and crosscutting con- cepts in A Framework for K-12 Science Education (National Research Council, 2012) and developing a long-term professional development program that would gradually deepen teachers’ knowledge and skill in teaching those topics. Key Indicator to Monitor Efforts to monitor progress toward this recommendation from Successful K-12 STEM Education should concentrate on the kinds of professional development activities that are supported by federal and state funding, in order to determine whether that funding is supporting coherent activities that are consistent with best practices identified by the research. In turn, these efforts should be used to generate information about the characteristics of professional development activities that lead to changes in teaching practice and to improved student outcomes. Indicator 13. State and federal expenditures dedicated to improving the K-12 STEM teaching workforce. This indicator should measure expenditures on preparation, recruitment, induction, recruitment, and subject-specific professional development over teachers’ careers. Examples of programs whose expenditures might be tracked include UTeach, NSF’s Mathematics and Science Partnerships, the California State University’s Math and Science Teacher Initiative, and the National Aeronautics and Space Administration (NASA) Pre-service Teacher Institute. The indicator should enable dis- tinction among different kinds of activities, to facilitate future analyses of the extent to which those activities are evidence based. Available and potentially available data. The NCES does not collect data on this indicator in a consistent way. At the federal level, this information could be gathered by identifying and 30 R02309 Monitoring K–12 STEM Ed-PRF3.indd 30 3/13/13 2:05 PM

OCR for page 12
Monitoring Progress Toward Successful K-12 STEM Education totaling the grants or grant programs that support teacher training and professional development at NSF, the U.S. Department of Education (Title II), and other agencies such as NASA and the U.S. Department of Energy. This approach could also be repeated for each state, though it would be considerably more complex and time consuming. Data and research needs. New kinds of data collection would be needed to more systemati- cally track district and state investments in recruitment, induction, and professional development. Addressing the research needs identified under Indicators 7 and 8 also would help to identify whether the spending on these activities is evidence based. Research to Enhance Understanding of STEM Schools, Practices, and Outcomes Successful K-12 STEM Education recommended that “federal funding for STEM-focused schools should be tied to a robust, strategic research agenda” (National Research Council, 2011, p. 28). That recommendation also noted that such an agenda would include research that • disentangles the effects of school practice from student selection; • takes into account the importance of contextual variables on teaching, learning, and student outcomes; and • allows for longitudinal assessment of student outcomes, including the three strategic goals of U.S. education in STEM and intermediate outcomes relative to those goals. These research needs can be met with a variety of methodologies. For example, it would be useful to have descriptive information on the number of people, schools, districts, and/or states offering or participating in a given activity or practice (e.g., the number of STEM-focused schools or pro- grams, principals’ participation in professional development) and the degree to which these activi- ties are addressing the needs of students from traditionally low-performing populations. Contrast studies with comparison or control groups would allow for determinations of the effectiveness of instructional strategies or models, curricula, and professional development activities in improv- ing student learning, engagement, and persistence. Interview studies, in-depth case studies, and classroom observations would be useful for understanding how practices and policies are being implemented, and for illuminating contextual influences on the teaching and learning process. And by blending theory building and the development of design principles, design-based research would guide, inform, and improve practice and research (Anderson and Shattuck, 2012). Research also is needed to develop appropriate instruments and to learn how to support implementation at scale of effective science and mathematics initiatives. Regardless of the design, this research should be conducted on a focused set of topics to generate information on the most pressing questions in science and mathematics education. 31 R02309 Monitoring K–12 STEM Ed-PRF3.indd 31 3/13/13 2:05 PM

OCR for page 12
Monitoring Progress Toward Successful K-12 STEM Education Key Indicator to Monitor Indicator 14. Federal funding for the research identified in Successful K-12 STEM Education. Available and potentially available data. The U.S. Department of Education’s Institute for Education Sciences (IES), various NSF directorates, and the National Institutes of Health (NIH) are the primary funders of the kinds of research recommended in Successful K-12 STEM Education. Data on the types of research funded and the knowledge accumulated from those investments currently are not compiled in a way that would provide readily accessible information for this indicator. Data and research needs. Information on this indicator could be gleaned by analyzing the portfolio of grants awarded by IES, NSF, and NIH. That analysis could examine how many stud- ies, and what proportion, addressed the research gaps identified in Successful K-12 STEM Education. Another approach would be to examine the pattern of publications in key journals, such as the American Educational Research Journal, Educational Evaluation and Policy Analysis, the Journal for Research in Mathematics Education, and the Journal for Research in Science Teaching.6 That examination could deter- mine how many studies address each of the gaps identified in Successful K-12 STEM Education and whether the pattern is changing over time. 6In2003, IES conducted an analysis similar to the one proposed here. The presentation of that analysis is available at http://ies. ed.gov/director/speeches2003/04_22/2003_04_22b.asp [August 2012]. 32 R02309 Monitoring K–12 STEM Ed-PRF3.indd 32 3/13/13 2:05 PM