CHAPTER 8
USING THE FRAMEWORK

What does it mean to make use of the Framework in investigating possible influences of nationally developed standards on mathematics, science, and technology education? In addressing that question, this chapter first recaps the argument for the Framework; then describes and illustrates ways of using the Framework; and, finally, presents the Committee’s aspirations for its use of such a framework.

Although this chapter outlines possible uses of the Framework, the Committee stops short of presenting a research agenda, which was not part of the Committee’s charge. In illustrating how the Framework can be applied, the chapter considers what will be involved in seeking sound, useful answers to this document’s overarching questions about the influence of nationally developed standards on programs, policies, practices, and student learning.

At this stage in the life cycle of national standards—more so for mathematics and science than technology—educators and policy makers would benefit from concerted efforts to address the two core questions: How has the system responded to the introduction of nationally developed standards? and What are the consequences for student learning?

THE FRAMEWORK IN REVIEW

This document’s central argument began with the premise that to answer questions about the effects of nationally developed standards on the education system and student learning, a Frame-



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 79
Investigating the Influence of Standards: A Framework for Research in Mathematics, Science, and Technology Education CHAPTER 8 USING THE FRAMEWORK What does it mean to make use of the Framework in investigating possible influences of nationally developed standards on mathematics, science, and technology education? In addressing that question, this chapter first recaps the argument for the Framework; then describes and illustrates ways of using the Framework; and, finally, presents the Committee’s aspirations for its use of such a framework. Although this chapter outlines possible uses of the Framework, the Committee stops short of presenting a research agenda, which was not part of the Committee’s charge. In illustrating how the Framework can be applied, the chapter considers what will be involved in seeking sound, useful answers to this document’s overarching questions about the influence of nationally developed standards on programs, policies, practices, and student learning. At this stage in the life cycle of national standards—more so for mathematics and science than technology—educators and policy makers would benefit from concerted efforts to address the two core questions: How has the system responded to the introduction of nationally developed standards? and What are the consequences for student learning? THE FRAMEWORK IN REVIEW This document’s central argument began with the premise that to answer questions about the effects of nationally developed standards on the education system and student learning, a Frame-

OCR for page 79
Investigating the Influence of Standards: A Framework for Research in Mathematics, Science, and Technology Education work is needed to represent the education system, how reform ideas (such as standards) move through the system, and possible system and learner responses to the standards. In essence, such a “framework” represents a conceptual map—a set of organizing categories and presumed relationships among them—and a set of guiding questions to prompt inquiry within the map’s territory. Schematically, this document’s Framework can be represented as in Figure 8–1. FIGURE 8–1 A Framework for Investigating the Influence of Nationally Developed Standards for Mathematics, Science, and Technology Education NOTE: A full-sized copy of this Framework appears as Appendix B.

OCR for page 79
Investigating the Influence of Standards: A Framework for Research in Mathematics, Science, and Technology Education The logic implicit in the Framework can be summarized through a group of interrelated propositions: Nationally developed standards in mathematics, science, and technology represent a set of fundamental changes in the way these subjects have traditionally been taught, placing new demands on teachers and students. The changes needed to move from established modes of teaching and learning to those advocated by the standards imply considerable new learning for both teachers and students. Standards-based practice presumes that teachers understand and have internalized much of what is asserted by national standards documents (e.g., that a core set of important ideas and skills in each content area can be identified and that all students can master those fundamental expectations). The expected influence of nationally developed standards on teaching practice and student learning is likely to be (a) indirect, taking place through proximate effects on other parts of the education system; (b) entangled (and sometimes confused) with other influential forces and conditions, such as broader state standards-based reforms; and (c) slowly realized and long term. In other words, within the nation’s decentralized system of education, notions of teaching and learning embedded in nationally developed standards do not have immediate pathways into classrooms. Rather, as these ideas move through channels that cross multiple levels of governance, various forces can alter how the standards are understood and acted upon. State-level standards-based reform movements, for example, have introduced numerous interpretations of “standards” and “assessments,” some of which may not be in accord with ideas conveyed in the National Research Council (NRC), the National Council of Teachers of Mathematics (NCTM), and the International Technology Education Standards (ITEA) standards documents. Given the amount of new learning implied and the complexity of the education system, it would take a long time, if ever, before the visions conveyed by national standards documents would be fully realized. Also,

OCR for page 79
Investigating the Influence of Standards: A Framework for Research in Mathematics, Science, and Technology Education because national standards in the three subject areas have had different timelines, varying degrees and types of influence across subject areas are to be expected at any one time. Three core channels exist within the education system through which nationally developed standards can influence teaching and learning. These channels of influence are (a) curriculum; (b) teacher development; and (c) assessment and accountability. The channels of influence are complex and interactive, and differ across subject-matter communities. In other words, the channels operate differently within mathematics, science, and technology, creating different opportunities for—or barriers to—influence by the standards. Jointly or separately, the channels may alter the way standards are understood and realized. Public, political, and professional reactions can also affect these channels and shape the way standards reach and influence teaching and learning. Variability within the education system implies that students and teachers are likely to experience different influences, depending on locality, resources, participant background, and other factors. Consequently, educational effects of national standards are unlikely to be monolithic. Instead, there may be effects that are constructive and others that are counterproductive, some weak, and others strong. The task for research—and hence for this Framework—is to help identify and document significant standards-based effects, as well as overall trends and patterns among those effects. That is, the task is to provide evidence-based descriptions of the channels and mechanisms through which those effects take place and determine what conditions may be associated with particular effects. The ultimate focus is on the changes in students’ knowledge and abilities that have occurred since standards have entered the system and that can be reasonably attributed to the influence of the standards. As part of this, it is essential to consider how standards have affected the achievement of all students, including those who were previously underrepresented in mathematics, science, and technology. Eventually, nationally developed standards will be judged

OCR for page 79
Investigating the Influence of Standards: A Framework for Research in Mathematics, Science, and Technology Education effective if resources, requirements, and practices throughout the system align with the standards and if students in standards-based classrooms demonstrate high achievement in knowledge and skills deemed important. Although there may be other grounds upon which individuals or groups elect to accept or reject the standards, the only empirical approach for making that judgment presumes that standards have had opportunities to permeate the education system, and, having done so, are associated with student-learning outcomes that can be judged as desirable or undesirable. HOW THE FRAMEWORK CAN BE USED The Framework is intended to help guide the sponsorship, design, and interpretation of research on nationally developed standards. The challenge is far from simple. The Framework lays out a complex domain of interacting forces and conditions that affect teaching and learning, any number of which can be touched by the influence of standards. Thus, no single study can investigate all the ways that national standards are, or could be, part of the education reform story. Rather, various types of studies, each guided by its own appropriate methodologies, will be needed to establish the scale and scope of influences, identify routes by which standards actually exert influence, and ascertain the direction and educational consequences of those influences. Table 8–1 contains several hypothetical examples that illustrate how different macro and micro studies can “cover” the terrain of the Framework, and respond to one or more questions posed earlier in this document. Each of those hypothetical studies addresses only part of the broad territory embraced by the Framework. Multiple studies could collectively paint a more satisfactory picture of the effects of nationally developed standards if they were designed to generate complementary databases and were carefully synthesized. In carrying out such research, the Framework offers assistance in several important ways: (1) situating existing studies within the

OCR for page 79
Investigating the Influence of Standards: A Framework for Research in Mathematics, Science, and Technology Education TABLE 8–1 Hypothetical Studies That Address One or More of the Framework Questions   How are the nationally developed standards being received and interpreted? What actions have been taken in response to the nationally developed standards? What has changed as a result of nationally developed standards? Who has been affected and how? Analysis of media coverage of mathematics education can determine how often national standards are mentioned, which components of the standards are highlighted, how they are interpreted, and what value is attached to each. ●       Expert reviews of documents can identify ways that state standards, assessments, and accountability systems may and may not reflect the content advocated in the national standards. ●       National teacher surveys can reveal how aware teachers are of national standards, whether—and in what ways—they believe they are orienting their professional practices to these standards, and in what ways they are supported in their efforts to realize the standards. ● ● ●  

OCR for page 79
Investigating the Influence of Standards: A Framework for Research in Mathematics, Science, and Technology Education Comparative studies of reform-based curricula at a particular grade level can build understanding about how curriculum developers interpret the standards and how those interpretations may affect what students have opportunities to learn. ● ●     Observational studies can reveal whether— and how—technology standards are realized in classroom practice within particular kinds of school settings.   ● ●   Case studies of district reform can explore the alignment of science curriculum and assessment policies with national standards and the nature and extent of district support teachers receive for teaching in standards-aligned ways.   ●     Quasi-experimental design studies can compare teachers’ responses to standards and their students’ performance in settings with differing degrees of exposure to, and support for, standards-based practices.     ● ● Case studies of standards-based classrooms can explore whether teachers adjust instruction appropriately, with respect for students’ cultural backgrounds as well as ways of learning.     ● ●

OCR for page 79
Investigating the Influence of Standards: A Framework for Research in Mathematics, Science, and Technology Education educational terrain relevant to the standards; (2) providing a conceptual tool for analyzing claims and inferences made by these studies; and (3) generating questions and hypotheses to be explored by future studies. Situating Current Studies The Framework can assist researchers in locating their work within a particular frame of reference and may highlight possible connections or lack of connections to other parts of the education system. Sponsors, investigators, and consumers of research findings should keep in mind aspects of the education territory that may and may not be addressed by particular studies or programs of investigation. Consider this study of standards implementation in a large urban district: Standards, Assessments, and What Else? The Essential Elements of Standards-Based School Improvement (Briars and Resnick, 2000). The Pittsburgh Public Schools developed a core curriculum framework based on the NCTM Standards, adopted a standards-based assessment system (New Standards Mathematics Reference Examination) to be used in conjunction with the Iowa Test of Basic Skills, and adopted the National Science Foundation (NSF)-funded Everyday Mathematics program for grades K-5. An NSF Local Systemic Change grant provided funding for extensive professional development to prepare teachers to teach that curriculum. A study was conducted to evaluate the effects of this “nearly complete standards-based system” in mathematics, looking at student achievement on both the standards-based and traditional assessments. Recognizing that teachers varied in the extent to which they were implementing the curriculum as intended by the developers, the researchers also compared the performance of students in strong and weak implementing classrooms, disaggregating the data to see if differential results were obtained for groups defined by race/ethnicity.

OCR for page 79
Investigating the Influence of Standards: A Framework for Research in Mathematics, Science, and Technology Education FIGURE 8–2 Parts of the Framework Addressed in the Briars and Resnick Study This descriptive study can be mapped onto the Framework as shown in Figure 8–2. The shaded areas identify aspects of the Framework and related questions that are addressed in this analysis. Studies with different purposes, designs, and evidence bases would cover different parts of the Framework. Consider this analysis: Mathematics and Science Content Standards and Curriculum Frameworks (Council of Chief State School Officers, 1997). An expert panel reviewed state frameworks, standards documents, and related materials developed or revised during the period 1994 to

OCR for page 79
Investigating the Influence of Standards: A Framework for Research in Mathematics, Science, and Technology Education 1997. The analysis sought to determine the extent that state curriculum frameworks, standards, and other materials were consistent with NCTM Curriculum and Evaluation Standards for School Mathematics, NRC National Science Education Standards, and American Association for the Advancement of Science Benchmarks for Scientific Literacy. The analysis also considered differences in content found in state mathematics and science frameworks or standards documents and main ideas and categories found in corresponding national standards, noting omissions and additions. The analysis pointed out how state documents acted as a “bridge” between nationally developed standards and local efforts to improve teaching and learning in these subject areas. In contrast to the more broad-based Briars and Resnick study, this investigation focused in depth on one Framework component: State Policy Decisions within the Curriculum channel. Thus, shading would highlight only that particular feature of the Framework. Examining Claims and Inferences Reported in Current Studies In addition to helping locate relevant areas of research, the Framework offers a conceptual tool for assessing claims made by researchers. By highlighting multiple influences on teaching and learning, the Framework can suggest plausible alternative explanations for research findings. Referring to the Framework, scholars and other consumers of research can decide whether investigators accounted for all the plausible channels of influence on teaching and learning within the settings under study. Without reference to a conceptual map such as the Framework, weak inferences may arise about the influence of standards. Ultimately, strong claims about positive or negative effects of nationally developed standards depend on a chain of evidence and inference linking promulgation of standards (at the national level) to particular sites (in schools and classrooms) within which standards-based

OCR for page 79
Investigating the Influence of Standards: A Framework for Research in Mathematics, Science, and Technology Education ideas may be found to exert influence. As noted earlier, developing such a chain of evidence and inference will usually require multiple, coordinated studies. An important caution: The Framework provides only a conceptual scheme for considering research claims, not all the information needed to assess research-based claims fully. A full analysis must include a host of “technical” considerations, such as standards of evidence, quality of measurement, and appropriateness of the research design. All of these concerns must be addressed in deciding whether particular research conclusions are trustworthy and rigorous. Assuming comparable technical quality, several hypothetical examples illustrate how the Framework can help determine the soundness of research-based inferences. A study of standards-based classroom practice in mathematics. Imagine an investigation of standards-based mathematics teaching practice in a high socioeconomic environment that supports this kind of instruction, using a curriculum that embodies the principles of the NCTM standards. Assume that teachers have been well trained in this form of teaching and are committed to it. If, after sufficient time passes for the curriculum to have affected student learning across grades, a well-designed study documents indifferent or poor student results on assessments keyed to NCTM standards, it would be reasonable to infer that national mathematics standards contributed little to student learning—or might even have detracted from it. That inference could be further substantiated if other school settings less committed to NCTM standards produced more favorable results with comparable students. A study of district investment in teacher professional development aligned with nationally developed content standards for technological literacy. In a group of districts heavily emphasizing the principles and themes of ITEA content standards in their professional development and support programs, assessments of teacher knowledge

OCR for page 79
Investigating the Influence of Standards: A Framework for Research in Mathematics, Science, and Technology Education and pedagogical approaches might show that they have acquired the intended knowledge and skills, and that the teachers are attempting to realize these ideas in their teaching. Assuming a well-designed investigation into teachers’ participation in standards-oriented professional development and the outcomes of that participation (including direct observations in their classrooms), it would be reasonable to infer that ITEA content standards had contributed to changes in those teachers’ thinking and practice. (Establishing this particular claim does not necessarily imply that students learned more; that inference would require a different study, or an additional component to this investigation.) Once again, comparisons with other school sites less invested in standards-related content would help to establish the claim. Note that the Framework helps to establish the conceptual soundness of research inferences by highlighting elements of the domain that, through a reasonable chain of evidence and inference, link national standards to classroom outcomes. By contrast, the following hypothetical examples involve unwarranted conceptual leaps in their reported conclusions: Analysis of student achievement gains in states that align their mathematics standards with NCTM standards. Impressive student achievement gains in states that apparently embrace national mathematics standards invite the possible conclusion that the standards contributed to the improvements in student performance. But even assuming a technically sound analysis of test score trends that took into account known correlates of student achievement scores (e.g., student socioeconomic status), the inference is weak at best, or even fallacious, if the analysis did not consider other components highlighted by the Framework. Those conditions include alignment of the mathematics achievement measures with the standards, local interpretation of state and national curricular guidance, and the extent of standards-based classroom practice. In

OCR for page 79
Investigating the Influence of Standards: A Framework for Research in Mathematics, Science, and Technology Education the absence of those considerations, there are too many other plausible explanations for the achievement gains to place any confidence in the inference that national standards had anything to do with them. An investigation of declining science scores in a district committed to NRC standards. Declines in student performance on district science assessments within a setting that has tried to encourage standards-related instruction may suggest to observers that the national standards are detrimental to student learning. Even if the investigation were carefully designed and executed, it would not support that conclusion, unless relevant components highlighted by the Framework were taken into account: alignment between the district’s science assessments and the curriculum, teachers’ interpretations of the standards and attempts to realize them in classroom instruction, and the extent of professional development for teachers unfamiliar with standards-based classroom practice. In short, consumers of research, with the Framework in hand, can examine the results and conclusions of studies—or sets of studies—guided by questions such as these: Does the study… establish a plausible, evidence-based chain of influence that connects nationally developed standards to particular elements of the system under investigation? The Framework highlights components that might be part of that chain of influence. address plausible alternative explanations that could be advanced to account for observed effects or outcomes? The Framework highlights alternative forces and conditions that may influence effects or outcomes. consider interactions among different channels of influence that can convey either mutually reinforcing or contradictory messages to teachers and schools about standards-based practices? The Framework lays out

OCR for page 79
Investigating the Influence of Standards: A Framework for Research in Mathematics, Science, and Technology Education the three primary channels of influence by which national standards could affect teaching and learning and notes ways these can interact with one another. allow sufficient time for the education components under investigation to have been affected by nationally developed standards? The Framework demonstrates the complexity of the system through which messages about standards-based practice must move, thus the effects may become visible only after an extended period of time. Again, these are not all of the important questions to be asked about the findings and conclusions of research related to nationally developed standards in mathematics, science, or technology education. Other important questions include congruence of the research design with the research questions, execution of the design, adequacy of the database, and quality of data-analysis approaches. Still, the Framework establishes a conceptual map that provides relevance and meaning for answers to such additional questions. Generating Questions and Hypotheses for Future Investigations The Framework can help pinpoint areas of potential influences operating within the education system that may or may not have been considered by particular studies. This third main application of the Framework has two parts: Assembling knowledge. The Framework offers a basis for assembling knowledge gained from existing studies. Using Framework components as organizers, research syntheses and reviews can summarize what has been learned about the extent to which the education system has changed in response to nationally developed standards, particularly in terms of classroom practice and student learning.

OCR for page 79
Investigating the Influence of Standards: A Framework for Research in Mathematics, Science, and Technology Education Identifying gaps. Gaps in current research can be identified by considering questions that could be (but have not been) asked about elements and relationships within the Framework. By highlighting where research attention has been most and least focused, the Framework can help researchers—and sponsors of research—target issues and areas of concern that merit more study. As noted earlier, relatively few studies have investigated the relationships among professional development, teacher knowledge, instructional practice, and student achievement, either generally or with regard to national standards (Kennedy, 1998; Wilson and Berne, 1999). This paucity of studies regarding potentially important avenues for standards to reach classroom practice and student learning may signal a need to fill the gap. Even, in areas where substantial numbers of studies have been completed, the Framework can highlight additional questions not yet extensively posed or answered. For example, a nagging concern within the broader standards-based reform movement regarding the equitable distribution of standards-based practice and equitable accountability systems (McKeon, Dianda, and McLaren, 2001) suggests an aspect of the story about the influence of national standards that may deserve greater attention. The relevant question in the Framework—Who is affected and how?—encourages researchers to explore possible differential effects of standards within diverse student populations and settings, while taking into account the varied capacities of teachers, schools, and districts to engage in standards-based practice. Other examples can be readily envisioned. One important advance in cumulative understanding of nationally developed standards in mathematics, science, and technology education would be to assemble and map current knowledge using the Framework so that gaps and opportunities for further study emerge.

OCR for page 79
Investigating the Influence of Standards: A Framework for Research in Mathematics, Science, and Technology Education ASPIRATIONS FOR FRAMEWORK-DRIVEN RESEARCH ON NATIONALLY DEVELOPED STANDARDS The Framework is offered in the hope that it will be useful to producers, consumers, and sponsors of research regarding central questions about the influence of nationally developed standards on mathematics, science, and technology education. Applications of the Framework described earlier will help to inform opinions and debate about those standards. Three major aspirations of the Committee regarding use of the Framework are highlighted below. The Framework should be regarded as an evolving conceptual picture, rather than a definitive final statement. In that spirit, the Framework should continue to evolve, informed by accumulating knowledge about standards-based reforms. It is essential that researchers build their understanding of the influence of nationally developed standards in terms of some overarching model of the education system (or subsystem) within which standards play out. The schematic of the Framework presented in this document can be regarded as one sketch of such a system, including the dynamics of influence contained within it. The Framework should stimulate different forms of inquiry into influences of nationally developed standards. Given the complex and interactive nature of the territory within which standards have been enacted, a mosaic of evidence from many different types of studies is more likely to build overall understanding of the influence of standards than the results of a few purportedly comprehensive studies. The Framework should help guard against the superficiality that often permeates debate about high-visibility national policies by stimulating a critical view of claims regarding either the success or the failure of the standards. Strong conclusions about effects or implications of nationally developed standards presume an understanding of the entire education system (encompassed by the Framework) and

OCR for page 79
Investigating the Influence of Standards: A Framework for Research in Mathematics, Science, and Technology Education presentation of a chain of evidence that connects the emergence of particular education practices, policies, or learning outcomes to the influence of standards. All U.S. youth deserve access to the best possible education in mathematics, science, and technology. In pursuit of that goal, the education community should complete a comprehensive, critical appraisal of the power and limits of nationally developed standards. That appraisal is still far from being realized. Public conversations about the worth and impact of standards in mathematics, science, and technology—or about standards-based reforms in general—will continue. The Framework offered here is intended to help the education research community contribute to that debate with reasoned voices based on evidence and sound inference.

OCR for page 79
Investigating the Influence of Standards: A Framework for Research in Mathematics, Science, and Technology Education This page in the original is blank.