Ensuring Quality and Impact: A Program to Advance Both Science and Practice
Much has been made in recent years of the quality of education research, with particular emphasis on the methodological weakness that is said to characterize the field. The discussion is a complex one, for issues of quality are regularly confounded with issues of complexity. Education needs high-quality research if the results are to be reliable for purposes of improving practice. Research also needs to be designed to be sensitive to the contexts of teaching and learning if it is to be of use to teachers.
In this chapter we articulate the features of the SERP organization and the proposed agenda that together support research quality and impact.
At the request of the U.S. Department of Education, a committee was recently convened at the National Research Council to examine and clarify the nature of scientific inquiry in education. Its report, Scientific Research in Education (National Research Council, 2002b), describes features of what that committee considered high-quality research. The committee’s six principles can serve as a point of departure for our discussion (National Research Council, 2002b):
Pose significant questions that can be investigated empirically;
Link research to relevant theory;
Use methods that permit direct investigation of the question;
Provide a coherent and explicit chain of reasoning;
Replicate and generalize across studies; and
Disclose research to encourage professional scrutiny and critique.
The principles are straightforward and are sound elements of scientific research in any discipline. However the challenge of conducting research that is responsive to these principles in education is somewhat daunting. The SERP proposal and research agenda provide the specifics with which to put flesh on these skeletal principles of quality research and consider how quality in education research might be effectively supported.
POSE SIGNIFICANT QUESTIONS THAT CAN BE INVESTIGATED EMPIRICALLY
The significance of the questions we pose is heightened by the consistent focus in the agenda on the questions that define educational practice. A systematic approach to reviewing what is known and unknown regarding those questions in each domain of study draws attention to research questions that are critical from the perspective of improving teaching and learning.
But while significant questions in education research are many, the ability to investigate those questions empirically is more constrained. Understanding effective instructional practice, for example, requires access to classrooms in which that practice can be observed. It requires careful data collection regarding the many features relevant to instruction, as well as the many features of the students themselves that may affect instructional outcomes.
Challenges abound. Putting data collection systems in place requires a substantial up-front investment. That investment may make little sense when a school or school district is participating in an individual study. Only in rare circumstances will the benefit to the researcher and that to the school warrant the cost. However, if a SERP organization and research program is in place, schools and school districts that function as field sites
would have an ongoing involvement in the research enterprise. The data collected could be used in many different studies, making the initial investment far more productive.
Moreover, the value of the data collection rises substantially when students are followed over many years. Longitudinal data sets are the workhorses of empirical research in the social sciences. Investigation of the long-term impacts that are central to effective policy making can be explored only when data are collected longitudinally. But because longitudinal data collection requires a long-term effort in an environment in which many of the individuals (researchers, principles, teachers) have short-term horizons, the presence of an institutional infrastructure that can ensure continuity is critical.
Still, schools may be reluctant to participate in data collection without the promise of both protection (privacy of information) and significant payoff. Access to the quality of data needed for direct investigation of instructional questions becomes more likely if an organization like SERP develops solid credentials at instituting and honoring standards for information privacy—even as it attends to making data maximally useful for researchers within those privacy constraints.
Payoff for schools will be required in the form of negotiated products of research and development that satisfy the needs of schools as well as those of the researchers. It is certainly likely that many schools would have no interest in a research partnership. But the school administrators who do look to the research community for help with puzzling questions of practice have little guidance about where to find that help. SERP can serve as a magnet for schools and school districts who are looking for partnerships with researchers. When the desire for partnership is mutual, the demands of careful data collection may be more easily met.
A major contributor to both quality and impact of the SERP program, then, is the ability of the organization to attract schools as field sites and to nurture long-term relationships. Cultivating those relationships will be critical in achieving the trust that will be required for access to classrooms and to student-level data that will make empirical investigation of the most important problems of practice possible.
LINK RESEARCH TO RELEVANT THEORY
The National Research Council report (2002b) argues, “it is the long-term goal of much of science to generate theories that can offer stable explanations for phenomena that generalize beyond the particular.” In the practice of education, much changes from one case to the next: students, teacher, principal, curriculum, textbooks, school environment, and home environment can all vary simultaneously. Progress in advancing theory can be challenging because finding stable relationships can be difficult when so little is held constant. Yet the ability to generalize beyond the particular is precisely what is needed not only for the quality of the research, but for supporting teaching as a profession.
Several features of the proposed agenda support a close link between research and theory. First, in the three domains of focus, we describe knowledge bases that provide a theoretical and empirical foundation relevant to each of the framing questions. In each domain the proposed research and development builds on that foundation, and on subject specific theory and knowledge. In some cases (e.g., the acquisition of early number knowledge) theory is well developed and empirically supported, and we suggest that curricula built on the theoretical underpinnings are ripe for further work. In other areas (e.g., reading comprehension), the theory is in dispute. The research proposed (e.g., observations of effective practice) can support theory building. In all cases, however, the starting point for the proposed work is a review of the current state of theory and knowledge, ensuring that the link between the research and relevant theory is strong.
Second, development and evaluation of instructional interventions can be, and often are, carried out with the sole purpose of discovering “what works.” Advancing theory, however, requires knowledge of why something works, for whom, and under what circumstances. The agenda we propose systematically pursues that theory-building knowledge.
Finally, theory development will be supported by the rich body of research that will be conducted in networks that are closely linked. The work conducted separately for science, mathematics, and reading can provide a rich knowledge base from
which new understandings and generalizations can emerge, moving the theoretical understanding of teaching and learning forward. The discussion of issues in early reading, for example, in many respects mirrors that in early mathematics. Both subjects pinpoint the need to develop fluency in a skill and yet not hold more challenging thinking about problems or text hostage to skill development. Both struggle with the problem of excessive focus on procedure and insufficient attention to meaning-making. The challenge for teachers to engage students in dialogue that takes their thinking the next step is a close parallel in these two, and other, subject areas. Patterns across subjects can support new theoretical understandings. The responsibility of the leadership of the networks for monitoring and synthesizing findings across the many strands of work will allow SERP to advance theory development more intentionally than is now the case.
USE METHODS THAT PERMIT DIRECT INVESTIGATION OF THE QUESTION
The Committee on Scientific Principles for Education Research argues, quite sensibly, that there is no best method of research in education or in any other field. Rather, the method must be appropriately matched to the research question asked. In the agenda outlined here, the proposed methods vary tremendously in accordance with the question. In some cases in which the knowledge base is weak, we suggest observation of practice to support hypothesis generation. In other areas in which curricula have been developed and subjected to preliminary evaluation, we suggest randomized trials in order to determine effectiveness of a curriculum.
Perhaps the more important point made by the report on scientific research in education is that rarely can a single method illuminate all the questions and issues in a line of inquiry. Indeed, our agenda proposes that a range of questions and companion methods are required as components of a program of research and development in order for research and practice to be powerfully linked. While qualitative research intended, for example, to generate hypotheses regarding effective teaching practice may be desirable when theory and evidence are weak,
the product of that research can be broadly useful to practice only when the hypotheses are rigorously tested. Similarly, a randomized trial to determine the learning outcomes of alternative curricula can provide high-quality data, but fully exploiting the findings and generating further improvements in the curriculum and in teaching practice requires observation of why and how teaching and learning change with the curriculum. Quality and impact adhere in the program that combines methods to answer the different kinds of questions critical to improving practice.
PROVIDE A COHERENT AND EXPLICIT CHAIN OF REASONING
The explanations and conclusions drawn from research, the NRC committee points out, “requires a logical chain of reasoning from evidence to theory and back again that is coherent, shareable, and persuasive to the skeptical reader” (p. 4). The strength of that chain of reasoning is critical in education research, because beliefs are often held passionately, and any case made for improving the teaching of mathematics, reading, or science will encounter a great many skeptics. Ultimately, however, many of the contested claims can be tested empirically. But because skepticism runs high, high-quality evidence will be required for impact.
Central to the quality of the evidence in the research we propose here is an investment in the development of credible outcome measures and control variables. If we want students to have a deeper understanding of science, a more flexible approach to mathematical problem solving, or an appreciation of nuance and complexity in text, we must have adequate measures of those outcomes to know if they have improved. Similarly, if we want to understand the conditions under which an instructional approach improves outcomes, we need to have carefully defined which “conditions” matter. These investments are critical to the quality of the research, and yet they are often shortchanged in poorly funded research studies.1
See, for example, the discussion of evaluations of curricula funded by the National Science Foundation in Chapter 3.
The quality of the evidence also improves substantially when data are collected longitudinally. As noted earlier, longitudinal data are required for measuring long-term results. But they can also be useful in suggesting patterns that might otherwise go unnoticed, calling attention, for example, to years or classrooms that are outliers and so hold information that may be valuable. And since much of what can be achieved in later years depends on what has been taught in earlier years, the ability to look across years is fundamental to designing instruction that effectively builds over time. The systematic, longitudinal data collection effort that SERP proposes can bolster the quality of the evidence from its research.
But even findings from good-quality data require interpretation and inference. The nature of the SERP networks as collaborations among researchers and practitioners will provide a fertile environment for interpretation of findings that draws on different perspectives, knowledge bases, and experiences.
REPLICATE AND GENERALIZE ACROSS STUDIES
The Committee on Scientific Principles for Education Research argued that ultimately, scientific knowledge advances when findings are reproduced in a range of times and places and when findings are integrated and synthesized (National Research Council, 2002b). While replication is required for scientific quality, it is at the very heart of impact on practice. As we have argued throughout this report, to know that an instructional program works well for students in a middle-class, suburban school provides little useful knowledge to teachers whose limited-English-proficient students live in a disadvantaged urban environment. A hallmark of the agenda we propose here is that it consistently builds replication of research results into each stage of the work. Moreover, we propose that the research be conducted with the range of students and under the range of circumstances to which the results would apply.
Both replication and generalization are facilitated by the SERP organization. An important function of the networks is to develop common research protocols to facilitate replication and strengthen interpretability across sites and studies. And a cen
tral function of the network leadership as described in the SERP report, Strategic Education Research Partnership, includes regular synthesizing of research results and stock-taking to alter or chart a course.
DISCLOSE RESEARCH TO ENCOURAGE PROFESSIONAL SCRUTINY AND CRITIQUE
Few would argue that this widely embraced norm of scientific research is critical to quality. And indeed, the SERP report proposes a scientific advisory board with responsibilities for quality monitoring and professional scrutiny of the research program. In addition to a formal review process, the SERP design includes a proposed web site that will provide an opportunity for feedback from a wide audience—skeptics included.
The scrutiny that is required for impact in the field of education, however, goes beyond the disclosure of research results that quality mandates. Much of the education research produced, both high-quality and otherwise, withers from neglect. Impact requires actively seeking out high-quality research that is important for educational practice and building on it—as we propose, for example, with respect to the Number Worlds research or the reciprocal teaching research.
But impact also requires the design of research studies that can take the knowledge from practice and incorporate it into testable propositions that can be shared publicly. Much of what teachers learn from repeated observation of student learning and response to instruction is never formally articulated, tested, or shared with others in their professional community (Hiebert et al., 2002). Research designed to learn from practice, like that proposed for reading comprehension, can formalize the knowledge of teachers, subject it to testing, and make those results available to be shared and scrutinized publicly.
Ultimately, both quality and impact are best supported not in single research studies, but in a coherent program of research that ensures replication, accumulation, and follow-through; a program with strong theoretical underpinnings. The quality of the program is elevated substantially when careful attention is paid to research protocols, the design and testing of outcome
and control variables, and careful data collection. But at the heart of quality and impact on education practice is an investment in both problems that matter to practice and in the development of communities of researchers and practitioners to carry the work forward. It is this that the SERP design calls for, and this that our proposed agenda will require.
As we consider what we know about high-quality educational practice in reading, mathematics, and science, the gaps in that knowledge base are striking. Widespread concern about the performance of the education system has led states and school systems to develop content and performance standards, and it has led to efforts at both the federal and state levels to hold schools accountable for results. But setting goals is unlikely to be helpful in the absence of a defined path to achieving them.
We have seen repeatedly in the chapters on reading, mathematics, and science that much path-defining work remains to be done. In all of the subject areas, for example, aspects of instruction that are fairly easily defined claim a disproportionate share of instructional time and dominate assessments. These tend to be procedural, and they can therefore be straightforwardly described. Teaching phonics, reproducing factual detail of a story, and teaching math facts or science facts are all cases in point. These are critical dimensions of instruction, but if they are the whole of instruction, then understanding will be shallow (National Research Council, 2000).
This instructional emphasis on the procedural has persisted over the history of public schooling, although rising standards suggest the ultimate goal of instruction is to foster deeper understanding and skill: to see nuance in a text’s meaning, to appreciate the difference between a finding and an opinion, to understand the nature of mathematical or scientific problems and how they are identified. But building understanding is, without question, challenging. It will require more interaction between teacher and students, as programs like questioning the author, reciprocal teaching, and modeling methods suggest. It will require making student thinking apparent, and working
with student ideas to take them to the next level of sophistication, as Everyday Mathematics or ThinkerTools suggest. Some teachers have the intuition, inclination, knowledge, and experience to provide their students with these types of instructional experiences. But for the many teachers who do not, the supports that would allow them to develop these abilities are rarely available. Yet success at reaching high academic standards depends on doing just that.
On some of the most basic instructional questions—for example, how can the components of reading comprehension be developed and assessed across the school years?—we have hardly begun the research necessary to support professional practice. Of equal concern, however, is that in those areas in which the knowledge base for improving practice is strong—like early mathematics, early reading, and physics—it has had little impact on practice. The examples we highlight of high-quality research and development have, for the most part, remained on the sidelines of educational practice. Perhaps the greatest squandered resource, however, is the excellent teaching practice that produces demonstrable effects on student achievement and yet remains an anomaly, its lessons for teaching never articulated, studied, or shared.
The work we propose here has the potential to substantially improve the knowledge base to support teaching and learning by pursuing answers to questions at the core of teaching practice. It calls for the linking of research and development—of instructional programs, assessment tools, teacher education programs and materials—that is now rare. In the course of doing so, we propose to draw on the largely untapped resources of effective teaching practice and high-quality research. The downstream proposals offer hope of an impact in the relatively near term, for they build on solid work with clear implications for practice. And the upstream proposals would begin to provide a foundation for professional practice in the future where none now exists.
The panel generated this agenda assuming the infrastructure and operation of a SERP organization to support the linked, multifaceted work that is envisioned. Realizing the potential of the proposed research and development for improving teaching and learning will require the organizational infrastructure that can support research on, with, and for practice.