Advancing Scientific Research in Education (2005)

This appendix is a summary of Understanding and Promoting Knowledge Accumulation in Education: Tools and Strategies for Education Research, the second workshop in the series conducted by the Committee on Research in Education. The workshop featured a discussion of conceptual ideas about reflections, definitions, and challenges associated with knowledge accumulation, generalizability, and replication in education research. It also included a discussion of tools to promote an accumulated knowledge base derived from education research, many of which we highlight in our recommendations.

Rather than issue a separate report summarizing the workshop, the committee decided to develop a summary of key points that provide context for, and help illuminate, the conclusions and recommendations in this report. This decision was based on the recognition that the ideas discussed at this workshop—while by no means an exhaustive review of the philosophy of science or nature of education research—provide an important intellectual foundation for all of the issues and strategies that were discussed during the workshop as well as throughout the workshop series (see Appendix A for a compilation of agendas).

The workshop had two objectives. The first was to provide a context for understanding the concept of knowledge accumulation, both generally and with respect to education research. No one study or evaluation—no matter how rigorous—can single-handedly chart the path of progress for education policy and practice, nor can one study adequately sum up the

state of understanding in a field or subfield. The challenge for research fields—including education research—is to find ways of facilitating the growth and refinement of knowledge about education phenomena over time. The second objective was to focus on concrete ways this progress of scientific knowledge in education research could be facilitated.

Thus, the workshop had two main parts: the first part featured a series of presentations and discussions designed to clarify phrases and terms like knowledge accumulation, generalizability, and replication in education. The second part featured discussions of three sets of tools for developing a more coherent body of knowledge from education research: developing common measures, sharing data, and taking stock of what is known. The appendix follows this structure, summarizing key points from each workshop part. Because much of what was discussed in the second part of the workshop—specific tools and strategies for promoting knowledge accumulation—is featured in boxes or in the conclusions and recommendations in the main body of the report, that section is significantly shorter than the first.

Kenji Hakuta, in transition at the time of the event between Stanford University and the University of California, Merced, began the day with a presentation that considered key terms and ideas associated with knowledge accumulation and then traced the example of research in bilingual education to illustrate them. Following this overview, Kenneth Howe, of the University of Colorado focused on the interaction between the progression of scientific understanding and the methodological frameworks that researchers have utilized to study education. Reflecting on these two presentations, representatives from three disciplines and fields outside of education offered their perspectives on how the nature of knowledge accumulation in their fields is similar to and different from that in education: Jay Labov, of the National Research Council (NRC), on the biological sciences; David McQueen, of the Centers for Disease Control and Prevention, on epidemiology; and Sidney Winter, of the University of Pennsylvania, on business. Finally, two presentations that traced lines of inquiry in education research illustrated these core ideas with concrete examples: David Cohen, of the University of Michigan; and Helen (Sunny) Ladd, of Duke University, on the role of resources in school and student achievement; and Barbara Rogoff, of the University of California at Santa Cruz

Foundation Professor of Psychology, on the relationship between culture and learning.

Overall, the discussion made clear that knowledge accumulation, generalizability, and replication are terms that seem to have a straightforward meaning on the surface but are less clear when examined more closely. In general, presenters seemed to agree that knowledge accumulation is a way to think about the progress of scientific research—how investigators make sense of, and build on, the studies that shed light on particular phenomena. As Cohen clarified, however, it is more than just the “heaping up” of findings. It involves extending previous findings, including the elaboration and revision of accepted theories, with an eye always toward the growth of systematic understanding. In some cases, knowledge accumulation can involve wholesale replacement of a paradigm, as pointed out by McQueen.

The progression of scientific knowledge is described in Scientific Research in Education in these terms: “the path to scientific understanding … is choppy, pushing the boundaries of what is known by moving forward in fits and starts as methods, theories, and empirical findings evolve” (National Research Council, 2002). This first workshop session elaborated this core idea of methods, theories, and empirical findings interacting and growing in nonlinear ways.

Several presenters described the dynamic relationship between theoretical or conceptual ideas in a field and the empirical studies that test their adequacy in modeling phenomena. Scientific understanding progresses when the field attends to both theoretical development and empirical testing and analysis; one without the other is not sufficient. Theory without empirical backing lacks real-world testing of its explanatory power. And data without theory leads to “dust-bowl” empiricism—that is, data that lack meaning or relevance to describing or modeling the phenomena of interest (teaching, learning, and schooling in education).

Rogoff provided the clearest illustration of how related lines of inquiry developed in cross-cultural psychology and sociolinguistics by researchers moving back and forth between periods of empirical investigation and theory building as one informed the other over time.

As she described it, in the 1960s and 1970s, there was a great deal of empirical investigation—about 100 studies—in the area of cross-cultural

psychology (see Munroe, Munroe, and Whiting, 1981). This work applied cognitive tests in international settings and led researchers to question previous assumptions about the generalizability of developmental and learning theories that were dominant at the time. Through the multitude of studies conducted in this arena, it became clear that context mattered for evaluating learning.

Following that era, in which a great deal of empirical work was carried out, a period of theory-building ensued. During the late 1970s to the early 1990s, the field of cultural psychology developed a theory that allowed researchers to take context into account, rather than assuming that what was being examined was a window on general functioning. An influential event in this regard was the translation of Lev Vygotsky’s work into English in 1978. It demonstrated how one could use both context and individual aspects of development in the same view. In short, his theory argued that individual development is a function of social and cultural involvements. Related cultural research with underserved populations in the United States also demonstrated the importance of considering the familiarity of the context in interpreting performance on cognitive tests and other contexts. Tests themselves are being investigated as a context with which some children are unfamiliar.

In her presentation on the role of resources in school and student achievement, Ladd demonstrated how different lines of inquiry in one broad area can emanate from different theoretical orientations or questions. She described three types of questions that have been prominent in this area of research:

1. What is the impact of school resources on educational outcomes? (The “effects” question.)

Ladd identified this question as the one addressed in the so-called education production function literature. In that literature, educational outcomes (defined as student achievement, educational attainment, or subsequent wage earnings) are typically modeled as a function of school inputs and family background characteristics (see, for example, the meta analyses by Hanushek [1986, 1997]). Such models have become increasingly sophisticated over time as a result of richer data sets and enhanced methodology. Notably missing from these models are teacher practices and institutional context. Hence, to the extent that particular practices are correlated with a specific school input, such as class size, across observations within a sample, the estimated impact of class size on student achievement reflects

not only class size but also any teacher practices correlated with variations in class size.

2. What resources would be needed to achieve a particular desired educational outcome? (The “adequacy” question.)

Since the early 1990s, several economists have focused on this line of work, mainly using expenditure data at the district level. Emerging from this approach is the conclusion that some students, for example, those from low-income families, are more challenging to educate, and hence, require more resources to achieve a given educational outcome, than do students from more affluent families. (See, e.g., Duncombe and Yinger, in press; Yinger, 2004; Reschovsky, 1994; Reschovsky and Imazeki, 2003.) Research being conducted by Cohen, Raudenbush, and Ball (2002) also falls into this category of adequacy research. They start out with the notion of instructional goals, and then ask what instructional strategies are most productive in reaching those goals, how much these strategies cost, and, hence, what resources would be required.

3. What can be done to make a given level of resources more productive toward the goal of educational outcomes? (The “productivity” question.)

This line of research examines what can be done to make a given level of resources more productive in achieving educational outcome goals. Much of the effective schools literature of the 1970s and 1980s falls under this category (see Stedman, 1985, for a summary). In this line of work, researchers studied schools identified as being effective in raising achievement for disadvantaged students to determine what practices were common across effective schools. Ladd also identified other important work that addresses the productivity question, including Monk’s (1992) discussion about the importance of investigating classroom practices and work summarized in Chapter 5 of the NRC report Making Money Matter (1999).

In addition to empirical observation and theoretical notions, a third dimension of the progression of research knowledge is the methods used to collect, analyze, and relate the data to the conceptual framework. Indeed, the main thesis put forth at the workshop by Howe was that questions concerning knowledge accumulation are difficult to disentangle from questions concerning broader methodological frameworks. He specifically argued that the experimental paradigm (that is, relying on random assignment designs) may encourage the accumulation of knowledge about easy-

to-manipulate, simplistic instructional approaches. He also suggested that since experimental studies in education typically employ random assignment, rather than random selection (and thus are typically limited to those who volunteer), the generalizability of the findings is limited. Finally, he argued that experiments do not provide knowledge about the precise mechanisms of causal phenomena, which is necessary for deeper knowledge-building.

The exchange that followed the presentations extended the discussion of methodology by focusing on the need to use multiple methodologies in appropriate ways to promote broad understanding of the complexities of teaching, learning, and schooling over time. Using research on class size reduction as an example, Harris Cooper, who at the time of the workshop was in transition between the University of Missouri, Columbia, and Duke University, pointed out that examining the findings from small, qualitative studies opened up the “black box” and revealed that not only were teachers spending less time on classroom management and more time on instruction in smaller classes, but also that they were conducting more enrichment activities. This finding alerted the research community to a host of potential new effects that traditional quantitative research and the main findings about the effects of class size reduction on achievement would not—could not—have illuminated.

Later in the day, other speakers picked up the same theme. Hugh (Bud) Mehan, of the University of California, San Diego, provided an example of how the skillful combination of quantitative and qualitative methodologies is not only powerful but may also be necessary in education research. In describing his work in scaling up school improvement efforts beginning in a few schools in San Diego, extending to the state of California, and then growing yet again to multiple states, Mehan argued that the methodological approaches required to conduct the research on this program and its growth were both quantitative and qualitative. He suggested that although in individual investigations, quantitative and qualitative research are typically carried out independently, in carrying out large-scale studies in which programs are introduced and studied in increasing numbers of sites, this separation can no longer be sustained.

Education research is often criticized for being endlessly contested, both among researchers and in the broader public community. Several par-

ticipants mentioned the role of the public in shaping research, in both education and other fields, underscoring the point that public criticism is not unique to education and that this interest has both positive and negative effects.

Cohen argued most directly that education research both benefits and suffers as a field from high public interest. This level of involvement from the public can be trying for researchers as they try to balance their approaches to research on education issues with public concern that can often take on a highly political character. Public interest also lends the potential for greater success for the research field. The National Institutes of Health (NIH) have benefited greatly from the public demand for high-quality medical research in terms of rising appropriations. However, the high level of public interest in education is less productive for increasing the use of research findings, because the public places low value on research in education. While the public is highly interested in questions of how best to educate children, they rarely look to research to provide answers of value.

And there are opportunity costs associated with letting policy and political issues drive a research agenda. Hakuta’s depiction of the bilingual education research following publicly framed dichotomies of program options—rather than research-driven, theoretically derived models based on practice—shows that this orientation led to a great deal of work trying to explain only a small fraction of the variation in how English-language learners best learn the language. Only recently, he argued, have researchers turned their attention to studies that focus on questions most relevant to understanding this key issue.

McQueen offered another example from research on the relationship between smoking and lung cancer. Ethical considerations obviously preclude randomly assigning people to smoke, so research findings were criticized as not being valid. Couple this fact with the involvement of interested parties (cigarette manufacturers, the antismoking lobby), McQueen posited, and research findings became even more contested. Winter offered another example from the realm of business, commenting that corporate governance is a “violently” contested area right now, with implications for research.

Finally, Labov elaborated that there are both internal and external controversies that play into claims regarding the contested nature of research. For example, in biology, evolution is an area that is hotly debated. Within the field, most biologists accept the idea of evolution as a key organizing principle of the discipline, but there is debate surrounding the mechanisms

by which evolution occurs—such as Charles Darwin’s idea of incremental change and Steven Jay Gould’s idea of punctuated equilibrium. Outside the field, this debate is interpreted as a more serious controversy, and some outsiders suggest this as evidence that the theory of evolution itself is in question or is challenged by most biologists.

An important contrast emerged with respect to the nature of theoretical ideas in fields of practice (like education and business) versus those in traditional scientific disciplines (like cell biology). McQueen articulated most explicitly that, in such applied fields as medicine and public health, theoretical ideas are different from those found in such disciplines as chemistry and biology. Medicine and public health are fields of action, he argued; as such, they are characterized by carrying out interventions, so practitioners draw on theories from multiple disciplines. He pointed out that when one works in a field rather than a discipline, it is difficult to set up a theoretical base whereby hypotheses and causal relationships are tested, as demanded by a more strict scientific model.

In his presentation, Hakuta provided a synopsis of the history of research on teaching students with limited English proficiency (LEP) that illustrated how education as a (contested) field of action influenced the creation of theoretical frameworks that shaped the associated field of research. In 1974, the Supreme Court decided in Lau vs. Nichols that addressing the needs of children who arrive in school not speaking English is a district and state responsibility. No particular approach was prescribed in the decision, so two general approaches for educating LEP students each developed a following: English-language immersion and a bilingual approach. LEP education became (and continues to be) a controversial issue; subsequently, a great deal of resources was invested in research to investigate the question of which was the better method. To date, the research shows a slight advantage in English-reading comprehension for LEP students who have been in transitional bilingual programs (see, e.g., Willig, 1985; Greene, 1998: National Research Council, 1997). However, comparatively very little research has focused on the gap in reading scores between LEP and non-LEP students and its growth as students progress through school. Specifically, in grade one, the gap in age equivalent scores of reading achievement between LEP and non-LEP students is equivalent to approximately one year in age. By fifth grade, that gap has increased to

two years. Nothing in the research to date can explain that gap. Public pressure in the applied field of education, Hakuta argued, has led to an overemphasis on a single research question, inhibiting the growth of knowledge on other important questions.

In his discussion of research on the effects of resources on school and student achievement, Cohen pointed to another area in which a great deal of time and effort has been devoted to researching a phenomenon in the applied field of education research that accounts for a small percentage of the differences in student achievement. In describing research that explores the relationships among resources, instruction, and learning outcomes, Cohen began by summarizing the seminal On Equality of Educational Opportunity, known as the Coleman report, of 1966. Coleman investigated differences in performance among schools and concluded that resources made little or no difference. Since that report was released, there has been a great deal of additional investigation into this topic.

However, as Cohen pointed out, 80 percent of the differences in student achievement lie within schools, not from school to school, so there is a great deal of variation that is not being examined in this line of research. While research is ongoing in examining differences in both the 80 percent and the 20 percent, the public debate framed the question and the theoretical conceptions early, persisting for decades.

Workshop speakers also argued that in a field like education, which is characterized by complex human interactions and organizational, cultural, and political influences, attending to context in the research process is critical. Thus, it is unreasonable to expect simple generalizations that are broadly applicable. That said, however, the field advances when investigators strive to make generalizations that build in descriptions of how context affects results. Furthermore, variation deriving from contextual factors is helping to reveal relationships: without variation, there is no way to learn about effects and patterns among variables and concepts. This context dependence is a theme that continued throughout the day, but in this session it became clear that it is not a characteristic that is unique to research in education.

In business, as Winter described, many situations depend on the interactions between employees, investors, and customers. These interactions can be quite complex and vary from one grouping to the next. As such,

those who conduct research on business practices encounter many of the same obstacles in trying to understand the extent to which findings are applicable to multiple settings that education researchers do. In other words, the strategy that business researchers found was employed with resounding success in Site A may not be at all effective in Site B.

The importance of context dependence in the conduct of research is further demonstrated by the history of physiological experimentation at NIH. As Labov pointed out, NIH came under a great deal of criticism about 25 years ago because clinical trials were being conducted primarily on white male subjects. However, such results often do not generalize from one gender to the other. As a consequence, many of the treatments for diseases that affect both men and women, such as heart disease, were not as effective for women as they were for men, but without explicitly designing research to estimate differential effects on men and women, physicians would not know to prescribe different regimens.

In one sense, participants characterized the fact that results vary across contexts as a challenge to efforts that aim to make summary statements applicable to multiple settings, times, and populations. Mehan, for example, quipped that the one core principle of ethnographic work is “it depends,” referring to this relationship between findings and contexts. However, explaining variation is the core purpose of research, so the variation that results from this context dependence also enables attempts to model differences in outcomes. Rogoff, echoed by a few other workshop participants, argued that the field of education ought to focus its efforts on elaborating theories and crafting “universal laws that account for this context dependence and thus reflect the complexity of educational phenomena.”

Extending the discussion of education and business as fields rather than disciplines, two dimensions of the relationship between practice and research were elaborated. First, David Klahr, of Carnegie Mellon University, when questioning the presenters, offered the idea that education research might be more comparable to an engineering discipline than a science. He continued by arguing that knowledge accumulates in engineering through practice. For example, there is a great deal of variability from one space shuttle to another, even though they are all in the same series. As one shuttle would be completed, he continued, engineers would apply what

was learned in the construction of that shuttle to the design and construction of the next.

Second, a conversation about the role of cases in education and business research further elaborated the close link between practice and research in these fields. Cohen’s description of a particular line of work in which he has been involved in the resources and student achievement area illustrated this idea in education. Along with his colleagues Steve Raudenbush and Deborah Ball, Cohen has spent considerable time examining the relationship between resources and student achievement. They have found that much of the research on school effects assumes a model in which there are desired outcomes that are directly caused by the input of resources. However, he argued, this is not plausible, because resources become active only when they are used. Therefore, in order to validly measure the effects of resources, the conditions in which they are used must be taken into account, and this requires attention to practice.

Winter also offered examples of how practice relates to research in business. First, he said that for students engaged in dissertation work, they are fortunate if they can carry out two or three years of work in an area without a merger or a regulatory incident interfering with their research site. He went on to say that the use of cases in business schools is to create effective managers that “more or less give people a vision of what it means to be pushing the levers that are available for controlling a management situation.”

Continuing to explore the idea of how theoretical ideas and research priorities can and should be driven by the practices of the field (education, business, medicine, etc.) and their surrounding political contexts, Lauress Wise pointed out that most NRC studies that integrate and summarize research on a topic across disciplines and fields do so at the request of public officials and are therefore at least partially shaped by the political and policy questions of the day.

Two talks on scaling up brought into sharp relief how research and practice can feed into one another in ways that benefit both. Robert Slavin, of Johns Hopkins University and chairman of the Success for All Foundation, illustrated the potential for mutually reinforcing relationships between educational practice and research and evaluation by detailing the history of the development of the Success for All program. According to Slavin, by the 1970s a body of evidence about the effectiveness of cooperative learning pointed to the value of such student team approaches (see Slavin, 1995).

At the same time, the idea was gaining a foothold among practitioners, and so their use became commonplace. However, the fundamental elements that research suggested needed to be in place for them to promote learning—groups were structured, purposes were clear and shared by all students, and each member had a specific task or role to play—were typically not in place in practice. Slavin told the group that the research findings and the disconnect between them and what was going on in schools was the “intellectual background” for the development of Success for All, which began in one school in Baltimore and is now operating in about 1,500 schools across the country. As the program grew, Slavin and his team have engaged in a development process of implementing programs, studying how they are used and what their influences are, and then feeding that knowledge back into program improvement but also, importantly, into the larger knowledge base on cooperative learning, comprehensive school reform, and program evaluation.

Mehan, too, touched on this idea by offering a lesson from his experience in scaling up school reform efforts in California and the fact that the research that documented and analyzed the expansion was an iterative process. The iterations were necessary, he argued, to strike the right balance between a standard set of questions and data collection protocols and the need to recognize and articulate what he termed “emergent phenomena.” Because program elements interact with local circumstances in different ways, Mehan argued that the kinds of issues and data that are relevant to understanding the implementation and effectiveness of the program will vary to some degree across sites.

A final theme raised in this initial workshop session was the crucial role of the community of investigators, including funding agencies, to support efforts to integrate and build on findings from related work. Hakuta said it plainly: “It is not just the methods that enable knowledge to accumulate,” but also fundamental are “the critiques and the questioning that happen in science.”

While such critique and debate in a field is healthy and promotes the growth of knowledge, workshop speakers suggested that it is important to keep the debate at a civil level. One audience member noted that a tone of derisiveness and lack of respect can creep into the discourse, especially across disciplines, which is to the detriment of the kind of building community

that can facilitate knowledge accumulation. Winter reiterated this point, suggesting that the kind of standards that would be most useful to researchers are standards for “intelligent debate.”

One issue that is closely related to community is the lack of common quality standards in education research. Hakuta suggested that standards could be helpful, but he cautioned that standards generated within the community are much more likely to be accepted by researchers than standards that are imposed from the top down. Across workshop speakers, opinions on the topic varied, with some suggesting that standards would serve as an impediment to research, and others suggesting that standards would improve research quality. Rogoff cautioned that standardization could be premature; it could short-circuit the empirical work that needs to be carried out in order to learn more about the regularities across communities and across contexts that would enable the understanding of how culture plays a role in human development. To do this, she argued, lines of research that build on prior studies are needed, because from each study, questions, theories, and ways of doing research are refined.

Other speakers addressed the idea of human capacity in research and its connections to knowledge accumulation. Mehan, for example, discussed the need for thoroughly trained research staff—preferably those who have been working with the team on the issues for some time—to collect data according to protocols and to be attuned to what he called relevant “emergent phenomena” in scaling up and studying the implementation and effects of the Achievement Via Individual Determination, or AVID, program. In a different vein, Harris Cooper, in describing the evolution of meta-analytic methods for summarizing research on effectiveness about a particular intervention, argued that “vote counting”—a way of summarizing literatures commonly used by researchers—is a demonstrably poor method for arriving at valid conclusions about what the research says collectively (in that it consistently leads to an underestimation of the program effect), suggesting that researchers with meta-analytic skills are needed for these tasks.

The discussion of human capacity extended beyond individual investigators. Daniel Berch, of the National Institute of Child Health and Human Development, offered a description of the important role of federal research agency personnel in both taking stock of what is known in an area and in using that information for setting research priorities. Depicting the unique bird’s eye view of the field or fields that agency staff has, Berch described a variety of activities that directors engage in as they work directly

with leading investigators. These include such activities as assembling panelists to participate in workshops that consider the current state of knowledge and potential areas for breakthrough, and listening in on peer review panels on which scholars review proposals for new work—all of which coalesce to inform the ongoing development of research programs.

Barbara Schneider, of the University of Chicago, began the second part of the workshop by focusing on the idea of replication, a concept, she argued, that provides an important, unifying idea for creating scientific norms that can unite a community of researchers from different disciplinary perspectives. She asserted that replication begins with data sharing—it is the sharing of information about studies, including the actual data on which findings are based, that makes replication possible. Replication involves applying the same conditions to multiple cases, as well as replicating the designs, including cases that are sufficiently different to justify the generalization of results in theories, she said. Without convergence of results from multiple studies, the objectivity, neutrality, and generalizability of research are questionable.

In addition to addressing more specific topics, David Grissmer, of the RAND Corporation, provided important insights about strategies for knowledge accumulation in education research that explicitly relate theory, data, and measures and connect to the themes described in the previous section. He argued that generating consensus is not a matter of gathering more data or generating better techniques. “It is much more a matter of whether we can get replicable and consistent measurements across social science in general, and education, as a basis for forming theories.” Until there are consistent measurements, he went on to say, it is not possible to build broader theories. Furthermore, it is the role of theory to cut down on the amount of data collected. “Without theory, you collect everything. With theory, you can design a very specific set of experiments to test.” He argued that currently the field of education research is oriented toward making more measurements. As a result, “we have much research, but little knowledge.” Grissmer suggested that progress depends on the field focusing much more on exploring and explaining why research results differ to enable nuanced generalizations that account for variations in findings and contexts.

Several of the ideas and strategies for promoting an accumulated knowledge base in education research discussed during the session are

described in the main body of this report. A very brief synopsis of issues covered and speakers featured in each session is provided here.

Central to the conduct of research is the gathering of data on various measures. Common measures for the types of data collected by researchers can help to promote the accumulation of knowledge by facilitating the comparison and analysis of results across studies in both similar and disparate environments. In a session dedicated to this topic, two speakers elaborated on moving toward more common definitions of important measures in education research. Claudia Buchmann, of Duke University, discussed the development of measures of family background, including socioeconomic status. Michael Nettles, who at the time of the workshop was in transition between the University of Michigan and the Educational Testing Service, discussed issues surrounding the measurement of student achievement.

In her presentation, Buchmann offered a rationale for why measures of socioeconomic status and family background are important in education research and charted the progression of measure development that reflects the challenges of developing a common core of measures in education. She argued that family background measures are required to conduct a fair assessment of educational outcomes by enabling the isolation of outcomes from differences in inputs: student populations in different schools differ from the beginning, so it is necessary to control for this variation. Giving careful thought to how to measure family background relates to the necessity to improve knowledge of the ways that the family, as an institution, affects children’s ability and motivations to learn and their academic achievement. The bulk of Buchmann’s presentation focused on tracing the evolution of the concept of family background, which she demonstrated has become increasingly complex over time. She described simple socioeconomic status measures expanding to include an array of measures targeting different dimensions of this concept: for example, family structure or demographic characteristics, as well as family social and cultural capital. Buchmann also showed, compared, and critiqued how a sampling of major surveys and data collection efforts measured these concepts and their effects on the quality of inferences that could be drawn about key questions across and within them.

Nettles approached the idea of a common set of measures from a slightly different standpoint, focusing on the benefits and drawbacks of

using the National Assessment of Educational Progress (NAEP) as a centralized measure of achievement. He argued that there is a great deal of fragmentation and questionable stability in measuring student achievement. Although NAEP is appealing for a number of reasons, Nettles raised a number of issues related to student motivation, representativeness across geographic areas and other categories, the validity of the test for making particular inferences, and equity and bias, that have significant bearing on research that relies on these measures of student achievement.

Another set of tools or strategies that can facilitate the continued development of a coherent knowledge base is the sharing of data. In her introductory talk, Schneider pointed to three points of leverage for encouraging data sharing and replication: professional associations, scholarly journals, and data banks.

A panel that focused on data sharing followed consisted of five scholars from a range of positions and roles in the research community: individual investigators, senior officials from federal agencies, and journal editors. Ronald Ehrenberg, of Cornell University, discussed his experience using and reanalyzing the Coleman data. Grissmer focused on the role of NAEP. Marilyn Seastrom, of the National Center for Education Statistics, described the agency’s efforts to maximize access to data while maintaining privacy and confidentiality. Norman Bradburn, of the National Science Foundation, extended Seastrom’s presentation by focusing on broad concepts and tools associated with access, privacy, and confidentiality. And finally, Gary Natriello, of Teachers College, offered ideas on the role of journals in facilitating and promoting data sharing. Key points from these presentations are discussed in Chapter 3.

The workshop concluded with a session focused on ways of taking stock—that is, efforts by researchers to summarize what is known in topic areas or subfields. In various ways, investigators in a field periodically assess what (they believe) they know and formally or informally integrate findings from individual studies into the larger body of knowledge. The practice of researchers attempting to replicate previous studies is one way to assess the extent to which findings hold up in different times, places, and

circumstances. Similarly, a researcher who has piloted and evaluated a program at a small number of sites might scale up to a larger number of sites to see if and how results transfer to other settings. Research synthesis and meta-analysis are yet another way to summarize findings across studies of program effectiveness. Explicit efforts to engage groups of investigators (and other stakeholders) in building professional consensus can also generate summative statements that provide an indication of what is known and not known at a particular point in time.

Five speakers offered ideas for how the field can promote the accumulation of research-based knowledge through such work. Mehan and Slavin focused their talks on how scaling up programs or reform models to increasing numbers of schools offers opportunities for contributing to the advancement of scientific understanding while improving program services for participating schools. Cooper described meta-analysis, a methodology used to summarize the findings from multiple studies of program effects. Drawing on personal experience working with committees charged with developing consensus about research findings in areas of education, Wise described the consensus-building process of the NRC. Finally, Berch described the ways in which the National Institute of Child Health and Human Development attempts to understand what is known, what is not known, and how to craft research agendas and competitions based on that understanding.

The presenters seemed to agree that the accumulation of knowledge in education is possible, but challenging. The studies, methods, and activities they described together showed that careful, rigorous attempts to provide summative statements about what is known as a foundation for the continued advancement of scientific research in education are possible. To be sure, impediments exist. Cooper mentioned the tendency of advocacy groups to selectively rely on research results to support their (previously established) political positions and a lack of civility among researchers as particularly acute problems to be overcome. Summing up these sentiments, Cooper put it this way: “knowledge accumulation is possible, but it is not for the faint of heart.”

Cohen, D.K., Raudenbush, S.W., and Ball, D.L. (2002). Resources, instruction, and research. In F. Mosteller and R. Boruch (Eds.), Evidence matters: Randomized trials in education research, (pp. 80-119). Washington, DC: Brookings Institution Press.

Duncombe, W.D., and Yinger, J. (In press). How much more does a disadvantaged student cost? Economics of Education Review.

Greene, J.P. (1998). A meta-analysis of the effectiveness of bilingual education. Claremont, CA: Tomas Rivera Policy Institute.

Hanushek, E.A. (1986). The economics of schooling: Production and efficiency in public schools. Journal of Economic Literature, 24, 1141-1177.

Hanushek, E.A. (1997). Assessing the effects of school resources on student performance: An update. Educational Evaluation and Policy Analysis, 19, 141-164.

Monk, D.H. (1992). Education productivity research: An update and assessment of its role in education finance reform. Educational Evaluation and Policy Analysis, 14(4), 307-332.

Munroe, R.H., Munroe, R.L., and Whiting, B.B. (Eds.). (1981). Handbook of cross-cultural human development. New York: Garland STPM Press.

National Research Council. (1997). Improving schooling for language-minority children: A research agenda. Committee on Developing a Research Agenda on the Education of Limited-English-Proficient and Bilingual Students. D. August and K. Hakuta (Eds.). Washington, DC: National Academy Press.

National Research Council. (1999). Making money matter: Financing America’s schools. Committee on Education Finance. H.F. Ladd and J.S. Hansen (Eds.). Commission on Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.

National Research Council. (2002). Scientific research in education. Committee on Scientific Principles for Education Research . R.J. Shavelson and L. Towne (Eds.). Center for Education. Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.

Reschovsky, A. (1994). Fiscal equalization and school finance. National Tax Journal, 47(1), 185-197. [Reprinted in Joel Slemrod (Ed.). (1999). Tax Policy in the Real World (pp. 209-221). New York: Cambridge University Press.]

Reschovsky, A., and Imazeki, J. (2003). Let no child be left behind: Determining the cost of improving student performance. Public Finance Review, 31(3), 263-290.

Slavin, R.E. (1995). Cooperative learning: Theory, research, and practice. Boston: Allyn & Bacon.

Stedman, L.C. (1985). A new look at the effective schools literature. Urban Education. 20(3), 295-326.

Vygotsky, L. (1978). Mind and society: The development of higher psychological processes (M. Cole, V. John-Steiner, S. Scribner, and E. Souberman, Trans.). Cambridge, MA: Harvard University Press. (Original work published 1930.)

Willig, A.C. (1985). A meta-analysis of selected studies on the effectiveness of bilingual education. Review of Educational Research, 55(3), 269-317.

Yinger, J. (Ed.). (2004). Helping children left behind: State aid and the pursuit of educational equity. Cambridge, MA: MIT Press.