Advancing Scientific Research in Education (2005)

The title of this report reveals its purpose precisely: to spur actions that will advance scientific research in education. The recommendations for accomplishing this goal, detailed in the pages that follow, build on the National Research Council (NRC) report Scientific Research in Education (National Research Council, 2002). That report offered an articulation of what constitutes high-quality scientific inquiry into education; this report recommends ways to promote it.

The field of education research as a distinct area of scholarly inquiry has evolved over roughly 100 years. It comprises a group of investigators from different disciplines, fields, and institutions who bring a range of theories, objectives, and orientations to their work. It is grounded in the social and behavioral sciences, but it lacks a disciplinary framework like those that shape the academic study of anthropology, or economics, or psychology. More akin in some ways to professional fields like social work or public policy, education research takes cues from the practice of teaching, the process of learning, and the organizational structures and routines of the many institutions with education-related missions. Sharing commonalities among these related scientific and professional fields, education research nonetheless bears its own distinctions as a field of study. Indeed, since its earliest days, education researchers have debated the core nature of

education research given the complexity of the subject and the diversity of the field of investigators.

Recent changes in the education policy landscape have drawn these debates out of the exclusive realm of academe, making for especially interesting times for education research. The proliferation of standards-based reforms and high-stakes accountability regimes over the past 20 years has slowly but steadily built demand for research “proven” strategies among educators.¹ Most recently, two pieces of federal legislation—the No Child Left Behind Act of 2001 and the Education Sciences Reform Act of 2002—have catapulted education research into the spotlight, with as yet unknown effects.

Both acts are premised on the idea that education research can and should shape policy and practice or, as the Department of Education’s Strategic Plan puts it, education should be “transformed into an evidence-based field” (U.S. Department of Education, 2002b). Key decision makers involved in crafting these policies often frame the issue this way: if patients expect that their physician’s practice is informed by the best available research evidence, why shouldn’t parents expect that teachers and administrators are delivering instruction and organizing learning environments in the same way? Evidence-based medicine is not as widespread a trend as one might expect or hope (Sackett, Richardson, Rosenberg, and Haynes, 1997; Institute of Medicine, 2001), and education is like and unlike medicine in complex ways. But evidence-based education is a comparatively new trend (National Research Council, 1999).

The explicit coupling of research and reform in federal education law has spurred a range of reactions and actions among education researchers. Some have attempted to seize the opportunity and work to promote the goals of evidence-based education. Others are troubled by the problems they see with these policies, focusing on the fact that both education laws

define legislatively (politically) what constitutes “scientifically based research” and expressing concern about what they view as an inappropriate encroachment on their profession.

In late 2000, when the reauthorization of the U.S. Office of Educational Research and Improvement was first under consideration by Congress (these and subsequent deliberations would eventually lead to the passage of the Education Sciences Reform Act of 2002), the NRC was approached to bring the expertise of education researchers and other scientists to bear on the complex question of how to characterize scientifically based research in education. The result was the publication Scientific Research in Education (National Research Council, 2002), which was authored by a group of researchers and intended to provide a scholarly rendering of the major issues and terms that had made their way into the standard rhetoric of education policy makers. There is some evidence to suggest that the book influenced revisions that were made to the definition of scientifically based research in the Education Sciences Reform Act (Eisenhart and Towne, 2003).

Regardless of the content of such definitions of scientifically based research that appear in education law, the inclusion of such definitions in federal statutes reflects a deep skepticism about the quality and rigor of education scholarship. So, too, have some scholars expressed deep concern about a lack of quality in education research (Kaestle, 1993; Levin and O’Donnell, 1999; Carnine, 2000; Vaughn and Damann, 2001). It is almost certain that some education research lacks quality, just as it is almost certain that some medical research, some biological research, and some neuroscience research lack quality. It is not necessary to denigrate or to defend the field on this point. What matters is that the current landscape offers a ripe opportunity for self-reflection and improvement, and that is our point of departure: scientific research in education could be improved, and the field should focus its energies on doing so.

The NRC convened the Committee on Research in Education to foster high-level dialogue with key participants in education research to promote such improvements. The committee embarked on two related strands of work to accomplish this objective: a five-part workshop series to engage a wide range of scholars, policy makers, and educators in an action-oriented dialogue to clarify issues and to discuss ways in which current

practice could be improved, and a series of reports on selected topics raised in the workshop series.

As described in more detail in the preface, the topic areas for the workshop series include peer review in federal agencies that support education research, strategies to promote the development of a knowledge base in education, implementation of randomized field trials in educational settings, the role of journals in contributing to a knowledge base in education, and doctoral programs for education researchers. The committee’s web site (http://www7.nationalacademies.org/core/) features a page dedicated to each of the events held on these topics. For a given event, an agenda with hyperlinks, broken down by presentation, enables easy viewing of biographical information about the speakers, Powerpoint slides if used, and (in most cases) verbatim transcripts of speaker remarks and audience participation.

The committee selected the workshop topics on the basis of a number of related factors. First, they were issues our sponsors were interested in pursuing as points of leverage for improvements in scientific education research. For example, recognizing that the federal government was poised to fund many more randomized field trials in education than had been conducted at any other time in history, the former National Educational Research Policy and Priorities Board was interested in implementation issues associated with such studies; therefore, the committee brought researchers and practitioners together to share best practices about how these types of studies can be successfully implemented in real-world educational settings. The William and Flora Hewlett Foundation invested in the committee’s work as well, charging us with focusing on concrete ways to facilitate a more integrated knowledge base in education, including the role that journals might play; thus, we convened workshops to discuss several potential strategies for promoting knowledge accumulation, including journal policies and practices.

A second factor we considered in designing the individual workshops was how the themes and ideas contained in Scientific Research in Education could be extended. For example, that book stressed the importance of a culture of scientific rigor among the field of investigators in promoting high-quality education research, and we designed the final workshop to focus on one key lever for instilling that culture and developing capacity: doctoral training in schools of education of future leaders in education research.

We also considered what topics were of particular salience to educa-

tion policy issues. Peer review, for example, is a concept that is featured in the definitions of “scientifically based research” in both the No Child Left Behind Act and the Education Sciences Reform Act and of course, a longstanding practice in many scholarly fields to promote high-quality reseach. Two of our workshops therefore focused on the role of peer review: one focused on peer review as it is used in federal agencies to vet proposals for research funding, and one included discussion of the review of manuscripts submitted to journals for publication. Overall, the series touched on a wide range of important issues and ideas and provided rich fodder for discussion and for the development of this report. We cannot claim, however, that we have conducted an exhaustive analysis, nor that these recommendations are the only ways scientific research in education could be improved.

With the publication of this report, we have issued three reports based on select topics in the workshop series. The first two, Implementing Randomized Field Trials in Education: Report of a Workshop (National Research Council, 2004a; available: http://books.nap.edu/catalog/10943.html) and Strengthening Peer Review in Federal Agencies That Support Education Research (National Research Council, 2004b; available: http://books.nap.edu/catalog/11042.html) are based on specific workshop discussions and issues. We chose to issue these two topical reports based on our judgment of the issues as most pressing and promising in the education policy and research circles. Randomized field trials have dominated much of the policy debate in the past few years, but since little had been written about implementation issues, we chose to issue a short report that summarized our workshop on the topic. Further, the Education Sciences Reform Act called for the formation of a new policy board—the National Board of Education Sciences—which will work with the director of the Institute of Education Sciences (IES) to formulate and oversee the agency’s peer review system. Thus, we issued a report that contains conclusions and recommendations about peer review systems in federal agencies that support education research (including, but not limited to IES) to be useful to such officials charged with developing or revamping peer review systems in the near future.

Over the course of the workshop series, common themes emerged from the discussions of a fairly diverse set of topics. This final report reflects those cross-cutting themes and ideas and points to a set of strategies the committee views as most promising for promoting targeted improvements in scientific research in education. It is therefore organized

around major themes from across the workshops, rather than by workshop topic.

The primary source of evidence for the conclusions and recommendations in this report is the presentations and discussions that took place in the workshop series. The committee designed these events to promote broad-based discussions of a range of complex issues in education among leading scholars in the field. Each event included ample time for the committee to ask questions of presenters, as well as for audience members to ask questions and to add their perspectives.

We also draw on our own collective experience as researchers and practitioners in education and other fields. Together, we have held several leadership positions in many of the key organizations we target, including journals, professional associations, and schools of education. These experiences supplement the workshop dialogue to support the conclusions and recommendations.

Drawing primarily on workshop discussions to support our conclusions and recommendations has important implications for the nature of the committee’s recommendations and for how we treat and use this type of evidence in the report. Because the kinds of issues raised in the workshop series and the associated underlying knowledge bases about them vary, so, too, do our recommendations. In most cases, the recommendations are statements of critical objectives to be pursued by the field with suggestions for possible mechanisms; in others, they contain specific strategies for action by specific organizations. This variability reflects the fact that the workshops themselves were designed to be explorations of major issues related to education research to build on—but not complete—the work of the committee that authored Scientific Research in Education. We also selectively chose ideas and strategies raised during the workshop series that we judge as most important to highlight and to pursue. Thus, the committee offers targeted recommendations to be considered broadly and implemented intelligently over time. Our goal is to push the conversation to the next level and to spur positive change.

Throughout the report, we offer examples from our workshops of issues related to and practices of federal agencies, professional associations, schools of education, and journals (our target audiences; see page 16).

These practices, however, are not necessarily representative, and do not constitute the thorough baseline assessment that will be needed to implement change effectively over time. That work remains to be done.

A similar caveat relates to the costs associated with implementing the recommendations. Cost estimates were rarely available, and indeed, because many of the committee’s recommendations are framed broadly, estimating cost would entail the development of more detailed plans for action. There is no doubt that new resources will be needed and that implementation will need to take place incrementally as resources become available. Implementing the recommendations effectively will take strategic investments that leverage existing resources and build capacity in and across organizations over time.

Since the committee’s primary source of evidence for our recommendations is this series of events and discussions, throughout the report we formally cite specific workshop speakers. Corresponding to these citations in the text, the reference list contains information about the date and topic area of each workshop and a link to the transcript of the specific presentation referenced from the committee’s website. This website also contains short biographical sketches of each presenter, as well as more detailed information about the committee and its work.

The focus in the workshop series, and therefore in this report, is on scientific research in education. As a committee of the NRC—the operating arm of the National Academy of Sciences—a basic premise of our work is that the pursuit of scientific understanding can be a powerful tool for the betterment of society. The committee approached the task of recommending strategies for improving scientific research in education with the strong belief that it can and should be used to improve education policy and practice.

Steadfast in this belief, we also recognize and respect that scholarly inquiry in the social sciences and education is not limited to scientific approaches. Indeed, Scientific Research in Education (National Research Council, 2002)—a book we did not author but that has shaped our work—proved to be controversial among some education researchers, for three main reasons. Critics faulted the book for accepting uncritically the premise that scientific research in education is possible and worthwhile, for depicting a flawed or outmoded view of what constituted scientific inquiry into

education phenomena, or for being silent on the role of politics in defining scientifically based research in education (Erickson and Guitterez, 2002; St. Pierre, 2002; Eisenhart and Towne, 2003).

Given these controversies, we think it is our responsibility to convey that there is more than one way to view the world and that science is not universally applicable to understanding all issues relevant to education or its improvement. Since the committee’s charge was to address ways in which scientific education research could be improved, however, we do not consider the relative merits or contributions of approaches to education research that do not define themselves as scientific, approaches that include such disparate paradigms as interpretivism, postmodernism, and critical theory.

The committee’s own epistemological and theoretical orientations—which vary among the members—clearly shape how we went about our work. We do not attempt to systematically explore these issues or to adopt a single framework, however. Rather, we have taken the insights and ideas generated at the workshops and worked together to reach consensus on a set of strategies that collectively we think can advance scientific education research, as well as for its use in promoting improvements in education policy and practice.

In addition, the focus of the report is on improving the capacity of the research communities to provide a scientific basis for proposed reforms and other policy decisions that affect education. We do not take up the complex question of how to promote the utilization of education research in ways that improve educational outcomes. That said, the two are inextricably linked. Indeed, there are places throughout the report where we consider the nexus between quality and utility quite explicitly. Thus, while the overarching goal is to facilitate research-based reform, we primarily tackle the “supply” side of the equation—that is, how to strengthen the quality of scientific education research, without directly addressing the “demand” side—that is, how to promote effective use of that research as a crucial part of the formulation and implementation of policy and practice.

The primary audience for this report is education researchers and the institutions that support them—universities, federal agencies, professional associations, and foundations. Since the focus is on advancing scientific research in education, these are the people and organizations most central

to the effective implementation of the recommendations the committee proposes. We found over the course of our deliberations—and know from our own work and experience—that it is often unclear where to target reform efforts in the field of education research. A multitude of participants and decision makers overlap in their authority, responsibility, and power, and existing incentives often work against change. We have attempted to focus the recommendations on major institutional leverage points, and in the final chapter, we provide a summary of these recommendations categorized by audience.

We also think that education policy makers involved in implementing evidence-based reforms and practitioners who are involved in research studies or engaged in using the results of such studies will find some of the issues in this report to be relevant to their work.

Across the topic areas addressed in the workshop series, we identified three strategic objectives for advancing scientific research in education:

1. promoting quality,

2. building the knowledge base, and

3. enhancing professional development.

Thus, the recommendations in this report have been organized to align with these strategies. Although some workshops track closely with these areas (for example, the workshop on doctoral programs in schools of education addresses professional development more so than the other areas), many cut across them.

A central idea that runs throughout the recommendations is that the diversity of the people in the field—with respect to the range of scholarly perspectives, training and background, and such demographic and social characteristics as race, ethnicity, gender, and age—can be very powerful if it rests on a common foundation. Diversity can promote quality, enhance legitimacy, and extend opportunity, but without reference to a common core, it can lead to fragmentation. Standardization without the flexibility to accommodate varying points of view leads to stultification. We seek to harness and extend the diversity of the field, while calling for attention to defining and reinforcing a common professional culture. One cannot exist

without the other, and our recommendations are designed to reflect that premise.

A second and related idea throughout our analysis and recommendations is the importance of an active field of peers working to develop and to reinforce a professional culture. Whether through peer review processes in vetting proposals for research funding or manuscripts for publication, doctoral training, or informal communications and relationships, it is the participation of researchers in activities that strengthen the field as a whole that will advance scientific research in education. By recognizing common goals and working together to achieve them, there is great potential to further the field.