The title of this report reveals its purpose precisely: to spur actions that will advance scientific research in education. Our recommendations for accomplishing this goal build on the National Research Council report Scientific Research in Education. That report offered an articulation of what constitutes high-quality scientific inquiry in education; this report recommends ways to promote it.
Two pieces of recent federal legislation—the No Child Left Behind Act of 2001 and the Education Sciences Reform Act of 2002—have catapulted education research into the spotlight. Both acts are premised on the idea that education research can and should contribute to policy and practice, that education should be an evidence-based field. At the same time, the inclusion of definitions of what constitutes “scientifically based research” in both acts reflects deep skepticism about the quality and rigor of education research.
Some education research lacks quality, just as does some research in medicine, neuroscience, economics, or any other field. It is not necessary to denigrate or to defend the field on this point. The point is that scientific research in education can be improved, and the field should focus its energies on doing so.
The National Research Council convened the Committee on Research in Education to foster high-level dialogue with key participants in
education research to promote such improvements. To carry out this task, the committee organized a five-part workshop series and published several reports on selected topics.
In this final report, we offer recommendations for improving scientific research in education, organized around three strategic objectives: promoting quality, building the knowledge base, and enhancing the professional development of researchers. This is a time of unprecedented opportunity for education researchers to initiate bold reforms. The enthusiasm—and angst—surrounding recent calls for “scientifically based research” can and should be harnessed to advance the field of education research. The time to act is now.
The intellectual predecessor to this report, Scientific Research in Education, was an attempt to articulate what is meant by quality with respect to scientific research in education. The definition is a combination of six guiding principles that underlie all scientific fields, along with several features of education that shape how these principles apply to research on teaching, learning, and schooling. We adopted that framework as our working definition of quality of scientific education research.
Recently, much attention has been focused on “upgrading” the methods used in education studies, with a particular emphasis on randomized field trials to help establish cause-and-effect relationships. Methodologies are the tools researchers use to do their work; their appropriate use is essential to promoting quality. However, matching appropriate methods to research questions is a necessary but not sufficient condition for ensuring scientific rigor. We conclude that the national conversation about methodological quality is but a part of a needed broader focus on how to define and uphold quality in scientific education research. Issues such as the development of theory and the use of replications to clarify generalizability are examples of aspects of scientific quality that are equally important to consider.
Our recommendations for ways to promote quality—broadly defined—focus on peer review systems in federal agencies that support education research, the implementation of research designs in school settings, and partnerships between researchers and school personnel.
Recommendation 1. In federal agencies that support education research, the criteria by which peer reviewers rate proposals should be clearly delineated, and the meaning of different score levels on each scale should be defined and illustrated. Reviewers should be trained in the use of these scales. Defining, revisiting, and upholding standards of quality in the peer review process enhances the development of high-quality education research over time by facilitating reliable and valid ratings of proposals for funding and feedback to applicants.
Recommendation 2. Federal agencies that support education research should ensure that as a group, each peer review panel has the research experience and expertise to judge the theoretical and technical merits of the proposals it reviews. In addition, peer review panels should be composed so as to minimize conflicts of interest, to balance biases, and to promote the participation of people from a range of scholarly perspectives and traditionally underrepresented groups. The group of peer reviewers assembled to judge education research proposals should have the expertise to judge the content areas of the proposed work, the methods and analytic techniques proposed to address the research question, and the policy and practice contexts in which the work is situated. Agency staff should seek to eliminate conflicts of interest among reviewers. However, because many of the best reviewers are likely to have some association with applicants, overly restrictive conflict of interest rules can dramatically shrink the pool of competent reviewers. Biases among peer reviewers need not be eliminated, but rather must be identified, discussed among the group, and balanced across panelists. Ensuring a breadth of perspectives in peer review panels promotes high-quality reviews over time by engaging different kinds of expertise and insights around a common set of proposals, issues, and evaluation criteria.
Recommendation 3. In research conducted in educational settings, investigators must not only select rigorous methods appropriate to the questions posed but also implement them in ways that meet the highest standards of evidence for those questions and methods. The choice of research design and method must be driven by the par-
ticulars of the question posed for investigation. The implementation of the design in school settings is equally important: challenges arise in working with schools, and strategies must be in place to anticipate them and to solve unanticipated problems as they arise. Partnerships between the research team and school personnel facilitate effective implementation.
Recommendation 4. Federal agencies should ensure appropriate resources are available for education researchers conducting large-scale investigations in educational settings to build partnerships with practitioners and policy makers. Time and money are needed to develop the partnerships necessary to ensure effective implementation of large-scale education research studies. Research project budgets should provide such resources.
BUILDING THE KNOWLEDGE BASE
Even if the quality of discrete education research projects has been ensured, if the field lacks the will or the tools to forge connections among studies, it will amass a multitude of studies that cannot support inferences about generalizability nor sustain the theory building that underlies scientific progress. We conclude that greater attention must be paid to reanalysis, replication, and testing the boundaries of theories with empirical inquiries, as well as to taking stock of what is known in areas of interest to education policy and practice on a regular basis.
Our recommendations for building the knowledge base focus on data sharing, infrastructure development, and journal policies.
Recommendation 5. Professional associations involved in education research should develop explicit ethical standards for data sharing. The American Educational Research Association and similar groups should be at the forefront of efforts to promote the sharing of education-related data among qualified investigators to enable reanalyses, replications, and further investigation with available data. Ethical standards should consider how to ensure the confidentiality of research participants, especially with qualitative data. The rights and protections of authors should also be specified.
Recommendation 6. Education research journals should require authors to make relevant data available to other researchers as a condition of publication and to ensure that applicable ethical standards are upheld. Cultural barriers and institutional disincentives that work against data sharing and related practices should be addressed candidly and reformed thoughtfully.
Recommendation 7. Professional associations and education research journals should work in concert with funding agencies to create an infrastructure that takes advantage of technology to facilitate data sharing and knowledge accumulation in education research. Promising mechanisms include data repositories, registries of initiated studies, bibliographic indexes of published studies, digitization of journal content, and open access.
Recommendation 8. Education research journals should develop and implement policies to require structured abstracts. Abstracts are used in the development of systematic reviews of multiple education studies on similar topics to identify the universe of relevant research. To facilitate these reviews and to promote better access to relevant studies among the many consumers of education research, all abstracts should contain basic information about the purpose, sample strategy, methodology, and other key features of the investigation.
ENHANCING PROFESSIONAL DEVELOPMENT
A diverse pool of well-trained education researchers is needed to contribute to deliberations about educational practice and policy in response to the complex questions being asked of education research. The question is: Are existing training and professional development activities sufficient to produce a capable cadre of investigators and to respond to the demands of practitioners and policy makers?
Our recommendations for improving the professional development of education researchers focus on doctoral programs in schools of education, and the peer review systems in both federal agencies and in journals.
Recommendation 9. Schools of education that train doctoral students for careers in education research should articulate the competencies those graduates should know and be able to do and design their programs to enable students to develop them. The articulation of competencies is essential for designing course work, organizing research experiences, and developing other program elements. Such an exercise would also define a minimum breadth of skills all would-be education researchers should have. This articulation may require differentiation within programs of schools of education, such as educational psychology and curriculum and instruction.
Recommendation 10. Schools of education that train doctoral students for careers in education research should design their programs to enable those students to develop deep substantive and methodological knowledge and skill in a specialized area. As students progress through their doctoral training, their course work and research experiences should hone their skills and understanding in the theoretical ideas, methodological tools, and existing research in the particular area in education research they intend to pursue. Interdepartmental collaborations can often facilitate in-depth training by providing opportunities for students to explore areas and to work with faculty outside schools of education.
Recommendation 11. Schools of education that train doctoral students for careers in education research should provide those students with a variety of meaningful research experiences. Research experience while in training is absolutely essential to a research career. Staging a series of research experiences over the course of doctoral study facilitates the development of research skills and provides opportunities for publishing research findings in peer-reviewed journals, presenting at conferences, and participating in other activities that are the foundation of the profession. Ensuring meaningful research experiences for doctoral students requires that they engage in research under the guidance of multiple faculty members who themselves are active in the field.
Recommendation 12. Peer review panels in federal agencies that fund education research should be composed to promote the participation of people from a range of scholarly perspectives and traditionally underrepresented groups and provide opportunities for professional development. Although not typically viewed as a vehicle for professional growth, properly designed peer review experiences can provide opportunities for interaction, feedback, and interdisciplinary conversations that promote learning among applicants, reviewers, and agency staff. Such design features as standing panels, which enable interactions over time, and clear and consistent feedback to applicants can be effective ways to promote this professional development.
Recommendation 13. Publishers of peer-reviewed education research should design their editorial and manuscript review systems to promote the professional development of education researchers who participate in that process. Just as peer review of research proposals can be an enriching experience for those involved, so too can the process of judging manuscripts submitted to journals for publication. Opportunities for editors, authors, and reviewers to benefit from each others’ ideas and critiques should be maximized.
We call on the three major institutions to which these recommendations are addressed—federal funding agencies, schools of education and universities, and professional associations—to work together towards promoting quality, building the knowledge base, and enhancing the professional development of researchers. A shared commitment among them can lead to the partnerships, strategic investments, and infrastructure support needed to advance scientific research in education.