Summary and Conclusion
In this final chapter, we present all of the committee’s recommendations and suggest directions for future work in advancing scientific research in education.
Recommendation 1. In federal agencies that support education research, the criteria by which peer reviewers rate proposals should be clearly delineated, and the meaning of different score levels on each scale should be defined and illustrated. Reviewers should be trained in the use of these scales.
Recommendation 2. Federal agencies that support education research should ensure that as a group, each peer review panel should have the research experience and expertise to judge the theoretical and technical merits of the proposals it reviews. In addition, peer review panels should be composed so as to minimize conflicts of interest and balance biases and promote the participation of people from a range of scholarly perspectives and traditionally underrepresented groups.
Recommendation 3. In research conducted in educational settings, investigators must not only select rigorous methods appropriate to the questions
posed but also implement them in ways that meet the highest standards of evidence for those questions and methods.
Recommendation 4. Federal agencies should ensure appropriate resources are available for education researchers conducting large-scale investigations in educational settings to build partnerships with practitioners and policy makers.
Recommendation 5. Professional associations involved in education research should develop explicit ethical standards for data sharing.
Recommendation 6. Education research journals should require authors to make relevant data available to other researchers as a condition of publication and to ensure that applicable ethical standards are upheld.
Recommendation 7. Professional associations and education research journals should work in concert with funding agencies to create an infrastructure that takes advantage of technology to facilitate data sharing and knowledge accumulation in education research.
Recommendation 8. Education research journals should develop and implement policies requiring structured abstracts.
Recommendation 9. Schools of education that train doctoral students for careers in education research should articulate the competencies their research graduates should know and be able to do and design their programs to enable students to develop them.
Recommendation 10. Schools of education that train doctoral students for careers in education research should design their programs to enable research students to develop deep substantive and methodological knowledge and skill in a specialized area.
Recommendation 11. Schools of education that train doctoral students for careers in education research should provide all students with a variety of meaningful research experiences.
Recommendation 12. Peer review panels in federal agencies that fund education research should be composed to promote the participation of people
from a range of scholarly perspectives and traditionally underrepresented groups and provide opportunities for professional development.
Recommendation 13. Publishers of peer-reviewed education research should design their editorial and manuscript review systems to promote the professional development of education researchers who participate in that process.
RECOMMENDATIONS BY TARGET AUDIENCE
The target audiences for this report are education researchers and the institutions that support them. Most of the recommendations we make will require effort on the part of individual researchers as well as multiple organizations. Table 5-1 organizes the recommendations according to the organizational entity that will need to lead its implementation; each has a role in pursuing the three strategic objectives of promoting quality, building the knowledge base, and enhancing the professional development of researchers. The three central organizations targeted are federal research agencies that support education research, professional associations and publishers of education research, and schools of education and universities.
The first column of the table reflects the lead role we envision for federal agencies in implementing recommendations related to promoting quality—they design the peer review systems that play a critical role in judging the quality of proposed work; they are a centralized source of financial support that can and should invest in critical infrastructure needs; and they can facilitate the development of partnerships between researchers and educators by supporting them both substantively and financially when funding large-scale research projects. Other organizations that rely on peer review—for example, philanthropic foundations in their grant-making processes and professional associations in the development of their annual meeting agendas—can and should play a similar role in designing their peer review systems to promote high quality scholarship. We also highlight the role that peer review systems in federal agencies can play in ongoing professional development of a diverse set of education researchers as another unsung yet powerful vehicle for enhancing the professional development of the field.
The middle column highlights the central role of professional associations like the American Educational Research Association (AERA) in developing a knowledge base of education research that accumulates over time.
TABLE 5-1 Recommendations by Lead Target Audience
AERA and related membership associations are a natural place to spearhead further infrastructure development by, for example, providing a forum for groups of individual investigators to explore avenues for implementation of these recommendations. In partnership with federal funding agencies, these associations should take a leadership role in developing an infrastructure that supports the development, maintenance, and use of databases, data repositories, and registries. Through policy changes, associations should actively encourage the sharing of data by incorporating an explicit standard into their ethical guidelines that pertain to investigators’ responsibility to the field. They should also tie the publication requirements of association journals to that standard by mandating authors to make relevant data avail-
able to competent researchers and by creating manuscript review processes that facilitate professional growth among education researchers.
Finally, in the last column we target schools of education and university leaders to reinforce their central role in promoting the development of a highly competent field of education researchers. Schools of education are most commonly known for training educators and administrators, but the role of those schools that offer doctoral programs in education research deserves far greater attention both within individual departments and at the highest ranks of university leadership. These programs are a key leverage point for developing a talented pool of education researchers capable of tackling the next generation of challenging research questions. Schools of education and universities can also play an important role in promoting quality by infusing high standards of rigor into the pursuit of both teaching (e.g., through the development of coursework, research experiences, and experiences working with schools and districts for students) and research (e.g., through hiring, tenure, promotion, and other mechanisms that reward faculty for scholarly contributions). They might also play a role in developing effective peer reviewers in their training programs, in promoting the participation of future teachers and administrators in education research through their Ed.D. or related degree programs.
Additionally, while not called out in our recommendations specifically, schools of education play a central role in training future education researchers to value, and to be competent in conducting, research that integrates, replicates, or summarizes existing data or publications. The recognition of the value of such work in developing a knowledge base should be reflected in the incentives that are built into the tenure and promotion policies of the universities and schools as well.
No one institution could or should implement these recommendations alone. Rather, to promote improvements in education research capacity and infrastructure as broadly defined by the committee, their implementation will require leadership and resources from the many organizations and individual investigators that constitute the diverse and diffuse field of education research. For example, we call on the major professional associations to develop standards for data sharing in Recommendation 5. The extent to which these standards are met and followed among education researchers, however, depends greatly on explicit support for data sharing by federal agencies and universities, through, for example, providing financial support to investigators to prepare their data for reuse and acknowledging the development of protocols for data sharing as a mean-
ingful, scholarly contribution, respectively. Also, although we did not focus on the role of foundations in our work, they are certainly prime candidates for partnering in the implementation of the infrastructure-building recommendations in particular, and their potential contributions should be considered and encouraged.
The committee’s recommendations are geared to the objective they most directly serve in the advancement of scientific education research: promoting quality, developing an accumulated knowledge base, and enhancing the professional development of education researchers. In reality, these objectives are closely related. For example, we have treated issues related to the quality of research as largely applying to individual investigations of educational phenomena and issues related to the building of the knowledge base as applying to the long-term development of lines of inquiry over time. These concepts are intertwined and interdependent, however; the quality of individual studies cannot be understood separately from the broader lines of inquiry within which they are situated. The quality of education research can and should be viewed in the long term, and therefore, issues of how to frame and promote high-quality portfolios of work in key areas lead directly to questions of how to support the building of a coherent knowledge base from education research investigations over time. Furthermore, focusing on the professional development of education researchers during their doctoral work and throughout their careers requires that training and growth opportunities be provided in ways that reinforce high standards of quality and that promote an understanding of, and strategies for, integrating research across studies, subfields, disciplines, and theoretical paradigms (i.e., building the knowledge base).
To marshal both the intellectual and financial resources that will be required to implement these recommendations, individual members of the education research communities will need to support the ideas behind them in everyday interactions: in faculty meetings, in teaching, in writing funding proposals, in carrying out investigations, and, especially, in leadership positions. It is through this kind of grassroots dialogue and reinforcement about how to advance the field as a whole that the ethos and policies of the organizations we target will change to promote the committee’s recommendations.
ISSUES FOR FUTURE CONSIDERATION
The committee’s deliberations about ways to improve scientific research in education were necessarily limited by the topical areas featured in the
committee’s workshop series. In our view, these events raised important issues and suggested a set of strategic objectives to pursue that have great promise for advancing the field. These meetings were not designed, however, to cover all of the important issues facing the field today. We view our work as one in a series of steps that should be taken by members of the education research communities to reflect upon and advance their work and profession. In this section, we offer a few ideas of issues and problems to be taken up in such future efforts.
The report highlights the importance of the will of individual investigators to contribute to the kind of community building work we recommend be done: secondary analyses and replications; crossing disciplinary and institutional boundaries for doctoral training of researchers; engaging in consistent efforts to take stock of what is known; and dedicating substantial time to conducting and participating in peer review systems. We have suggested that the institutional incentive structures that shape the types of activities individual investigators choose to pursue at best tend to devalue these efforts, and at worst, they erect formidable barriers to engaging a cadre of talent in this crucial work. For example, few grant-making organizations support important yet costly syntheses or replications (preferring “new” investigations); major academic journals similarly discourage attempts to reanalyze data or otherwise extend previous work; investigators are hesitant to share their data because of the cost of preparation, the risk of someone else publishing from their data before they have the opportunity of doing so, and concerns about protecting the rights of research participants; and cross-disciplinary projects and endeavors require a considerable expenditure of energy that rarely “count” toward faculty tenure or promotion and that may not result in publication as lead authors in elite disciplinary journals critical to professional advancement. These are serious issues that require serious analysis and reconsideration of core principles and policies.
A second area worthy of investigation has to do with the relationship between the “supply” of education research and the “demand” for it by educators. This metaphor is featured frequently in discussing aspects of evidence-based education; indeed, we use it in the opening chapter to describe the parameters of our own work. In reality, however, the concepts of quality (supply) and utility (demand) are related and, in many cases, interdependent. Several workshop discussions illustrated this idea clearly: in discussing quality criteria and the types of expertise needed on peer review panels, for example, consideration of whether and how practitioners and stakeholders should be involved led immediately to a tangle of ques-
tions about how to think about quality in an applied field like education. Discussions about the role of partnerships between researchers and educators, and our recommendations regarding their encouragement and development, underscore the interdependence of both groups of professionals in promoting both high-quality research and research that has utility for schools. The successful implementation of research designs, and thus the validity of the conclusions drawn from the research, depends heavily on building relationships, establishing trust, and designing studies to advance both research purposes and the long-term needs of schools and school districts. Finally, the dual purpose of training future leaders in education and in education research within schools of education was characterized as a double-edged sword: the marriage can meaningfully embed the complex issues of practice into research conceptualization, implementation, and interpretation; but it can also slow the development of discipline-like norms and practices among researchers. We think in-depth consideration of such issues at the nexus of quality and utility in education research is needed.
The mix of researchers and educators in the training programs in schools of education not only highlights the important and complex ties between quality and utility, but it also raises issues about the training and participation of practitioners in education research. In our deliberations about the role of doctoral programs in schools of education, we made a strategic decision to focus on those schools that trained education researchers, recognizing that not all schools have such programs. However, improvements in education research also depend on the development of a profession in education practice that understands, values, and authenticates research. There are important questions about whether and how teacher and administrator candidates should be trained in the conduct or use of education research.
A final issue we deem worthy of in-depth consideration in future work pertains to the role and selection of journal editors in peer-reviewed education research publications. Our deliberations on this topic suggest that this key position be conceived as one that not only heavily influences the nature and quality of published work, but that also furthers the professional development of the field. From the handful of editors we heard from in the workshop series, the logistical burdens placed on education research journal editors seemed to far outstrip those of their peers at social and behavioral science journals. These burdens exact a hefty price: without adequate resources to support an efficient and effective editorial process, editors cannot devote their time to focus on the critical substantive responsibilities
they have in promoting high-quality scholarship or to develop the professional capacities of the field. And easing the burden of editorship in education research journals would help to expand the talent pool of people who could be editors: for example, qualified investigators from small, typically cash-strapped universities, could more feasibly participate. Attending to these resource issues and carefully planning selection processes for editorships are critical issues worthy of further consideration.
The committee presents its recommendations while recognizing the hard work and investment of scarce resources that lies ahead to make progress toward advancing scientific research in education. We believe that it is a fundamental professional responsibility for individual investigators to contribute their talent and time to developing the core capacity of their community. It is in this spirit that we call on education researchers to focus on pursuing objectives that serve the common goals of the field: promoting research quality, facilitating the development of an accumulated knowledge base, and enhancing professional development.
This is a time of unprecedented opportunity for the various institutions and individuals who make up the field to initiate bold reforms. The enthusiasm—and angst—surrounding recent calls for “scientifically based research” can and should be harnessed to advance scientific research in education. The time to act is now.