Strengthening Peer Review in Federal Agencies That Support Education Research (2004)

Peer review is a method used to inform decision making by engaging experts in a critical evaluation of the merits of a product or proposal. It is most commonly known as a mechanism for judging the quality of proposals for research funding, or manuscripts submitted for publication in academic journals.

The focus of this report is on peer review as it is applied to the evaluation of proposals for federal funding of education research projects. The primary source of evidence we use to set forth our conclusions and recommendations about this topic is a workshop we convened in February 2003 in Washington, DC. The agenda for that event, a full transcript of the day’s presentations and discussions, and a paper we commissioned to lay the groundwork for our deliberations are all available on the web at http://www7.nationalacademies.org/core/Peer%20Review.html.

A long-standing tool of science policy in the United States, peer review is widely recognized as the preferred method for judging the merits of proposals for research funding. Across the federal government, it is used in a variety of contexts and for a variety of purposes—both scientific and political in nature. It is at once a tool with which scientific judgment is formalized and decisions about the allocation of scarce public resources are legitimized.

In the many federal agencies that support education research,¹ peer review practices vary widely. Historically, this variation has largely been a function of differences in culture, tradition, mission, and the fields and disciplines each agency supports. These factors are important inputs to the development of peer review systems, and each agency should have the flexibility to create a system tailored to meet its needs and to accommodate its circumstances. While we acknowledge this variation and encourage agency flexibility, all peer review systems can and should uphold basic principles of fairness and merit.

For federal agencies that support education research, we recommend that in addition to these considerations, peer review should be explicitly designed, managed, and evaluated to promote two key objectives: to identify and support high-quality scientific research in education and to promote the professional development of the field. Adopting these objectives as guideposts will enable agencies to think strategically about the merits of, and trade-offs associated with, particular practices and design options. To illustrate this central idea, we analyze key elements of peer review through this lens.

Peer review can foster the development of a high-quality research portfolio in federal agencies that support education research. Two central issues form the core of how to develop a system that supports this goal: deciding who counts as a peer and defining and consistently implementing standards of what high-quality means.

Assembling the group of reviewers is the very crux of the matter: the peer review process, no matter how well designed, is only as effective as the people involved. Judging the competence of peers—individually and as a group—in any research field is a complex task requiring assessment on a

number of levels. In education research, it is particularly complex because the field is so diverse (e.g., with respect to disciplinary training and background, epistemological orientation) and diffuse (e.g., housed in various university departments and research institutions, working on a wide range of education problems and issues).

The types of expertise reviewers should bring to the peer review table to identify high-quality education research include the content areas of the proposed work; the theoretical models, methods, and analytic techniques proposed to address the research questions; and the practice and policy contexts in which the work is situated. No single individual is expected to contribute expertise in all of these areas. Rather, panels must bring combined expertise; thus, expertise in the context of assembling peer reviewers should be viewed in terms of the panel as a whole.

Vetting and publicly disclosing reviewers’ potential conflicts of interest and biases is essential to identifying panelists who will fairly judge proposals on their merits. In practice, many top-flight researchers predictably have potential conflicts or biases with respect to particular applicants in a research competition: they may have collaborated on projects, coauthored papers, or mentored or taught the investigator seeking funding. Reviewers must be as free as possible from such potential conflicts of interest, as they cast doubt on their ability to judge the proposals on their merit alone. However, because many of the best reviewers are likely to be linked in some way to the work under consideration, if conflict of interest rules are too strict, the talent pool of reviewers can shrink dramatically. Also, the biases, or preferences, that reviewers bring to their work should be aired and balanced across panelists so that no single paradigm or perspective dominates the panel review. The key is public disclosure of potential conflicts and biases so that others can gauge the significance of any identified conflict and bring up those that might not be identified.

Two broad types of diversity are relevant to assembling high-quality panels and to promoting education research quality through peer review: diversity of scholarly perspectives and diversity of groups traditionally underrepresented in education research. Engaging researchers who approach the topic under consideration from a range of perspectives can enrich peer review deliberations by bringing together a diverse set of expertise around a common set of issues and problems. And explicitly reaching out to traditionally underrepresented populations to participate in peer review fosters an environment in which questions are provoked and issues raised that otherwise might not have surfaced, helping to ground the review in

the cultural and social contexts in which the work is proposed to be conducted and promoting the quality of the research over time. Actively pursuing diversity along both of these dimensions in an agency’s peer review system has other benefits as well, including lending the process legitimacy and enhancing and extending powerful learning opportunities that take place in peer review deliberations.

There is a final and particularly contentious issue related to diversity and to identifying the peers that review in federal agencies that support education research: how education practitioners and community members should be involved. Because education research is applied and attention to the relevance of the work crucial, it is essential to include practitioners and community members in the work of such agencies. Whether and how they participate on panels, however, is a difficult question. A major concern with the practice of including reviewers without research expertise² on panels is that it could lead to inadequate reviews with respect to technical merit criteria, a critical aspect of research proposal review in all agencies. In addition, since the field of education research is in the early stages of developing scientific norms for peer review, this important process could be complicated or slowed by the participation of individuals who do not have a background in research. We do see the potential benefits of including practitioners and community members on panels evaluating education research funding applications for identifying high-quality proposals and contributing to professional development opportunities for researchers, practitioners, and community members alike. Thus, we conclude that this option is one of four possible strategies—including reviewing proposals alongside researchers, reviewing proposals after researchers’ reviews, serving on priority setting or policy boards, or participating in retrospective reviews of agency portfolios—that agencies could adopt to actively engage practitioner and community members groups in their work.

To promote high-quality education research that is rigorous and relevant, peer review must rest on consistently applied and well-defined quality standards, and these standards should be made clear to applicants and to reviewers. Although the specific criteria by which applications are assessed

in the review process can and should vary by agency, it is important that each develop specific scales and anchors for reviewers to apply to each evaluation criterion to promote a fair and reliable process. Furthermore, it is essential that reviewers be trained in how to apply the review criteria prior to their evaluation of proposals, so all reviewers approach the task with a set of common understandings about evaluative criteria.

Peer reviewers provide important input to agency leaders about what research applications should be funded. However, it is most often the case that staff is authorized to make final funding decisions. Thus, internal decision making should also be driven by quality considerations. Simply going down a list of applications that peer reviewers have ranked in terms of quality and funding them until available dollars are depleted, for example, can lower the quality of the portfolio as a whole if the quality of the proposals drops off at place that is above (or below) the point at which funding runs out.

Finally, risk is an important element of a high-quality education research portfolio: if agencies never support new work that strikes off in a new direction, challenges core ideas, or approaches a problem from a novel perspective, the potential for significant progress, or even breakthroughs, will be substantially curtailed. The value of risk-taking and innovation should be reflected in funding decisions. This can be accomplished in a number of ways: by including “innovation” or a related construct in the evaluation criteria for the reviews, by allowing agency decision makers to fund applications “out of order” to support riskier proposals, or by retaining a funding mechanism outside the peer review process that is designed to support highly innovative work.

Peer review in federal agencies is rarely designed with the intent of using the tool to enhance the capacity of the field to support a culture of rigorous inquiry. We view this objective as crucial and recommend that peer review systems be designed to support its attainment explicitly. Practices that can support this goal include the targeted inclusion of panelists from a range of scholarly perspectives and traditionally underrepresented groups in education research (note that this strategy promotes this professional development goal as well as the goal of identifying and supporting high-quality research), the use of standing panels, the con-

sistent provision of comprehensive feedback to applicants, the development of agency staff, and extending opportunities for training and professional development among applicants, reviewers, and staff.

The act of deliberating on the merits of research proposals can be a powerful learning opportunity for everyone involved. To encourage the use of peer review as a tool for raising the capacity of the field as a whole, agencies should expand their pool of reviewers by reaching out to groups traditionally underrepresented in education research. Similarly, finding ways to support the participation of junior scholars in the peer review process can be an effective mechanism for mentoring the next generation of education researchers.

Standing panels in which scholars from a range of disciplines and perspectives meet regularly to review research in a given area can offer a rich setting for integrating knowledge across domains and fostering ongoing learning opportunities for participants. They also offer the kind of stability and institutional knowledge that can facilitate positive outcomes when investigators resubmit an application previously reviewed under their auspices. Standing panels do have their drawbacks, however; they can institutionalize bias and narrow the kinds of expertise that can and should be brought to bear on peer review deliberations. Their use should be designed to offset such weaknesses, through the use of such practices as staggered terms for members.

The professional development of applicants—both successful and unsuccessful—can be enhanced through the provision of consistent, comprehensive feedback on their proposals. In this way, the considerations and perspectives of the range of expertise and experiences represented in the peer review process can be communicated to applicants, enhancing the likelihood that their future work will be strengthened. Similarly, interactions between agency staff and prospective applicants can serve as a communication channel between these two key groups in education research.

Since agency staff are important actors in the education research community generally and in the peer review process specifically, their role requires careful consideration. Authorizing staff to be substantively involved in all aspects of research competitions—from writing requests for proposals to running review panels—can draw on and strengthen their expertise and provide useful continuity throughout the process. However, it can also raise thorny questions about fairness. If the same staff work with potential applicants, select reviewers, and play important roles in final decisions about

funding, it can create at least the perception that researchers who know agency staff will have an unfair advantage over their less connected peers. Federal agencies that support education research should balance the provision of professional development opportunities for staff and the effective use of their expertise in the process against the need to ensure fair reviews and a system that is viewed as legitimate among a range of stakeholders.

Finally, agencies can enhance the use of peer review as a tool for expanding the capacity of the field by providing training to those involved in the peer review process and offering professional development opportunities for staff and a range of field researchers. Agencies should share this responsibility for making high-quality training widely available with relevant scientific and professional associations.

Effective organizational practices and strong infrastructures are essential to any well-functioning peer review system, including those supported by federal agencies that fund education research. Organizational excellence should be supported by a focus on evaluation, and agencies should consistently analyze the extent to which their practices support stated goals. In addition, effective management requires planning and organization in advance of a review. Scheduling reviews with appropriate amounts of lead time in federal agencies that support education research has been hampered in the past by unpredictable timing of appropriations and widely fluctuating levels of funding. Internal barriers within agencies that slow down program announcements, make peer review difficult to schedule, and result in complicated or burdensome logistics should be minimized. Similarly, legislative mandates that prescribe the details of peer review systems should be minimized, as they can hinder the development of quality systems and impede progress.

An agency infrastructure built to support peer review systems should include, at a minimum, knowledgeable staff, systems for managing the logistics of peer review, and the strategic use of information technologies to support review and discussion of proposals. In particular, we suggest agencies invest in the development of databases that house detailed information on past and prospective reviewers to facilitate the identification of high-quality peers.

Peer review has been held up as a quality standard for the conduct and use of education research. It is clear that understanding the complexities and trade-offs associated with this tool is required for the standard to be applied consistently and well. We offer this brief treatment to encourage its use as a tool for promoting important objectives in the improvement of education research and to provide a framework for research policy makers charged with overseeing peer review systems designed to assess proposals for education research funding. Despite its flaws, peer review is a system worth preserving and improving. It can be a powerful driver for the improvement of education research and the field—provided that those charged with overseeing the processes understand the strengths and weaknesses of various approaches and implement them with clarity of purpose.