Advancing Scientific Research in Education (2005) / Chapter Skim
Currently Skimming:
2 Promoting Quality
2 Promoting Quality
Pages 19-35
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 19... ...
It is beyond the scope of this report to provide a fair and comptehensive description of the many important issues that have been raised in recent years with respect to how to define quality in scientific education research, or to comment on how the committee views them. Rather, in this chapter we begin with a brief discussion of how we define quality, taking our cue from Scientific Research in Education, and provide illustrations of select elements of quality that emerged in the committee's workshops.
|
From page 20... ...
, widh a particular emphasis on randomized field trials to help establish cause-andeffect relationships (see, e.g., U.S. Department of Education, 2002,2004; What Works Clearinghouse, 2004)
|
From page 21... ...
· Pose significant questions that can be investigated empirically A key idea embedded in this principle is that research questions should address important issues of practice, policy, and processes. During the peer review workshop, for example, participants highlighted the importance of ensuring that diverse groups of stakeholders be involved in developing federal agencies' research agendas, prioritizing research questions, and conducting the actual research.
|
From page 22... ...
. The diverse and diffuse nature of the investigators in the field of education research make common standards elusive; however, the workshops highlighted three leverage points for actively promoting high-quality research: peer review processes within federal agencies, implementation of research designs in educational settings, and partnerships between education research and practitioners.
|
From page 23... ...
The training of reviewers should focus deeply on the criteria used to evaluate research by defining those criteria very clearly, and training people to use them reliably. If reviewers do not have a clear understanding of what the criteria are, they carry their own frame of reference as a defining point into the review process, resulting in lower reliability of peer review, whether for manuscripts submitted to professional journals or for research grant proposals submitted for funding (Cichetti, 2003)
|
From page 24... ...
At the workshop, Teresa Levitin of the National institute on Drug Abuse provided several useful ideas for how to illustrate key concepts to renewers about the renew criteria in a relatively short amount of time (see Box 2-1)
|
From page 25... ...
Workshop: Key Speaker: Related Product: Peer Review of Education Research G rant Applications Implications, Considerations, and Future Directions February 25-26, 2003 Transcript available at: htEp:// www7. nationalacadem ies.org/core/ Teresa Levitin, National institute on Drug Abuse Strengthening Peer Review in Federal Agencies That Support Education Research hop ://books.
|
From page 26... ...
In addition, peer review panels shoudd be composed so as to minimize conflicts of interest, to badance biases, and to promote the participation of people Mom a range of scholarly perspectives and traditionally underrepresented groups. Deciding who counts as a peer is central to quality considerations: the peer review process, no matter how well designed, is only as good as the people involved.
|
From page 27... ...
If, for example, a specialized quantitative tesearch design is being proposed, at least some of the reviewers should have expertise in this design; if a specialized qualitative research design is proposed, some reviewers should have expertise in this design. In addition, it is the range of proposal types and program priorities, not their frequency or conventionality, that should determine the scope of the panel's experience and expertise.
|
From page 28... ...
Both a e serious limitations of peer renew and can probably be addressed in the long term only by expanding the pools of qualified reviewers, through training and outreach to experts traditionally underreptesented in the process. In assembling peer review panels, attention to the diversity of potential reviewers with respect to disciplinary orientation as well as sociad background characteristics also is important to promote quality.
|
From page 29... ...
As described above, a critical scientific principle is The idea That The Choice of methods used in particular studies should be driven by The nature of The question being investigated. This notion was extended in The workshop on The conduct of one medhod—randomized field trials in educational settings to focus attention on the importance of rigorous implementation of research methods in educational settings.
|
From page 30... ...
Pairs of researcher-practitioner teams described Their experiences designing and conducting randomized field trials in schools in Baltimore and suburban Pittsburgh and made clear dhat The selection of This medhod is not sufficient to ensuring dhat a rigorous study is conducted—implementation matters. The challenges They described are daunting.
|
From page 31... ...
Recommendation 4: Federad agencies shoudd ensure appropriate resources ate available for education researchers conductmg largescade investigations in educationad settungs to build partnerships with practitioners and police makers. As we have argued above, a key lesson That emerged from The workshop on implementing randomized field trials in education is That The quality of any large-scale research conducted in districts or schools—largely independent of medhod or design—depends significantly on relationships built between researchers and district- and school-based personnel.
|
From page 32... ...
In each of dhe dhree studies featured at dhe workshop, researchers were able to gain access to dhe schools, to ensure cooperation in faidhfully carrying out dhe interventions, and to make prog ess toward mutual goals by establishing trust and encouraging open communication. Their experiences suggest dhat it is nearly impossible for researchers to conduct randomized field trials—or any odher large-scale study—in districts and schools unless bodh researchers and education service providers take time to understand each odhers' goals and develop a study design dhat will help bodh parties to teach dhem.
|
From page 33... ...
Creating These partnerships requires time and money. To implement dhe model for dhe series of large-scale randomized field trials described in Box 2-2, for example, the researcher-practitioner team estimate dhe need for a year of work Genre dhe research is formally launched.
|
From page 34... ...
In their presentation, Kellam and Chinnia described how their partnenship helped both the education community and the research team meet their goals. Kellam asserted that when a partnership is in place based on "mutual self-interests at multiple levels," obtaining the consent of the parents of participating children requires far less logistical work than othenwise might be the case—illustrating how key implementation tasks such as recruitment are facilitated by the relationship.
|
From page 35... ...
Workshop: Key Speakers: Related Product: Randomized Field Trials in Education: Implementation and implications September 24, 2003 Transcript available at: hop // www7. nationalacademies.org/core/ Sheppard Kellam, American institutes for Research Linda Chinnia, Baltimore City Public School System Implementing Randomized Field Trials in Education: Report of a Workshop http://books.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.