National Academies Press: OpenBook

Implementing Randomized Field Trials in Education: Report of a Workshop (2004)

Chapter: 4 How Can Randomized Field Trials be Conducted in an Educational Setting?

« Previous: 3 When are Randomized Field Trials Feasible?
Suggested Citation:"4 How Can Randomized Field Trials be Conducted in an Educational Setting?." National Research Council. 2004. Implementing Randomized Field Trials in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10943.
×
Page 27
Suggested Citation:"4 How Can Randomized Field Trials be Conducted in an Educational Setting?." National Research Council. 2004. Implementing Randomized Field Trials in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10943.
×
Page 28
Suggested Citation:"4 How Can Randomized Field Trials be Conducted in an Educational Setting?." National Research Council. 2004. Implementing Randomized Field Trials in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10943.
×
Page 29
Suggested Citation:"4 How Can Randomized Field Trials be Conducted in an Educational Setting?." National Research Council. 2004. Implementing Randomized Field Trials in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10943.
×
Page 30
Suggested Citation:"4 How Can Randomized Field Trials be Conducted in an Educational Setting?." National Research Council. 2004. Implementing Randomized Field Trials in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10943.
×
Page 31
Suggested Citation:"4 How Can Randomized Field Trials be Conducted in an Educational Setting?." National Research Council. 2004. Implementing Randomized Field Trials in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10943.
×
Page 32
Suggested Citation:"4 How Can Randomized Field Trials be Conducted in an Educational Setting?." National Research Council. 2004. Implementing Randomized Field Trials in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10943.
×
Page 33

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

4 How Can Randomized Field Trials Be Conducted in an Educational Setting? T he researchers and practitioners who spoke at the workshop offered several practical lessons for planning and conducting randomized field trials in education. At the conclusion of the workshop, committee member Kay Dickersin summarized these lessons, suggesting that success in conducting a randomized field trial in an educa- tional setting involves four interdependent steps: (1) develop a true partnership with the school community, (2) ensure the internal validity of the study design and implementation, (3) focus on recruiting sufficient numbers of study participants, and (4) plan for future implementation of an intervention if the study shows it is effective. DEVELOPING A PARTNERSHIP By far, the single strongest message that emerged from the dialogue during the workshop is that developing and nurturing a true partnership between the research team and the relevant education communities is criti- cally important to success in carrying out any research in schools, including randomized field trials. In each of the three studies featured at the work- shop, researchers were able to gain entry to the schools, to ensure coopera- tion in faithfully carrying out the interventions, and to make progress to- ward mutual goals only by establishing trust and encouraging open communication. Their experiences suggest that it is nearly impossible for 27

28 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION researchers to conduct randomized field trials in schools unless both re- searchers and education service providers take time to understand each oth- ers' goals and develop a study design that will help both parties to reach them. It is very difficult for researchers to walk into any type of school setting cold turkey and say we would like to engage in a randomized field trial study. ­Olatokunbo (Toks) Fashola, Johns Hopkins University All three studies showed the value of partnering (see Boxes 3-1, 3-2, and 3-3). For example, the Baltimore After-School Program Study was fa- cilitated by the existing relationship between Fashola (the chief researcher) and key staff members of the Child First Authority (CFA) after-school pro- gram where the study was conducted. The Power4Kids study is based on a formal partnership between Myers and the research team and the Allegh- eny Intermediate Unit, a group that works with a consortium of school districts outside Pittsburgh that is participating in the study. The Baltimore Whole-Day First-Grade Program Study demonstrates most clearly the value of taking the time to identify the education community's goals and interests. In their presentation, Kellam and Chinnia described how their partnership helped both the education community and the research team meet their goals. Kellam asserted that when a part- nership is in place based on "mutual self-interests at multiple levels," then "consent sounds like a silly word"--illustrating how key implementation tasks such as recruitment are facilitated by the relationship. Chinnia de- scribed some of the "self-interests" that led to the long-term partnership between the Baltimore City Schools and the American Institutes for Re- search. She explained that randomized field trials helped to meet several of the school system's goals, including intervening early in elementary school to enhance and maintain student achievement, identifying best practices for instruction and classroom management, and promoting parent involve- ment in students' progress. She noted that the current study could help to sustain best practices in a whole-day first-grade program, and that the goal of creating and sustaining whole-day first-grade programs is included in the Baltimore City Public School System's master plan.

HOW CAN RANDOMIZED FIELD TRIALS BE CONDUCTED? 29 In order to do this type of research, especially randomized field trials, it's important that we have very strong partner- ships. Not only do we have partnerships within the school system but also with the community at large. Some of this [partnership building] has taken two to three years of plan- ning. ­Linda Chinnia, Baltimore City Public School System Workshop speakers also suggested that in large, heavily minority urban school districts, researchers must be sensitive to issues of race and power when seeking to develop such partnerships. For example, Jones said that, partly because of the lack of diversity in the research community, some minorities do not trust researchers because they feel that "poor and minor- ity groups are the most evaluated and researched populations while they have no input into the process." Lewis expressed similar views, saying, "We are difficult to work with, yes we are, but many of us have been burned, so we have reason to be difficult." As Dickersin put it in her closing remarks, creating "culturally competent" research teams who have experience work- ing with urban schools is critically important to the success of the research. Some workshop speakers argued that creating racially and ethnically diverse research teams can be an important step toward enhancing cultural competence, building trust, and developing partnerships. Lewis suggested that, to accomplish this goal, research organizations could, for example, find competent black researchers through the American Educational Re- search Associations' special interest groups focusing on urban and minority education (American Educational Research Association, 2003). In addi- tion, she proposed that these more diverse research teams collaborate closely with researchers employed by urban school districts. She said that several large urban school districts (including Houston and Atlanta) have outstand- ing research staff who would welcome the opportunity to collaborate in randomized field trials. Jones echoed these sentiments, suggesting that such efforts can help to break down historical power dynamics between research- ers (often white) and students and education professionals in urban school districts (often nonwhite), which might otherwise pose barriers to estab- lishing mutual trust. Questioning the capacity of the current cadre of re- searchers to develop partnerships with inner-city schools in particular, Lewis

30 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION argued that researchers are frequently unfamiliar with inner-city schools and unsure of how to work with them. Many of the researchers are not skilled in working with people in urban centers. ­Sharon Lewis, Council of the Great City Schools Finally, establishing close partnerships can help schools not only to identify but also to implement scientifically based best practices. These best practices may in turn help schools make the adequate yearly progress re- quired by the No Child Left Behind Act. Developing and nurturing this partnership between the researchers and professionals working in the study sites facilitates all other implementation tasks--including the three de- scribed next. ENSURING INTERNAL VALIDITY Dickersin also pointed out that several workshop speakers emphasized the need to spend time in the early design and planning phases to ensure internal validity of the study. Indeed, in debating the merits of randomized field trials in education, many scholars have focused on the trade-off that is necessary between internal validity--that is, the extent to which it can be concluded that the treatment led to the effect, or difference, between one group and another on a particular outcome--and external validity--the extent to which the findings of a particular study hold in other times, places, and populations. For example, critics argue that the strict protocols of the studies, which are required to maximize internal validity (e.g., program options are consistently and comprehensively implemented in both the ex- perimental and control groups), do not reflect typical school operations, and thus the usefulness of the results for real life is questionable (Cronbach et al., 1980; National Research Council, 2002). Several workshop speakers and audience members raised questions about a particularly important aspect of comparative studies, including ran- domized field trials, in ensuring internally valid results: how to measure and account for the implementation of the experimental and control treat- ments. Gueron argued that it is important for investigators to monitor and to understand how the interventions are being applied during the study to

HOW CAN RANDOMIZED FIELD TRIALS BE CONDUCTED? 31 assist with the interpretation of the comparison of outcomes between the groups. She cited the collection of "equivalent and high-quality data" on process and contextual factors in both groups as essential to the success of a randomized field trial. Shavelson, too, urged researchers to spend the time and money required to observe and characterize implementation, since how a program plays out in schools can often be quite different from what was planned or expected at the outset of the study. Kellam, too, argued that it is extremely important to measure implementation in the control group "with the same intensity" as in the experimental group. Two speakers offered specific strategies, based on early planning, that helped to ensure that the intervention is carried out faithfully and that the randomized field trial would yield internally valid results. In the Baltimore After-School Program Study, Fashola arranged for the CFA to hire new non-first-grade teachers who were not a part of the regular CFA program to provide one-on-one tutoring to the experimental group of first graders. Using new teachers helped to ensure that the existing CFA did not provide similar tutoring to students in the control group. In the Power4Kids study in the Pittsburgh area, Myers described a similar approach. He asked each participating school to nominate one teacher to provide the remedial read- ing interventions, replacing these teachers in their usual positions with long- term substitutes for the entire school year. These teachers were given spe- cific training and materials to help ensure that they carry out the alternative reading programs as designed. In this way, Myers argued that they are un- likely to "accidentally" provide the remedial reading tutorials to students not in their experimental groups, because they will not be acting as regular classroom teachers. Researchers employing randomized field trials can also use observational methods to detect factors that might contaminate the results of the study. For example, in the Comer study described by Kelly, many ethnographers were hired to study and characterize program imple- mentation, a major factor in the researchers' ability to identify threats to internal validity. RECRUITING SUFFICIENT NUMBERS OF STUDY PARTICIPANTS As summarized in Chapter 3, Dickersin reiterated that recruiting study participants is critical to ensure sufficient sample sizes and to be able to draw valid and reliable conclusions from a randomized field trial in turn. The implementation of this step, too, depends in large part on the broader

32 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION partnership between researchers and educators. According to several work- shop speakers, success in recruiting study participants can be aided by key intermediaries. At the highest level, a board including school, community, and research officials--such as the Baltimore City Community and Institu- tional Board, which oversees the long-term research program there--can help to both oversee and develop support for the study. When many school districts are involved in a large-scale study, an intermediary such as the Allegheny Intermediate Unit led by Durno, who has strong working rela- tionships with the schools, can help to forge partnerships. At the level of the individual school, a designated site coordinator can help to communi- cate with teachers and parents and ensure that random selection and ex- perimental and control processes flow smoothly, and that differences be- tween the experimental and control groups are preserved over the course of the study. In describing the use of such a strategy, Durno put it this way, "the on-site coordinator must be an excellent communicator who can meet the school on that school's terms." Chinnia trained a community organizer, who was already trusted by parents, to help explain the study and win the support of students and parents. Now in a full-time position as a "family classroom partnership aide," the coordinator has won consent among 100 percent of parents when recruiting students to participate in the most re- cent generation of randomized field trials. One of the challenges that we experienced in working with the schools is the difference in the climate and the culture of each school. ­Donna Durno, Allegheny Intermediate Unit PLANNING FOR IMPLEMENTATION When educators consider the possibility of participating in a random- ized field trial, workshop speakers suggested that the pivotal factor is often whether the study is likely to lead to an actual improvement in their own schools. Researchers can help educators reach this objective by designing the study to support future implementation of an intervention if it has been shown as effective and, after the study is completed, by working with the school and funders to provide the effective intervention to all students. The Baltimore Whole-Day First-Grade Program Study plans for and sup-

HOW CAN RANDOMIZED FIELD TRIALS BE CONDUCTED? 33 ports future implementation of the intervention by including analysis of the cost-effectiveness of the whole-day program, in light of its potential to reduce the future costs associated with drug abuse and school failure. In all three of the studies described at the workshop, research teams provided materials and training to administrators and teachers, at no cost to the schools. Workshop speakers describing these studies suggested that investigators wishing to conduct a successful randomized field trial should consider whether they have adequate funding to provide such services. With adequate funding, researchers can form a research partnership that will help schools reach their goals, which include implementing (not just studying) educational interventions if they are found to be successful. Providing a proven program to all students following the study ensures that the study results are used and useful, not only by policy makers, but also by class- room teachers. LOOKING AHEAD Taken as a whole, the workshop presentations and discussions sug- gested that the outlook for continuing and expanding randomized field trials in schools, based on strong community-research partnerships, is good. In her concluding remarks, Dickersin said that one of the most important things she had learned from the workshop was that some educators were "very positive about participating" in randomized field trials "if the barriers . . . can be overcome." She referred to an earlier session, in which she had asked education representatives how they would respond if she proposed to conduct a randomized field trial on an issue important to their schools, with adequate resources and a culturally sensitive research team. In response, Lewis said, "I would jump at it," while Jones said that her students (study- ing to become teachers and education researchers) would also want to be partners in the research. The three studies featured at the workshop also indicate that when designed to help meet their goals, educators and school officials would welcome the opportunity to participate in, and help carry out, randomized field trials. In presenting each case, representatives of the education com- munities described why they viewed participating in such studies as benefi- cial for their schools. Their enthusiasm at the workshop appeared similar to what Gueron said she had encountered among state and local welfare offi- cials who have willingly participated in repeated randomized field trials over the years, because "they actually believe it makes a difference and . . . can bring visibility and resources to their community."

Next: References »
Implementing Randomized Field Trials in Education: Report of a Workshop Get This Book
×
Buy Paperback | $29.00 Buy Ebook | $23.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The central idea of evidence-based education-that education policy and practice ought to be fashioned based on what is known from rigorous research-offers a compelling way to approach reform efforts. Recent federal trends reflect a growing enthusiasm for such change. Most visibly, the 2002 No Child Left Behind Act requires that "scientifically based [education] research" drive the use of federal education funds at the state and local levels. This emphasis is also reflected in a number of government and nongovernment initiatives across the country. As consensus builds around the goals of evidence-based education, consideration of what it will take to make it a reality becomes the crucial next step. In this context, the Center for Education of the National Research Council (NRC) has undertaken a series of activities to address issues related to the quality of scientific education research. In 2002, the NRC released Scientific Research in Education (National Research Council, 2002), a report designed to articulate the nature of scientific education research and to guide efforts aimed at improving its quality. Building on this work, the Committee on Research in Education was convened to advance an improved understanding of a scientific approach to addressing education problems; to engage the field of education research in action-oriented dialogue about how to further the accumulation of scientific knowledge; and to coordinate, support, and promote cross-fertilization among NRC efforts in education research. The main locus of activity undertaken to meet these objectives was a year-long series of workshops. This report is a summary of the third workshop in the series, on the implementation and implications of randomized field trials in education.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!