Click for next page ( 28


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 27
4 How Can Randomized Field Trials Be Conducted in an Educational Setting? T he researchers and practitioners who spoke at the workshop offered several practical lessons for planning and conducting randomized field trials in education. At the conclusion of the workshop, committee member Kay Dickersin summarized these lessons, suggesting that success in conducting a randomized field trial in an educa- tional setting involves four interdependent steps: (1) develop a true partnership with the school community, (2) ensure the internal validity of the study design and implementation, (3) focus on recruiting sufficient numbers of study participants, and (4) plan for future implementation of an intervention if the study shows it is effective. DEVELOPING A PARTNERSHIP By far, the single strongest message that emerged from the dialogue during the workshop is that developing and nurturing a true partnership between the research team and the relevant education communities is criti- cally important to success in carrying out any research in schools, including randomized field trials. In each of the three studies featured at the work- shop, researchers were able to gain entry to the schools, to ensure coopera- tion in faithfully carrying out the interventions, and to make progress to- ward mutual goals only by establishing trust and encouraging open communication. Their experiences suggest that it is nearly impossible for 27

OCR for page 27
28 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION researchers to conduct randomized field trials in schools unless both re- searchers and education service providers take time to understand each oth- ers' goals and develop a study design that will help both parties to reach them. It is very difficult for researchers to walk into any type of school setting cold turkey and say we would like to engage in a randomized field trial study. Olatokunbo (Toks) Fashola, Johns Hopkins University All three studies showed the value of partnering (see Boxes 3-1, 3-2, and 3-3). For example, the Baltimore After-School Program Study was fa- cilitated by the existing relationship between Fashola (the chief researcher) and key staff members of the Child First Authority (CFA) after-school pro- gram where the study was conducted. The Power4Kids study is based on a formal partnership between Myers and the research team and the Allegh- eny Intermediate Unit, a group that works with a consortium of school districts outside Pittsburgh that is participating in the study. The Baltimore Whole-Day First-Grade Program Study demonstrates most clearly the value of taking the time to identify the education community's goals and interests. In their presentation, Kellam and Chinnia described how their partnership helped both the education community and the research team meet their goals. Kellam asserted that when a part- nership is in place based on "mutual self-interests at multiple levels," then "consent sounds like a silly word"--illustrating how key implementation tasks such as recruitment are facilitated by the relationship. Chinnia de- scribed some of the "self-interests" that led to the long-term partnership between the Baltimore City Schools and the American Institutes for Re- search. She explained that randomized field trials helped to meet several of the school system's goals, including intervening early in elementary school to enhance and maintain student achievement, identifying best practices for instruction and classroom management, and promoting parent involve- ment in students' progress. She noted that the current study could help to sustain best practices in a whole-day first-grade program, and that the goal of creating and sustaining whole-day first-grade programs is included in the Baltimore City Public School System's master plan.

OCR for page 27
HOW CAN RANDOMIZED FIELD TRIALS BE CONDUCTED? 29 In order to do this type of research, especially randomized field trials, it's important that we have very strong partner- ships. Not only do we have partnerships within the school system but also with the community at large. Some of this [partnership building] has taken two to three years of plan- ning. Linda Chinnia, Baltimore City Public School System Workshop speakers also suggested that in large, heavily minority urban school districts, researchers must be sensitive to issues of race and power when seeking to develop such partnerships. For example, Jones said that, partly because of the lack of diversity in the research community, some minorities do not trust researchers because they feel that "poor and minor- ity groups are the most evaluated and researched populations while they have no input into the process." Lewis expressed similar views, saying, "We are difficult to work with, yes we are, but many of us have been burned, so we have reason to be difficult." As Dickersin put it in her closing remarks, creating "culturally competent" research teams who have experience work- ing with urban schools is critically important to the success of the research. Some workshop speakers argued that creating racially and ethnically diverse research teams can be an important step toward enhancing cultural competence, building trust, and developing partnerships. Lewis suggested that, to accomplish this goal, research organizations could, for example, find competent black researchers through the American Educational Re- search Associations' special interest groups focusing on urban and minority education (American Educational Research Association, 2003). In addi- tion, she proposed that these more diverse research teams collaborate closely with researchers employed by urban school districts. She said that several large urban school districts (including Houston and Atlanta) have outstand- ing research staff who would welcome the opportunity to collaborate in randomized field trials. Jones echoed these sentiments, suggesting that such efforts can help to break down historical power dynamics between research- ers (often white) and students and education professionals in urban school districts (often nonwhite), which might otherwise pose barriers to estab- lishing mutual trust. Questioning the capacity of the current cadre of re- searchers to develop partnerships with inner-city schools in particular, Lewis

OCR for page 27
30 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION argued that researchers are frequently unfamiliar with inner-city schools and unsure of how to work with them. Many of the researchers are not skilled in working with people in urban centers. Sharon Lewis, Council of the Great City Schools Finally, establishing close partnerships can help schools not only to identify but also to implement scientifically based best practices. These best practices may in turn help schools make the adequate yearly progress re- quired by the No Child Left Behind Act. Developing and nurturing this partnership between the researchers and professionals working in the study sites facilitates all other implementation tasks--including the three de- scribed next. ENSURING INTERNAL VALIDITY Dickersin also pointed out that several workshop speakers emphasized the need to spend time in the early design and planning phases to ensure internal validity of the study. Indeed, in debating the merits of randomized field trials in education, many scholars have focused on the trade-off that is necessary between internal validity--that is, the extent to which it can be concluded that the treatment led to the effect, or difference, between one group and another on a particular outcome--and external validity--the extent to which the findings of a particular study hold in other times, places, and populations. For example, critics argue that the strict protocols of the studies, which are required to maximize internal validity (e.g., program options are consistently and comprehensively implemented in both the ex- perimental and control groups), do not reflect typical school operations, and thus the usefulness of the results for real life is questionable (Cronbach et al., 1980; National Research Council, 2002). Several workshop speakers and audience members raised questions about a particularly important aspect of comparative studies, including ran- domized field trials, in ensuring internally valid results: how to measure and account for the implementation of the experimental and control treat- ments. Gueron argued that it is important for investigators to monitor and to understand how the interventions are being applied during the study to

OCR for page 27
HOW CAN RANDOMIZED FIELD TRIALS BE CONDUCTED? 31 assist with the interpretation of the comparison of outcomes between the groups. She cited the collection of "equivalent and high-quality data" on process and contextual factors in both groups as essential to the success of a randomized field trial. Shavelson, too, urged researchers to spend the time and money required to observe and characterize implementation, since how a program plays out in schools can often be quite different from what was planned or expected at the outset of the study. Kellam, too, argued that it is extremely important to measure implementation in the control group "with the same intensity" as in the experimental group. Two speakers offered specific strategies, based on early planning, that helped to ensure that the intervention is carried out faithfully and that the randomized field trial would yield internally valid results. In the Baltimore After-School Program Study, Fashola arranged for the CFA to hire new non-first-grade teachers who were not a part of the regular CFA program to provide one-on-one tutoring to the experimental group of first graders. Using new teachers helped to ensure that the existing CFA did not provide similar tutoring to students in the control group. In the Power4Kids study in the Pittsburgh area, Myers described a similar approach. He asked each participating school to nominate one teacher to provide the remedial read- ing interventions, replacing these teachers in their usual positions with long- term substitutes for the entire school year. These teachers were given spe- cific training and materials to help ensure that they carry out the alternative reading programs as designed. In this way, Myers argued that they are un- likely to "accidentally" provide the remedial reading tutorials to students not in their experimental groups, because they will not be acting as regular classroom teachers. Researchers employing randomized field trials can also use observational methods to detect factors that might contaminate the results of the study. For example, in the Comer study described by Kelly, many ethnographers were hired to study and characterize program imple- mentation, a major factor in the researchers' ability to identify threats to internal validity. RECRUITING SUFFICIENT NUMBERS OF STUDY PARTICIPANTS As summarized in Chapter 3, Dickersin reiterated that recruiting study participants is critical to ensure sufficient sample sizes and to be able to draw valid and reliable conclusions from a randomized field trial in turn. The implementation of this step, too, depends in large part on the broader

OCR for page 27
32 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION partnership between researchers and educators. According to several work- shop speakers, success in recruiting study participants can be aided by key intermediaries. At the highest level, a board including school, community, and research officials--such as the Baltimore City Community and Institu- tional Board, which oversees the long-term research program there--can help to both oversee and develop support for the study. When many school districts are involved in a large-scale study, an intermediary such as the Allegheny Intermediate Unit led by Durno, who has strong working rela- tionships with the schools, can help to forge partnerships. At the level of the individual school, a designated site coordinator can help to communi- cate with teachers and parents and ensure that random selection and ex- perimental and control processes flow smoothly, and that differences be- tween the experimental and control groups are preserved over the course of the study. In describing the use of such a strategy, Durno put it this way, "the on-site coordinator must be an excellent communicator who can meet the school on that school's terms." Chinnia trained a community organizer, who was already trusted by parents, to help explain the study and win the support of students and parents. Now in a full-time position as a "family classroom partnership aide," the coordinator has won consent among 100 percent of parents when recruiting students to participate in the most re- cent generation of randomized field trials. One of the challenges that we experienced in working with the schools is the difference in the climate and the culture of each school. Donna Durno, Allegheny Intermediate Unit PLANNING FOR IMPLEMENTATION When educators consider the possibility of participating in a random- ized field trial, workshop speakers suggested that the pivotal factor is often whether the study is likely to lead to an actual improvement in their own schools. Researchers can help educators reach this objective by designing the study to support future implementation of an intervention if it has been shown as effective and, after the study is completed, by working with the school and funders to provide the effective intervention to all students. The Baltimore Whole-Day First-Grade Program Study plans for and sup-

OCR for page 27
HOW CAN RANDOMIZED FIELD TRIALS BE CONDUCTED? 33 ports future implementation of the intervention by including analysis of the cost-effectiveness of the whole-day program, in light of its potential to reduce the future costs associated with drug abuse and school failure. In all three of the studies described at the workshop, research teams provided materials and training to administrators and teachers, at no cost to the schools. Workshop speakers describing these studies suggested that investigators wishing to conduct a successful randomized field trial should consider whether they have adequate funding to provide such services. With adequate funding, researchers can form a research partnership that will help schools reach their goals, which include implementing (not just studying) educational interventions if they are found to be successful. Providing a proven program to all students following the study ensures that the study results are used and useful, not only by policy makers, but also by class- room teachers. LOOKING AHEAD Taken as a whole, the workshop presentations and discussions sug- gested that the outlook for continuing and expanding randomized field trials in schools, based on strong community-research partnerships, is good. In her concluding remarks, Dickersin said that one of the most important things she had learned from the workshop was that some educators were "very positive about participating" in randomized field trials "if the barriers . . . can be overcome." She referred to an earlier session, in which she had asked education representatives how they would respond if she proposed to conduct a randomized field trial on an issue important to their schools, with adequate resources and a culturally sensitive research team. In response, Lewis said, "I would jump at it," while Jones said that her students (study- ing to become teachers and education researchers) would also want to be partners in the research. The three studies featured at the workshop also indicate that when designed to help meet their goals, educators and school officials would welcome the opportunity to participate in, and help carry out, randomized field trials. In presenting each case, representatives of the education com- munities described why they viewed participating in such studies as benefi- cial for their schools. Their enthusiasm at the workshop appeared similar to what Gueron said she had encountered among state and local welfare offi- cials who have willingly participated in repeated randomized field trials over the years, because "they actually believe it makes a difference and . . . can bring visibility and resources to their community."