Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 12
3 When Are Randomized Field Trials Feasible? A s summarized in the previous chapter, workshop speakers suggested that investigators choose research designs using randomized field trials because they can help answer important questions about the systematic nature of effects and yield credible, useful findings. Since there have been very few randomized field trials conducted in educational set- tings to date, however, the field as a whole is in the early stages of learning how to conduct them. In each of the three studies described in detail at the workshop, a re- searcher-practitioner team described how they implemented randomized field trials in educational settings. Boxes 3-1, 3-2, and 3-3 briefly summa- rize their experiences, illustrating the different approaches to implementing the same underlying logic of randomized field trials in urban school envi- ronments. The lessons they learned about what it takes to successfully con- duct these studies are highlighted thematically in this and the final chapter. Many of the challenges are not particularly unique [to imple- menting randomized field trials]. ­Judith Gueron, MDRC As the discussion that follows makes clear, there are a number of prag- matic issues that must be addressed. Challenges include meeting ethical and legal standards, recruiting and randomly assigning students or schools 12

OCR for page 12
WHEN ARE RANDOMIZED FIELD TRIALS FEASIBLE? 13 or both, understanding the local educational context, and having adequate resources. Although Gueron pointed out that most of these challenges are not unique to randomized field trials, they all affect the extent to which investigators and educators can successfully implement randomized field trials in schools. MEETING ETHICAL AND LEGAL STANDARDS As described in Chapter 1, in a randomized field trial researchers at- tempt to create groups that are (statistically) equivalent except for the inter- vention or interventions assigned. For an investigator focusing on the reli- ability and validity of the results, this enables comparisons between the two groups that can support inferences about the effect of the controlled factors on any observed differences in specified outcomes. From an educator's per- spective, however, any research (including randomized field trials) that in- volves controlled assignment to an intervention requires them to relinquish power in determining what type of education their students (or classrooms or schools) experience. Some workshop speakers expressed ethical concerns about controlled assignment, while others suggested ways to address it. For example, Howard University dean of education Vinetta C. Jones said that educators in underserved schools with large numbers of minority students often believe (whether correctly or not) that randomized field trials "involve denying beneficial services or interventions to some students." In her remarks, Jones acknowledged that such perceptions may or may not be true, but nonetheless urged researchers that these concerns were real and must be taken seriously. To avoid "poisoning the environment for future work" by failing to meet ethical standards, Gueron warned researchers: "Do not deny entitle- ments. Do not reduce service levels." Sharon Lewis, research director for the Council of the Great City Schools (a coalition of 60 of the largest urban school systems in the United States), agreed that researchers should not reduce service levels. She said that large urban school districts support ran- domized field trials when they are used to test program variations but op- pose them when they involve excluding students from promising interven- tions in order to create control groups. However, committee member Jack Fletcher, in moderating the presentations and discussion of the three stud- ies earlier in the day, argued that if interventions have not been subject to rigorous scrutiny, it is impossible to know whether the services are benefi- cial, have no effect, or may even be harmful.

OCR for page 12
14 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION In his presentation of the Power4Kids study (see Box 3-2), David Myers, vice president of Mathematica Policy Research, noted that this study was designed to partially address the concern that students may not receive a promising intervention. Each of the participating schools has been as- signed to test one of four tutorials, so the study design does not exclude any school from these interventions. Nevertheless, in each participating school, some students will receive the tutorials, while a control group will continue to receive conventional instruction. Furthermore, Gueron argued that a randomized field trial can provide a fair and objective way to simultaneously allocate the new service (since the process of random assignment ensures that everyone has an equal chance of being selected) and investigate its impact. For example, some cities (e.g., New York) have instituted the use of random lotteries to allocate limited school voucher funding to interested parents when demand outstrips fund- ing levels. The use of these lotteries enables investigators to conduct a ran- domized field trial by comparing the outcomes of students who received vouchers to attend private schools with those students who applied but did not win the lottery and continued to attend public schools. Workshop speakers emphasized that it is imperative for researchers to meet high ethical and legal standards today in order to overcome negative perceptions that have their roots in the history of social research. For example, Gueron said that some people have referred to MDRC researchers conducting randomized field trials of welfare programs as "perpetrators of Tuskegee," and Myers said that he and others at Mathematica have also been accused of "doing something like [the] Tuskegee" study. Their com- ments refer to the well-known Tuskegee Syphilis Study. For four decades, beginning in 1932, the U.S. Public Health Service carried out a longitudi- nal study of the natural history of syphilis among black men in the Tuskegee, Alabama, area. In that study, the researchers withheld treatment from 399 men with late-stage syphilis as well as 201 men free of the disease in order to study its progression (Reverby, 1998). The men in both groups were told they were participating in an experiment and receiving treatment. Although the Tuskegee study was a natural history study and not a randomized field trial, the fact that investigators withheld treatment that was known to be effective from the sick men has influenced the public's response to many forms of research, including randomized field trials in school systems. As discussed further below (see Chapter 4), most workshop speakers agreed that it is possible to overcome these negative views by forg-

OCR for page 12
WHEN ARE RANDOMIZED FIELD TRIALS FEASIBLE? 15 ing the respectful partnerships necessary to carry out successful and ethical research in a school setting. Gueron argued that researchers conduct a randomized field trial only when ethical and legal standards can be met in both the design and imple- mentation of the study. For example, when randomly assigning individuals or families in a school-based study, she said, ethical and legal standards require that investigators inform parents of the research and obtain their consent. She also emphasized that investigators must take steps to ensure that individual data and identifying information are kept confidential. Indeed, partly in response to public concern about the Tuskegee study, the federal government now regulates research involving human participants under the "common rule" (45 Code of Federal Regulations, Part 36.102i) (National Research Council, 2003). Under these regulations, universities and other research organizations have established institutional review boards to ensure that researchers provide human participants with full informa- tion about the potential risks and benefits of participating in certain kinds of research and obtain their informed consent. Because education research often involves human participants, the need to meet ethical and legal standards (including the standards imposed by institutional review boards) applies not only to randomized field trials but also to other types of educa- tion research. ESTABLISHING ADEQUATE SAMPLE SIZE AND RECRUITING PARTICIPANTS Several workshop participants highlighted the importance of the plan for randomization and analysis in randomized field trials. Two related con- cepts were discussed: the sample size (that is, the number of participants) and the unit of randomization (that is, what entity or entities are assigned to the experimental and control groups--student, classroom, school, or some combination). Gueron made the general point that for studies de- signed to address policy questions, the sample size must be large enough to detect policy-relevant effects of the intervention. In the presentations of each of the specific studies described at the workshop, sample size was an important topic of discussion.1 Later in the day, Jones suggested that en- suring adequate sample sizes in urban settings may be difficult due to the 1A technique called power analysis can help determine the sample size necessary to detect effects of varying sizes in a particular study.

OCR for page 12
16 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION high mobility rates in central cities. Indeed, in at least one of the studies featured at the workshop, this was a problem: Olatokunbo (Toks) Fashola, research scientist at the Johns Hopkins University, described the specific problems she encountered recruiting enough participants into the Balti- more After-School Program Study and the small sample size that resulted (see Box 3-1). A related issue is what unit is randomized. As Shavelson argued in his talk, all aspects of design depend heavily on the particular question that is posed. However, workshop discussions made clear that there are important nuances and constraints that influence the choice of unit of randomization in conducting randomized field trials in education. In a line of questioning related to the Baltimore After-School Program Study (in which 70 students were initially randomized) and the Power4Kids study (in which 52 schools and 772 students in those schools were initially randomized), a member of the audience observed that focusing on students as the unit of randomiza- tion may well be preferable from a cost perspective (for example, it is easier and cheaper to collect adequate data on 40 students than it is to collect adequate data on 40 schools). He raised concerns, however, about potential drawbacks. As he described it, the basic problem is that such a design typi- cally requires the researcher to make the (often unrealistic) assumption that the effect of teachers on students is the same across different classrooms, leading to questionable conclusions.2 Once the plan for randomization and analysis has been established, the next step of the process is recruiting the participants. When required, the process of obtaining informed consent of participants in randomized field trials (and other research) in educational settings involves both technical and political challenges.3 As described in Box 3-1, the first case study pre- 2This issue is a common methodological challenge associated with how to model out- comes in schools that are by their very nature "nested" (students within classrooms, class- rooms within schools, and so on), and the role of "fixed" and "random" effects in multilevel modeling in particular. Regardless of whether the unit of randomization is the school, the student, or other entity, the effect of interventions in such nested environments can be esti- mated if sampling is conceptualized as multilevel. See Bryk and Raudenbush (1992) for a detailed treatment of this issue. 3Some education research is exempt from human subjects regulations because it does not present risk to the participants. As discussed in Scientific Research in Education (National Research Council, 2002): "education research, including evaluation studies, rarely presents any true risk to the participant so long as care is taken to protect identities and that researchers understand and are responsive to the needs of individual participants" (p. 152). Other stud-

OCR for page 12
WHEN ARE RANDOMIZED FIELD TRIALS FEASIBLE? 17 sented at the workshop was a randomized field trial designed to estimate the effect of a one-on-one reading program for first graders who needed remediation and were enrolled in the Child First Authority (CFA) after- school program in Baltimore. First, the research plan was reviewed by the Johns Hopkins Institutional Review Board. Fashola, the principal investi- gator of the study, explained that, because Johns Hopkins "has been in and out of the news" due to concerns about the protection of human research participants, dealing with the institutional review board was hard, taking away time that she had planned to use to implement and study the pro- gram. Next, during the fall of 2002, she obtained approval from the Balti- more City Public School System, which she described as another tedious process. Fashola said that she could not provide informed consent forms to the teachers until she had obtained these approvals from the institutional review board at Johns Hopkins and the school system. Finally, after the 2002-2003 school year had begun, teachers recruited students into the study, distributing consent forms to them to obtain their parents' written consent. Fashola explained that the schools had difficulty obtaining parent signatures, particularly from the parents of first graders most likely in need of the one-on-one reading tutorial, even though she extended the period to sign up for the program. Outside factors (see Box 3- 1) slowed communication with parents. Ultimately, only 50 students (in- cluding experimental and control groups) remained in the study, limiting its ability to detect (statistically significant) effects. As a technical issue, other workshop speakers suggested that such chal- lenges associated with consent might be addressed by allowing ample time to communicate with students and their parents. Donna Durno, executive director of the Allegheny Intermediate Unit, who was in the early stages of the large Power4Kids study at the time of the workshop (see Box 3-2), said that even though their team started meeting with groups of parents and other stakeholders between six and eight months before the study began, the process was rushed. Indeed, Shep Kellam, public health psychiatrist of the American Institutes for Research, and Linda Chinnia, of the Baltimore City Public School System described a partnership that took two to three years to build in advance of the study of the whole-day program in Balti- more (see Box 3-3). ies do not require written informed consent because no risk is involved or bias will be present in terms of who can provide consent. In all these cases, an institutional review board may approve the study under an exempt or expedited category.

OCR for page 12
18 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION BOX 3-1 Baltimore After-School Program Study Olatokunbo (Toks) Fashola, a Johns Hopkins University re- search scientist, and Loretta McClairn, Child First Authority (CFA) program coordinator at Dr. Bernard Harris Elementary School, out- lined a study of an after-school tutorial program. The program was based on reading curricula used in the Success For All school reform model as implemented in the Baltimore City Public School System. The study built on previous research indicating that the reading programs used by Success For All were effective in helping disadvantaged children learn to read as well as research on after- school programs (Fashola and Slavin, 1997; Ross, Smith, and Casey, 1997; Fashola, 2001). Fashola chose a randomized field trial design to answer the question, "What is the effect of a one-to-one tutorial reading pro- gram on the standardized test scores of first grade students in need of remediation attending an after-school program in Baltimore city?" She said she and her colleagues at the Johns Hopkins University Center for Research on the Education of Students Placed at Risk have advocated for randomized field trials as a way to provide sci- entifically based evidence of program effectiveness. Fashola noted that it was timely to study an after-school program, because the Baltimore City Public School System master plan had established the goal of increasing academic achievement by implementing aca- demically oriented after-school programs. Noting that it is very difficult for researchers to enter a school "cold turkey," Fashola said she focused on first graders who were enrolled in the CFA after-school program with which she had a pro- fessional relationship. Fashola told the audience that, when describ- ing the study to the CFA director, she explained that the funding was not adequate to provide the one-on-one tutorials to all CFA students, but that the study would pay to hire and train teachers to deliver the tutorials to an experimental group and to provide the schools (at no cost) with tutorial materials they could keep beyond the time of the study. The executive director, principals, and pro- gram coordinators welcomed the study as a way to help achieve the city school system's goal of providing academically oriented after-school programs. McClairn explained that CFA offers academic enrichment, cul- tural enrichment, and homework help to about 170 students in eight Baltimore schools, from 2:30 pm to 5:00 pm, four days a week. Although CFA has many first-grade teachers in place (to keep a low

OCR for page 12
WHEN ARE RANDOMIZED FIELD TRIALS FEASIBLE? 19 student-teacher ratio), McClairn said, these teachers didn't feel that they were in competition with the additional teachers hired with study funds to deliver the tutorials "because we're teammates." The study was conducted during the 2002-2003 school year in four schools. Due to the unexpectedly long process of obtaining required approvals from Johns Hopkins University and the Balti- more City Public School System, the CFA teachers did not begin recruiting students to participate until after the school year began. Other problems, including a fatal fire in a CFA family's home that killed four CFA students and one parent volunteer, a change in the Maryland governorship that temporarily closed the CFA after-school program, and snow days in the severe winter of 2002-2003, also hurt recruitment. Ultimately, sample sizes in the four participating schools ranged from 8 to 16, and the total number of students en- rolled in the study from beginning to end was 70, although due to attrition the final sample size was 50. In the fall of 2002, participating students were pretested using three subtests of the Woodcock Reading Mastery Test (letter identi- fication, word identification, and word attack). They were then ran- domly assigned according to school into either an experimental or a control group. From November 2002 until May 2003, students in the experimental group were provided with individual tutoring ses- sions lasting 30 minutes three times per week. Students in the con- trol group had opportunities for academic activities that included homework help, group tutoring, and enrichment programs, but they did not receive the Success For All individual tutoring intervention. As one way to ensure that students in the control group did not receive the tutoring intervention, only teachers in the experimental group were trained. In addition, in order to minimize "transfer" of elements of the program to participants in the control group, the regular school-day first-grade teachers were not allowed to deliver tutorial sessions after school. The specially hired teachers did not interact with students in the control group. At the end of the tutorial period, students in both groups were administered post-tests. Results of the study showed that although all students performed better on the post-tests, and although the experimental group outperformed the control group students on all measures, the differences between the two groups were statisti- cally significant only on the word attack subtest. Funding for the study was provided by the U.S. Department of Education's Office of Educational Research and Improvement, now the Institute of Education Sciences.

OCR for page 12
20 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION BOX 3-2 Power4Kids Study David Myers, vice president of Mathematica Policy Research, and Donna Durno, executive director of the Allegheny Intermediate Unit, described the background and implementation of this ongoing study. Descriptive research indicates that 40 percent of fourth graders in the United States do not read at their grade level. Further research by Florida State University professor Joseph Torgesen indicates that, by the time students reach grades three through five, there is a large gap in reading ability between students who read at their grade level and those who do not. On the basis of that research, Torgesen has called for intensive interventions that could bring students up to grade level, possibly within a single school year. The Power4Kids study builds on this research, Myers said. As a first step, the Haan Foundation for Children convened 15 to 20 pub- lishers of reading tutorials to "show their wares." Following a review, Torgesen and other members of the research team (which includes Mathematica Policy Research and the American Institutes for Research) selected four tutorials for inclusion in the Power4Kids study. In making its selections, the team considered previous research on the effectiveness of the programs, including small ran- domized field trials and quasi-experimental studies. Myers said that the study was designed to address the following questions: · Can children who have reading difficulties in middle to late elementary school acquire adequate reading skills in a short period of time if they are taught with intensity and skill? · Can intensive interventions affect all critical reading skills, such as accuracy, comprehension, and fluency? · Do some children benefit more or less from these intensive and well-implemented reading interventions? Power4Kids includes evaluation of the four selected tutorials in a pullout program, an impact study, a fidelity study, and a cost study. The impact study is currently under way in several suburban school districts near Pittsburgh, all affiliated with the Allegheny Intermedi- ate Unit. In order to assess the effect of the four reading tutorial interventions, Mathematica researchers chose a "scientifically rig-

OCR for page 12
WHEN ARE RANDOMIZED FIELD TRIALS FEASIBLE? 21 orous" randomized field trial design that will include collection of longitudinal data for three years. Durno described the challenges of recruiting over 40 schools to be randomly assigned to one of four different reading interven- tions. She explained that, as executive director of the Allegheny Intermediate Unit, which provides resources, instruction, and edu- cation services to the affected school districts, she had a "credible" relationship with the schools, so "it works out well if we do the re- cruiting." Nevertheless, there have been challenges in implement- ing this large-scale study. Each school district has its own climate and culture, and there are philosophical differences that had to be overcome. In addition, each school district has its own informal power structure and different decision-making processes regarding instructional programming. Ultimate authority could rest with the school board, the superintendent, the school principal, or the cur- riculum coordinator. Through frequent and consistent communica- tion, these challenges have been addressed. Myers explained his strategy for ensuring that each group re- ceives only one of the four interventions. He said he had asked each participating school to nominate one teacher to provide the remedial reading interventions, replacing these teachers with long- term substitutes for the entire school year. These teachers were given specific training and materials, to help ensure that they carry out the alternative reading programs as designed. They are unlikely to "accidentally" provide the remedial reading tutorials to students not in their experimental groups, because they will not be acting as regular classroom teachers during the 2003-2004 school year. Researchers tested third and fifth grade students to identify readers with reading proficiencies below the 20th percentile. Among those eligible to participate, researchers randomly assigned some to receive the form of tutorial assigned to the school, and others to a control group. Those receiving the tutorial will work in small groups of no more than three children with one teacher for one hour each day, receiving a total of about 100 hours of instruction. Since this study is just under way, the results are not yet known. Power4Kids is funded by the U.S. Department of Education. The Haan Foundation for Children helped formulate the idea for the study and brought the partners together.

OCR for page 12
22 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION BOX 3-3 Baltimore Whole-Day First-Grade Program Study Public health psychiatrist Sheppard Kellam of the American Institutes for Research and Linda Chinnia of the Baltimore City Pub- lic School System described this study in the context of the larger, long-term Baltimore Prevention Program. Kellam explained that the program is based on research and theory in child and adolescent development (Kellam and Van Horn, 1997). The goal of the prevention program is to get children off to a good start in school in order to prevent later school failure, sub- stance abuse, and mental and behavioral disorders among teenag- ers (Johns Hopkins University School of Public Health, 2003). The program is based on earlier studies that show how children be- have, learn, and feel about themselves in first grade are good indi- cators of whether they will have problems as teenagers (e.g., Boyd et al., 2003). Previous randomized field trials have assessed the impact of different first-grade interventions designed to reduce these behaviors and learning problems (e.g., Kellam et al., 1998). Kellam said that the current randomized field trial is designed to assess the effects of an integrated set of preventive first-grade interventions. The interventions are directed at improving (1) teach- ers' classroom behavior management, (2) family-classroom part- nerships regarding homework and discipline, and (3) teachers' in- structional practices regarding academic subjects, particularly reading. These interventions have been combined in a single whole- day first-grade classroom program. Other speakers suggested that the challenge of obtaining informed con- sent from large numbers of students requires addressing not only technical but also deeper political issues. For example, Chinnia noted that the study of the whole-day program in Baltimore schools requires "highly intrusive activities" that disrupt normal school and family patterns. These include the investigation of teaching practices and curricula, teachers contacting families about student progress, and classroom behavior management strat- egies that affect peer group relationships. In addition, random assignment of teachers and students is a change in normal school practices. However, she said that "strong institutional and community partnerships" and "shared values and mutual respect" helped sustain support for random assignment and other aspects of the study. Building on Chinnias' remarks, Kellam said,

OCR for page 12
WHEN ARE RANDOMIZED FIELD TRIALS FEASIBLE? 23 In the current study, researchers will assess the effect of the program by randomly assigning children and teachers to interven- tion classrooms or standard program classrooms (control condi- tion) in elementary schools. They will also measure variation in the impact of the program that may be due to variation both in the ways teachers implement the specific elements of the program and in the children themselves. Chinnia explained that the Baltimore City Public School Sys- tem supports this study because it lays the foundation for translat- ing its findings into policy and practice. In addition to assessing the impact, as outlined above, the researchers will follow the first-grade children as far as the end of third grade, and they will also follow their first-grade teachers over two subsequent cohorts of first grad- ers. This long-term observation will allow researchers to test whether the multiple levels of support and training for teachers sus- tain high levels of program practice. The study will also test in the fourth year whether the support and training structure is successful in training nonprogram teachers. Another element of the study-- which has not yet been funded--could potentially be very useful to the schools. If funding is obtained, the researchers will conduct a cost-effectiveness study of the program, comparing program costs with the potential long-term cost savings that would result from re- ductions in drug use, behavior problems, and dropping out during the teenage years. The study is supported by the National Institute on Drug Abuse and other funders. "It's a mistake to think of the IRB (institutional review board) process and the . . . informed consent process separately from the partnership." The importance of such partnerships was a recurring theme in the workshop, and we return to it in more detail in Chapter 4. GROUNDING THE STUDY IN THE RELEVANT EDUCATIONAL CONTEXT Many workshop speakers agreed that a randomized field trial is most appropriate when it is responsive to the current political and economic context of schools. For example, Chinnia suggested that researchers take time to analyze the social and political structure of the school district, learn

OCR for page 12
24 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION the school's vision, and understand its challenges and goals. Because Kellam and his colleagues took this approach, they have formed a partnership with the Baltimore schools that has successfully supported three generations of randomized field trials. The studies are designed to evaluate approaches to improved management and interaction in first-grade classrooms and to as- sess the effect of these approaches in reducing later drug abuse, crime, and academic failure. Some workshop speakers suggested that the No Child Left Behind Act may discourage schools from participating in randomized field trials, even though the law explicitly encourages "scientifically based" education re- search. Wesley Bruce of the Indiana Department of Education said, in try- ing to comply with the goals of the act, teachers and administrators in his state are focusing less on students and more on test scores as measures of schools' performance. To comply with the law, the Indiana Education De- partment has identified the 25 percent of all schools that have failed to make "adequate yearly progress" and has assigned departmental employees to work with these schools. Bruce said that the education department staff will be able to help only the poorest performing of the bottom 25 percent of schools, leaving the top schools in the group with only a list of best practices. Although the What Works Clearinghouse (see Chapter 1) may be helpful in this regard, Bruce questioned whether there was enough time to conduct randomized field trials to identify, learn more about, and imple- ment best practices within the time periods set by the legislation. Referring to the adequate yearly progress requirements, Bruce said, "you dont' have three years, four years to conduct research and get results back to schools about good practice," because every year "the bar has been raised higher for the level of performance." When we look at how you conduct this research [randomized field trials] in schools, researchers need to understand that for every single school the stakes are high and have gotten higher. ­Wesley Bruce, Indiana Department of Education Bruce went on to note several challenges to implementing randomized field trials that paralleled earlier comments: (1) the political reality that schools, school districts, and teachers like local control and may not wel- come federally funded research or researchers (echoing a similar statement

OCR for page 12
WHEN ARE RANDOMIZED FIELD TRIALS FEASIBLE? 25 by Durno); (2) teachers who are satisfied with their own teaching methods may not faithfully implement the educational intervention being studied; (3) "good teachers will share ideas that seem to be working," which may mean that they share ideas from the intervention, which could lead to the control group students receiving intervention strategies; and (4) it may be difficult to provide adequate review to ensure that ethical and legal stan- dards are met. Finally, he said, researchers would need buy-in throughout a school system when conducting a randomized field trial, because superin- tendents change positions frequently. Lewis agreed with Bruce that national requirements for improvement in test scores may discourage school districts from participating in random- ized field trials. She said that such participation can take time away from instruction at a time when No Child Left Behind requires improvement in test scores. SECURING ADEQUATE RESOURCES Gueron suggested that a randomized field trial is more likely to suc- ceed when resources are adequate to address the feasibility criteria and challenges outlined above. Reflecting on Shavelsons' presentation, Gueron agreed that researchers should select research methods appropriate to the questions being asked. She cautioned participants, however, that answer- ing policy-relevant questions about the effectiveness of interventions de- pended in part on having adequate resources. Although questions about whether a widely used educational intervention has a systematic effect may best be answered with a large-scale randomized field trial, she said that such studies "cant' be done on the cheap in terms of resources or time." Noting that a successful randomized field trial requires creativity, flexibil- ity, and "operational and political savvy," Gueron said that funding should be adequate to support the salaries of "very senior people" who possess these abilities. Financial resources are needed to successfully carry out the random assignment process and to gather data on both control and experi- mental groups over "an adequate length of time," Gueron said. When funding is available, she said, it is also useful to replicate a randomized field trial of a promising intervention in several different areas, to test effectiveness in diverse settings. Gueron explained that although large-scale randomized field trials re- quire considerable resources, they may be more cost-effective overall than studies using alternative research designs (e.g., quasi experiments). She

OCR for page 12
26 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION High-quality large-scale field studies with primary data col- lection are very expensive, and actually they're getting more expensive. And this is true for randomized field trials but it is also true for the alternative [designs]. --Judith Gueron, MDRC noted that any large study involving the collection of primary data in the field is expensive, but a randomized field trial requires higher costs in the start-up and operational stages than designs that do not feature randomiza- tion. This cost differential stems from the start-up phase of designing, sell- ing, and initiating the study, which Gueron called the "make it or break it" point for success. In this start-up phase, researchers and educators develop what Kellam described as "the essential partnership." In the operational phase, randomized field trials have some added costs (for policing the imple- mentation of random assignment, which Gueron called an "all or nothing process") but otherwise are comparable to quasi-experimental studies in the resources required to obtain equivalent, high-quality data on both experi- mental and control/comparison groups, as well as to gather information on the process and context for the program. However, she maintained that the later stages, including the analysis of data and diffusion of study results, were less expensive than in other types of large-scale studies of social pro- grams. When looked at from the broader perspective of policy impact per dollar spent, Gueron concluded randomized field trials may be less expen- sive than quasi-experimental research designs that have high "political and financial costs" when they "end in methodological disputes."