Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
2 Why Are Randomized Field Trials Used? W orkshop speakers suggested that investigators choose a random- ized field trial research design to answer important questions about the effects of social programs or services and to obtain credible results that are likely to be used by policy makers. ANSWERING IMPORTANT QUESTIONS In his opening remarks, Stanford University professor of education Richard J. Shavelson, who chaired the committee that produced Scientific Research in Education (National Research Council, 2002), emphasized that researchers use study designs appropriate to answer "the important ques- tions." Summarizing the main findings from Scientific Research in Educa- tion, Shavelson argued that important questions can be divided into three classes: "What is happening [in education]?" "Is there a systematic effect [e.g., of an educational program]?" and "Why or how is it happening?" The first set of questions can be answered best using descriptive research meth- ods. Descriptive research methods can help researchers and policy makers to identify and describe particular education problems (e.g., dropout rates) and may also aid in designing interventions, he said. Once an intervention has been proposed, a randomized field trial is often the best method to help researchers understand whether the intervention has the intended (causal) 7
8 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION effect on an educational outcome of interest, in Shavelsons' view. Through this approach, investigators may be able not only to establish a systematic effect of the intervention, but also to reliably estimate the magnitude of the effect. To answer the third set of important research questions in education-- "How or why is it [the effect] happening?"--Shavelson suggested that re- searchers combine several methods, including randomized field trials, quasi- experimental designs, and descriptive techniques. For example, classroom observations, interviews, and ethnographic studies in California (Borhnstedt and Stecher, 1999) complemented randomized field trials in Tennessee (Achilles, 1999) to identify teacher behavior as the key factor in explaining why smaller class sizes improved student achievement. You ought to design a study to answer the question that you think is the important question, not create the question to fit the design. Richard J. Shavelson, Stanford University Later in the day, three studies that feature randomized field trials in educational settings were described in detail. In each case, investigators followed the approach outlined by Shavelson, using previous descriptive and quasi-experimental research to identify and articulate important re- search questions, and choosing methods appropriate to answer their ques- tions. Described in detail in Chapter 3, all three studies build on previous research. The designers of all three studies chose a randomized field trial to assess whether various interventions targeted to elementary school students had a systematic effect on specific academic and behavioral outcomes. YIELDING USEFUL RESULTS Workshop speakers observed that researchers choose a randomized field trial design not only because it can answer one class of important questions, but also because of its capacity for generating valid and reliable results that are trusted and used by policy makers. Judith Gueron, president of MDRC, a large nonprofit research corporation specializing in randomized field tri- als, made this case convincingly. Over the past 30 years, she said, her com-
WHY ARE RANDOMIZED FIELD TRIALS USED? 9 pany has conducted 35 to 40 large-scale randomized field trials involving about 400,000 people in 250 locations around the United States, and "we've never had a serious challenge to their credibility." With alternative research methods, she said, "this is much less true." With randomized field trials, you can more confidently sepa- rate fact from advocacy, and with alternative [designs] this is much less true. Judith Gueron, MDRC Acknowledging that "people dont' wake up in the morning wanting to be in a randomized field trial," Gueron explained that MDRC has built a constituency over time by persuading participants that these methods are essential to improve policy. She said that strong statements about the valid- ity of findings from randomized field trials by prestigious groups, including National Research Council committees (e.g., National Research Council, 1985, 1994), have helped to convince policy makers of the credibility of their findings. This credibility, in turn, has helped to ensure that these findings are translated into laws and programs. For example, when Con- gress amended the Social Security Act in 1988 (the Family Support Act of 1988, P.L. 100-485), it continued to allow states to waive provisions of the Aid to Families with Dependent Children law in order to test new ap- proaches to welfare reform, but the waivers were available only if states assessed these new approaches. From the early 1980s through 1996, under both Republican and Democratic administrations, the U.S. Department of Health and Human Services interpreted this law as requiring states to con- duct randomized field trials of the new approaches (Gueron, 1997). Out- side observers agree that the studies conducted by MDRC have had a strong impact on welfare policy and practice, particularly on the Family Support Act of 1988 (e.g., Baum, 1991; Haskins, 1991). Although most workshop speakers agreed that randomized field trials can potentially yield valid and reliable results, George Mason University professor Anthony (Eamonn) Kelly raised the most pointed questions about their viability in educational settings and thus their ultimate utility for im- proving policy and practice. Kelly argued that researchers face significant barriers translating the "ideal" design of a randomized field trial into a real-
10 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION life education study.1 He illustrated these real-world "threats to internal and external validity" by describing problems he identified in a randomized field trial used to evaluate the Comer School Development Program in Chicago schools (Cook, Hunt, and Murphy, 2000). The Comer program aims to improve student achievement by improving the social climate in the school. In this study, researchers randomly assigned schools to the Comer program or to a control group. Kelly reported that due to high turnover of school principals, all elements of the Comer program were not carried out faithfully in the experimental group, which reduced the validity of the results. More generally, he argued that results from randomized field trials must be implemented and "diffused," and the field knows little about the factors that guide successful implementation. Kelly also suggested that it is often difficult to generalize the findings from randomized field trials because local factors may differ significantly from those of the schools in the study. There are randomized field trials as intended, and there are randomized field trials as carried out. You're working with a system that's in flux. Anthony (Eamonn) Kelly, George Mason University Gueron, however, argued that designing randomized field trials in- volves trade-offs between internal and external validity that must be made in light of the goals of the study as well as other considerations, such as the state of knowledge in a field. For education, Gueron argued that since the use of randomized field trials is in its early stages, the first challenge is to show that such studies can be successfully conducted. In that context, she urged researchers to give priority to generating internally valid results, ar- 1Kelly sketched some supplementary methods that could inform the design of ran- domized field trials as well as implementation and diffusion studies. Describing what is typically referred to as "design research," he argued that studying, understanding, and im- proving educational practice must be framed "in terms of exploration and prototyping" until the operative factors and variables in the complex reality of schooling are better understood. He cited a recent special issue of Educational Researcher (Kelly, 2003) as providing further elaboration and criticism of these ideas.
WHY ARE RANDOMIZED FIELD TRIALS USED? 11 guing that "it's better to learn something with confidence versus push for external validity and not understand at the end what you've got." In response to a question at the workshop, Gueron described the dis- advantages of not using this design. During the 1970s, she said, Congress passed a law guaranteeing jobs for young people in selected poor communi- ties, if the young people would stay in--and make progress in--school. To carry out the law, the U.S. Department of Labor saturated the selected communities with funds to create jobs and MDRC tracked the young peoples' participation in school and work. When a National Research Coun- cil committee (1985) later reviewed the MDRC study, it agreed with the study's conclusion that the program produced a dramatic increase in work while the job guarantee was operating. However, committee members ques- tioned the conclusion that it also produced a longer term increase in em- ployment rates, asking whether more young people were working because of outside variables, such as changes in local labor markets in Baltimore, Cleveland, and other cities. In retrospect, Gueron said, she wished that the study team had used random assignment in some communities to yield stronger conclusions about the long-term effectiveness of the program.
