Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
1 What Is a Randomized Field Trial? P eople behave in widely varying ways, due to many different causes, including their own individual volition (conscious choices). Social scientists often seek to understand whether or not a specific inter- vention may have an influence on human behavior or performance. For example, a researcher might want to examine the effect of a driver safety course on teenage automobile accidents or the effect of a new reading pro- gram on student achievement. But there are many forces that might cause a change in driving or reading skills, so how can the investigator be confident that it was the intervention that made the difference? An effective way to isolate the effect of a specific factor on human behavior and performance is to conduct a randomized field trial, which is a research method used to estimate the effect of an intervention on a particular outcome of interest. As a first step, investigators hypothesize that a particular intervention or "treatment" will cause a change in behavior. Then they seek to test the hypothesis by comparing the average outcome for individuals in the group who were randomly assigned to receive this intervention with the average outcome for individuals in the group who do not. This method helps social scientists to attribute changes in the outcome of interest (e.g., read- ing achievement) to the specific intervention (e.g., the reading program), rather than to the many other possible causes of human behavior and performance. 1
2 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION MAJOR FEATURES In this section, we sketch the defining features of randomized field trials. In particular, we focus on the two key concepts of randomization and control and then briefly situate randomized field trials within the broader context of establishing cause-and-effect relationships. A research design is randomized when individuals (or schools or other units of study) are put into an "experimental" group (which receives the intervention) or a "control"1 group (which does not) on the basis of a random process like the toss of a coin.2 The power of this random assign- ment is that, on average, the two groups that result are initially the same, differing only in terms of the intervention.3 This allows researchers to more confidently attribute differences they observe between the two groups to the intervention, rather than to the myriad other factors that influence human behavior and performance. As in any comparative study, research- ers must be careful to observe and account for any other confounding vari- ables that could differentially affect the groups after randomization has taken place. That is, even though randomization creates (statistically) equivalent groups at the outset, once the intervention is under way, other events or programs could take place in one group and not the other, under- mining any attempt to isolate the effect of the intervention. Randomized field trials are also controlled; that is, the investigator controls the process by which individuals (or other entities of study) are assigned to receive the intervention of interest. If the assignment of indi- viduals or entities is outside the investigator's control, then it is generally 1A control group is a comparison group in a randomized field trial that acts as a contrast to the group receiving the intervention of interest. In randomized field trials involving hu- mans, research participants in the control group typically either continue to receive existing services or receive a different intervention. 2Tossing a coin is a useful way of explaining the situation in which the participants have a 50-50 chance of being assigned to either of two groups: the experimental or the control group. Randomized field trials can have more than two groups; as long as the assignment process is conducted on the basis of a statistical process that has known probabilities (0.5 or otherwise), the groups will be balanced on observable and unobservable characteristics. 3It is logically possible that differences between the groups may still be due to idiosyn- cratic differences between individuals assigned to receive the intervention or to be part of the control group. However, with randomization, the chances of this occurring (a) can be explic- itly calculated and (b) can be made very small, typically by a straightforward manipulation like increasing the number of individuals assigned to each group.
WHAT IS A RANDOMIZED FIELD TRIAL? 3 much more difficult to attribute observed outcomes to the intervention being studied. For example, if teachers assigned some students to experi- ence a novel teaching method and some to a comparison group that did not experience it based on their judgment of which students should experience the method, then other factors (such as student aptitude) may confound or obscure the specific effect of the novel teaching method on student learn- ing outcomes.4 Thus, randomization and control are the foundation of a systematic and rigorous process that enables researchers estimating the effect of an intervention to be more confident in the internal validity of their results-- that is, that differences in outcomes can be attributed to the presence or absence of the intervention, rather than to some other factor. External va- lidity--the extent to which findings of effectiveness (or lack of effective- ness) hold in other times, places, and populations--can be established only when the intervention has been subjected to rigorous study across a variety of settings. The ultimate aim of randomized field trials is to help establish cause- and-effect relationships. They cannot, however, uncover all of the multiple causes that may affect human behavior. Instead, randomized field trials are designed to isolate the effect of one or more possible treatments that may or may not be the cause(s) of an observed behavioral outcome (such as an increase in student test scores) (Campbell, 1957). Furthermore, a single study--no matter how strong the design--is rarely sufficient to establish causation. Indeed, establishing a causal relationship is a matter of some complexity. In short, it requires that a coherent theory predict the specific relationship among the program, outcome, and context and that the re- sults from several studies in varying circumstances are consistent with that prediction. A few final clarifications about terminology are in order. Some observ- ers consider the term "randomized field trial" to be limited only to very 4In some cases, an investigator may conduct a randomized field trial when an interven- tion is allocated to individuals based on a random lottery. As discussed in Chapter 3, some school districts have used randomized lotteries to allocate school vouchers, in order to equita- bly distribute scarce resources when demand exceeds available funding for vouchers. In these cases, the investigator typically does not directly control the random assignment process, but as long as the process is truly random, the statistically equivalent groups that result isolate the relationship between group membership (treatment or control) and outcome from confound- ing influences and the essential features of a randomized field trial are retained.
4 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION large medical studies or studies conducted by pharmaceutical companies when testing the safety and efficacy of new drugs. Randomized designs, however, can be part of any research in any field aimed at estimating the effect of an intervention, regardless of the size of the study. In this report, we use the term "randomized field trial" to refer to studies that test the effectiveness of social interventions comparing experimental and control groups that have been created through random assignment. Although most of the workshop discussions focused on large-scale randomized field trials, the key elements for education research do not involve the size of the study, but the focus on questions of causation, use of randomization, and the construction of control groups that do not receive the intervention of inter- est. Indeed, even small "pilot" studies can use randomization and control groups to determine the feasibility of scaling an intervention. CURRENT DEBATES AND TRENDS At the workshop, University of Pennsylvania professor Robert Boruch described how randomized field trials have been used in a range of fields over time. Since World War II, he explained, randomized field trials have been used to test the effectiveness of the Salk polio vaccine and the antibi- otic streptomycin, and these designs are now considered the "gold stan- dard" for testing the effects of different interventions in many fields. Boruch went on to describe the growing use of randomized field trials to evaluate social programs since the 1970s (Boruch, de Moya, and Snyder, 2002) and noted that the World Bank, the government of the United Kingdom, the Campbell Collaboration, and the Rockefeller Foundation, all held confer- ences promoting the use of randomized field trials during 2002 and 2003. Trends in other fields notwithstanding, scholars of education have long debated the utility of this design in education research. Those who ques- tion its usefulness frequently argue that the model of causation that under- lies these designs is too simplistic to capture the complexity of teaching and learning in diverse educational settings (e.g., Cronbach et al., 1980; Bruner, 1996; Willinsky, 2001; Berliner, 2002). Others, in contrast, are enthusias- tic about using randomized field trials for addressing causal questions in education, emphasizing the unique ability of the design to isolate the im- pact of interventions on a specified outcome in an unbiased fashion (e.g., Cook and Payne, 2002; Mosteller and Boruch, 2002; Slavin, 2002). In the past five years, as calls for evidence-based education have be- come common, these debates have intensified and expanded beyond aca-
WHAT IS A RANDOMIZED FIELD TRIAL? 5 demic circles to include policy makers and practitioners. Most visibly, the No Child Left Behind Act, passed by Congress in 2001 and signed by the President in 2002, includes many references to "scientifically based" educa- tional programs and services. The law defines scientifically based research as including research that "is evaluated using experimental or quasi-experi- mental designs in which individuals, entities, programs or activities are as- signed to different conditions and with appropriate controls to evaluate the effects of the condition of interest, with a preference for random-assign- ment experiments, or other designs to the extent that those designs contain within-condition or across-condition controls." Furthermore, in its strategic plan for 2002-2007, the U.S. Department of Education has established as its chief goal to "create a culture of achieve- ment" by, among other steps, encouraging the use of "scientifically based methods in federal education programs." The strategic plan also aims to "transform education into an evidence-based field" (U.S. Department of Education, 2002, pp. 14-15). The Institute of Education Sciences, the De- partment of Educations' primary research arm, has established the What Works Clearinghouse to help reach these goals by identifying "interven- tions or approaches in education that have a demonstrated beneficial causal relationship to important student outcomes" (What Works Clearinghouse [2003], can be found at http://www.w-w-c.org/july2003.html). The expert technical advisory group guiding the clearinghouse has established quality standards to review available research on such critical education problems as improving early reading and reducing high school dropout rates. These standards place high priority on randomized field trials, which are seen as "among the most appropriate research designs for identifying the impact or effect of an educational program or practice" (What Works Clearinghouse [2003], can be found at http://www.w-w-c.org/july2003.html). They also acknowledge that there are circumstances in which they are not feasible, suggesting that quasi experiments (which are comparative studies that attempt to isolate the effect of an intervention by means other than random- ization) may be useful under such circumstances.5 5In a quasi-experimental study, researchers may compare naturally existing groups that appear similar except for the intervention being studied. In this research design, investigators often use statistical techniques to attempt to adjust for known confounding variables that are associated with both the intervention and the outcome of interest, thus invoking additional assumptions about the causal effects of the intervention. While these statistical techniques can address known differences between study groups, they may inadequately adjust unknown
6 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION The National Research Council report Scientific Research in Education (National Research Council, 2002) was designed to help clarify the nature of scientific inquiry in education in this rapidly changing policy context. That report links design to the research question and, for addressing causal questions (i.e., "what works") about specified outcomes, highlights ran- domized field trials as the most appropriate research designs when they are feasible and ethical. This report summarizes a workshop in which partici- pants addressed the question: When randomized field trials are conducted in social settings like schools and school districts, how can they be imple- mented and what procedures should be used in implementation? confounding variables. The major drawback of quasi-experimental designs is the possibility that the groups are systematically different (a problem known as "selection bias"), and thus investigators may be less confident about conclusions reached using these methods (National Research Council, 2002, p. 113). In contrast, randomization theoretically creates groups that are not systematically influenced by both known and unknown confounding variables.
