Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 1
1
What Is a Randomized Field Trial?
P eople behave in widely varying ways, due to many different causes,
including their own individual volition (conscious choices). Social
scientists often seek to understand whether or not a specific inter-
vention may have an influence on human behavior or performance. For
example, a researcher might want to examine the effect of a driver safety
course on teenage automobile accidents or the effect of a new reading pro-
gram on student achievement. But there are many forces that might cause a
change in driving or reading skills, so how can the investigator be confident
that it was the intervention that made the difference? An effective way to
isolate the effect of a specific factor on human behavior and performance is
to conduct a randomized field trial, which is a research method used to
estimate the effect of an intervention on a particular outcome of interest.
As a first step, investigators hypothesize that a particular intervention
or "treatment" will cause a change in behavior. Then they seek to test the
hypothesis by comparing the average outcome for individuals in the group
who were randomly assigned to receive this intervention with the average
outcome for individuals in the group who do not. This method helps
social scientists to attribute changes in the outcome of interest (e.g., read-
ing achievement) to the specific intervention (e.g., the reading program),
rather than to the many other possible causes of human behavior and
performance.
1
OCR for page 2
2 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION
MAJOR FEATURES
In this section, we sketch the defining features of randomized field
trials. In particular, we focus on the two key concepts of randomization and
control and then briefly situate randomized field trials within the broader
context of establishing cause-and-effect relationships.
A research design is randomized when individuals (or schools or other
units of study) are put into an "experimental" group (which receives the
intervention) or a "control"1 group (which does not) on the basis of a
random process like the toss of a coin.2 The power of this random assign-
ment is that, on average, the two groups that result are initially the same,
differing only in terms of the intervention.3 This allows researchers to more
confidently attribute differences they observe between the two groups to
the intervention, rather than to the myriad other factors that influence
human behavior and performance. As in any comparative study, research-
ers must be careful to observe and account for any other confounding vari-
ables that could differentially affect the groups after randomization has
taken place. That is, even though randomization creates (statistically)
equivalent groups at the outset, once the intervention is under way, other
events or programs could take place in one group and not the other, under-
mining any attempt to isolate the effect of the intervention.
Randomized field trials are also controlled; that is, the investigator
controls the process by which individuals (or other entities of study) are
assigned to receive the intervention of interest. If the assignment of indi-
viduals or entities is outside the investigator's control, then it is generally
1A control group is a comparison group in a randomized field trial that acts as a contrast
to the group receiving the intervention of interest. In randomized field trials involving hu-
mans, research participants in the control group typically either continue to receive existing
services or receive a different intervention.
2Tossing a coin is a useful way of explaining the situation in which the participants have
a 50-50 chance of being assigned to either of two groups: the experimental or the control
group. Randomized field trials can have more than two groups; as long as the assignment
process is conducted on the basis of a statistical process that has known probabilities (0.5 or
otherwise), the groups will be balanced on observable and unobservable characteristics.
3It is logically possible that differences between the groups may still be due to idiosyn-
cratic differences between individuals assigned to receive the intervention or to be part of the
control group. However, with randomization, the chances of this occurring (a) can be explic-
itly calculated and (b) can be made very small, typically by a straightforward manipulation
like increasing the number of individuals assigned to each group.
OCR for page 3
WHAT IS A RANDOMIZED FIELD TRIAL? 3
much more difficult to attribute observed outcomes to the intervention
being studied. For example, if teachers assigned some students to experi-
ence a novel teaching method and some to a comparison group that did not
experience it based on their judgment of which students should experience
the method, then other factors (such as student aptitude) may confound or
obscure the specific effect of the novel teaching method on student learn-
ing outcomes.4
Thus, randomization and control are the foundation of a systematic
and rigorous process that enables researchers estimating the effect of an
intervention to be more confident in the internal validity of their results--
that is, that differences in outcomes can be attributed to the presence or
absence of the intervention, rather than to some other factor. External va-
lidity--the extent to which findings of effectiveness (or lack of effective-
ness) hold in other times, places, and populations--can be established only
when the intervention has been subjected to rigorous study across a variety
of settings.
The ultimate aim of randomized field trials is to help establish cause-
and-effect relationships. They cannot, however, uncover all of the multiple
causes that may affect human behavior. Instead, randomized field trials are
designed to isolate the effect of one or more possible treatments that may
or may not be the cause(s) of an observed behavioral outcome (such as an
increase in student test scores) (Campbell, 1957). Furthermore, a single
study--no matter how strong the design--is rarely sufficient to establish
causation. Indeed, establishing a causal relationship is a matter of some
complexity. In short, it requires that a coherent theory predict the specific
relationship among the program, outcome, and context and that the re-
sults from several studies in varying circumstances are consistent with that
prediction.
A few final clarifications about terminology are in order. Some observ-
ers consider the term "randomized field trial" to be limited only to very
4In some cases, an investigator may conduct a randomized field trial when an interven-
tion is allocated to individuals based on a random lottery. As discussed in Chapter 3, some
school districts have used randomized lotteries to allocate school vouchers, in order to equita-
bly distribute scarce resources when demand exceeds available funding for vouchers. In these
cases, the investigator typically does not directly control the random assignment process, but
as long as the process is truly random, the statistically equivalent groups that result isolate the
relationship between group membership (treatment or control) and outcome from confound-
ing influences and the essential features of a randomized field trial are retained.
OCR for page 4
4 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION
large medical studies or studies conducted by pharmaceutical companies
when testing the safety and efficacy of new drugs. Randomized designs,
however, can be part of any research in any field aimed at estimating the
effect of an intervention, regardless of the size of the study. In this report,
we use the term "randomized field trial" to refer to studies that test the
effectiveness of social interventions comparing experimental and control
groups that have been created through random assignment. Although most
of the workshop discussions focused on large-scale randomized field trials,
the key elements for education research do not involve the size of the study,
but the focus on questions of causation, use of randomization, and the
construction of control groups that do not receive the intervention of inter-
est. Indeed, even small "pilot" studies can use randomization and control
groups to determine the feasibility of scaling an intervention.
CURRENT DEBATES AND TRENDS
At the workshop, University of Pennsylvania professor Robert Boruch
described how randomized field trials have been used in a range of fields
over time. Since World War II, he explained, randomized field trials have
been used to test the effectiveness of the Salk polio vaccine and the antibi-
otic streptomycin, and these designs are now considered the "gold stan-
dard" for testing the effects of different interventions in many fields. Boruch
went on to describe the growing use of randomized field trials to evaluate
social programs since the 1970s (Boruch, de Moya, and Snyder, 2002) and
noted that the World Bank, the government of the United Kingdom, the
Campbell Collaboration, and the Rockefeller Foundation, all held confer-
ences promoting the use of randomized field trials during 2002 and 2003.
Trends in other fields notwithstanding, scholars of education have long
debated the utility of this design in education research. Those who ques-
tion its usefulness frequently argue that the model of causation that under-
lies these designs is too simplistic to capture the complexity of teaching and
learning in diverse educational settings (e.g., Cronbach et al., 1980; Bruner,
1996; Willinsky, 2001; Berliner, 2002). Others, in contrast, are enthusias-
tic about using randomized field trials for addressing causal questions in
education, emphasizing the unique ability of the design to isolate the im-
pact of interventions on a specified outcome in an unbiased fashion (e.g.,
Cook and Payne, 2002; Mosteller and Boruch, 2002; Slavin, 2002).
In the past five years, as calls for evidence-based education have be-
come common, these debates have intensified and expanded beyond aca-
OCR for page 5
WHAT IS A RANDOMIZED FIELD TRIAL? 5
demic circles to include policy makers and practitioners. Most visibly, the
No Child Left Behind Act, passed by Congress in 2001 and signed by the
President in 2002, includes many references to "scientifically based" educa-
tional programs and services. The law defines scientifically based research
as including research that "is evaluated using experimental or quasi-experi-
mental designs in which individuals, entities, programs or activities are as-
signed to different conditions and with appropriate controls to evaluate the
effects of the condition of interest, with a preference for random-assign-
ment experiments, or other designs to the extent that those designs contain
within-condition or across-condition controls."
Furthermore, in its strategic plan for 2002-2007, the U.S. Department
of Education has established as its chief goal to "create a culture of achieve-
ment" by, among other steps, encouraging the use of "scientifically based
methods in federal education programs." The strategic plan also aims to
"transform education into an evidence-based field" (U.S. Department of
Education, 2002, pp. 14-15). The Institute of Education Sciences, the De-
partment of Educations' primary research arm, has established the What
Works Clearinghouse to help reach these goals by identifying "interven-
tions or approaches in education that have a demonstrated beneficial causal
relationship to important student outcomes" (What Works Clearinghouse
[2003], can be found at http://www.w-w-c.org/july2003.html). The expert
technical advisory group guiding the clearinghouse has established quality
standards to review available research on such critical education problems
as improving early reading and reducing high school dropout rates. These
standards place high priority on randomized field trials, which are seen as
"among the most appropriate research designs for identifying the impact or
effect of an educational program or practice" (What Works Clearinghouse
[2003], can be found at http://www.w-w-c.org/july2003.html). They also
acknowledge that there are circumstances in which they are not feasible,
suggesting that quasi experiments (which are comparative studies that
attempt to isolate the effect of an intervention by means other than random-
ization) may be useful under such circumstances.5
5In a quasi-experimental study, researchers may compare naturally existing groups that
appear similar except for the intervention being studied. In this research design, investigators
often use statistical techniques to attempt to adjust for known confounding variables that are
associated with both the intervention and the outcome of interest, thus invoking additional
assumptions about the causal effects of the intervention. While these statistical techniques
can address known differences between study groups, they may inadequately adjust unknown
OCR for page 6
6 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION
The National Research Council report Scientific Research in Education
(National Research Council, 2002) was designed to help clarify the nature
of scientific inquiry in education in this rapidly changing policy context.
That report links design to the research question and, for addressing causal
questions (i.e., "what works") about specified outcomes, highlights ran-
domized field trials as the most appropriate research designs when they are
feasible and ethical. This report summarizes a workshop in which partici-
pants addressed the question: When randomized field trials are conducted
in social settings like schools and school districts, how can they be imple-
mented and what procedures should be used in implementation?
confounding variables. The major drawback of quasi-experimental designs is the possibility
that the groups are systematically different (a problem known as "selection bias"), and thus
investigators may be less confident about conclusions reached using these methods (National
Research Council, 2002, p. 113). In contrast, randomization theoretically creates groups that
are not systematically influenced by both known and unknown confounding variables.
Representative terms from entire chapter:
field trials
