|
Items in cart [0] |
Questions? Call 888-624-8373 |
| ||||||||||||
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 7
2
Why Are Randomized Field Trials Used?
W orkshop speakers suggested that investigators choose a random-
ized field trial research design to answer important questions
about the effects of social programs or services and to obtain
credible results that are likely to be used by policy makers.
ANSWERING IMPORTANT QUESTIONS
In his opening remarks, Stanford University professor of education
Richard J. Shavelson, who chaired the committee that produced Scientific
Research in Education (National Research Council, 2002), emphasized that
researchers use study designs appropriate to answer "the important ques-
tions." Summarizing the main findings from Scientific Research in Educa-
tion, Shavelson argued that important questions can be divided into three
classes: "What is happening [in education]?" "Is there a systematic effect
[e.g., of an educational program]?" and "Why or how is it happening?" The
first set of questions can be answered best using descriptive research meth-
ods. Descriptive research methods can help researchers and policy makers
to identify and describe particular education problems (e.g., dropout rates)
and may also aid in designing interventions, he said. Once an intervention
has been proposed, a randomized field trial is often the best method to help
researchers understand whether the intervention has the intended (causal)
7
OCR for page 8
8 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION
effect on an educational outcome of interest, in Shavelsons' view. Through
this approach, investigators may be able not only to establish a systematic
effect of the intervention, but also to reliably estimate the magnitude of the
effect.
To answer the third set of important research questions in education--
"How or why is it [the effect] happening?"--Shavelson suggested that re-
searchers combine several methods, including randomized field trials, quasi-
experimental designs, and descriptive techniques. For example, classroom
observations, interviews, and ethnographic studies in California
(Borhnstedt and Stecher, 1999) complemented randomized field trials in
Tennessee (Achilles, 1999) to identify teacher behavior as the key factor in
explaining why smaller class sizes improved student achievement.
You ought to design a study to answer the question that you
think is the important question, not create the question to fit
the design.
Richard J. Shavelson, Stanford University
Later in the day, three studies that feature randomized field trials in
educational settings were described in detail. In each case, investigators
followed the approach outlined by Shavelson, using previous descriptive
and quasi-experimental research to identify and articulate important re-
search questions, and choosing methods appropriate to answer their ques-
tions. Described in detail in Chapter 3, all three studies build on previous
research. The designers of all three studies chose a randomized field trial to
assess whether various interventions targeted to elementary school students
had a systematic effect on specific academic and behavioral outcomes.
YIELDING USEFUL RESULTS
Workshop speakers observed that researchers choose a randomized field
trial design not only because it can answer one class of important questions,
but also because of its capacity for generating valid and reliable results that
are trusted and used by policy makers. Judith Gueron, president of MDRC,
a large nonprofit research corporation specializing in randomized field tri-
als, made this case convincingly. Over the past 30 years, she said, her com-
OCR for page 9
WHY ARE RANDOMIZED FIELD TRIALS USED? 9
pany has conducted 35 to 40 large-scale randomized field trials involving
about 400,000 people in 250 locations around the United States, and "we've
never had a serious challenge to their credibility." With alternative research
methods, she said, "this is much less true."
With randomized field trials, you can more confidently sepa-
rate fact from advocacy, and with alternative [designs] this is
much less true.
Judith Gueron, MDRC
Acknowledging that "people dont' wake up in the morning wanting to
be in a randomized field trial," Gueron explained that MDRC has built a
constituency over time by persuading participants that these methods are
essential to improve policy. She said that strong statements about the valid-
ity of findings from randomized field trials by prestigious groups, including
National Research Council committees (e.g., National Research Council,
1985, 1994), have helped to convince policy makers of the credibility of
their findings. This credibility, in turn, has helped to ensure that these
findings are translated into laws and programs. For example, when Con-
gress amended the Social Security Act in 1988 (the Family Support Act of
1988, P.L. 100-485), it continued to allow states to waive provisions of the
Aid to Families with Dependent Children law in order to test new ap-
proaches to welfare reform, but the waivers were available only if states
assessed these new approaches. From the early 1980s through 1996, under
both Republican and Democratic administrations, the U.S. Department of
Health and Human Services interpreted this law as requiring states to con-
duct randomized field trials of the new approaches (Gueron, 1997). Out-
side observers agree that the studies conducted by MDRC have had a strong
impact on welfare policy and practice, particularly on the Family Support
Act of 1988 (e.g., Baum, 1991; Haskins, 1991).
Although most workshop speakers agreed that randomized field trials
can potentially yield valid and reliable results, George Mason University
professor Anthony (Eamonn) Kelly raised the most pointed questions about
their viability in educational settings and thus their ultimate utility for im-
proving policy and practice. Kelly argued that researchers face significant
barriers translating the "ideal" design of a randomized field trial into a real-
OCR for page 10
10 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION
life education study.1 He illustrated these real-world "threats to internal
and external validity" by describing problems he identified in a randomized
field trial used to evaluate the Comer School Development Program in
Chicago schools (Cook, Hunt, and Murphy, 2000). The Comer program
aims to improve student achievement by improving the social climate in
the school. In this study, researchers randomly assigned schools to the
Comer program or to a control group. Kelly reported that due to high
turnover of school principals, all elements of the Comer program were not
carried out faithfully in the experimental group, which reduced the validity
of the results. More generally, he argued that results from randomized field
trials must be implemented and "diffused," and the field knows little about
the factors that guide successful implementation. Kelly also suggested that
it is often difficult to generalize the findings from randomized field trials
because local factors may differ significantly from those of the schools in
the study.
There are randomized field trials as intended, and there are
randomized field trials as carried out. You're working with a
system that's in flux.
Anthony (Eamonn) Kelly, George Mason University
Gueron, however, argued that designing randomized field trials in-
volves trade-offs between internal and external validity that must be made
in light of the goals of the study as well as other considerations, such as the
state of knowledge in a field. For education, Gueron argued that since the
use of randomized field trials is in its early stages, the first challenge is to
show that such studies can be successfully conducted. In that context, she
urged researchers to give priority to generating internally valid results, ar-
1Kelly sketched some supplementary methods that could inform the design of ran-
domized field trials as well as implementation and diffusion studies. Describing what is
typically referred to as "design research," he argued that studying, understanding, and im-
proving educational practice must be framed "in terms of exploration and prototyping" until
the operative factors and variables in the complex reality of schooling are better understood.
He cited a recent special issue of Educational Researcher (Kelly, 2003) as providing further
elaboration and criticism of these ideas.
OCR for page 11
WHY ARE RANDOMIZED FIELD TRIALS USED? 11
guing that "it's better to learn something with confidence versus push for
external validity and not understand at the end what you've got."
In response to a question at the workshop, Gueron described the dis-
advantages of not using this design. During the 1970s, she said, Congress
passed a law guaranteeing jobs for young people in selected poor communi-
ties, if the young people would stay in--and make progress in--school. To
carry out the law, the U.S. Department of Labor saturated the selected
communities with funds to create jobs and MDRC tracked the young
peoples' participation in school and work. When a National Research Coun-
cil committee (1985) later reviewed the MDRC study, it agreed with the
study's conclusion that the program produced a dramatic increase in work
while the job guarantee was operating. However, committee members ques-
tioned the conclusion that it also produced a longer term increase in em-
ployment rates, asking whether more young people were working because
of outside variables, such as changes in local labor markets in Baltimore,
Cleveland, and other cities. In retrospect, Gueron said, she wished that the
study team had used random assignment in some communities to yield
stronger conclusions about the long-term effectiveness of the program.
Representative terms from entire chapter:
field trials
