Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 27
4
How Can Randomized Field Trials Be
Conducted in an Educational Setting?
T he researchers and practitioners who spoke at the workshop
offered several practical lessons for planning and conducting
randomized field trials in education. At the conclusion of the
workshop, committee member Kay Dickersin summarized these lessons,
suggesting that success in conducting a randomized field trial in an educa-
tional setting involves four interdependent steps: (1) develop a true
partnership with the school community, (2) ensure the internal validity of
the study design and implementation, (3) focus on recruiting sufficient
numbers of study participants, and (4) plan for future implementation of
an intervention if the study shows it is effective.
DEVELOPING A PARTNERSHIP
By far, the single strongest message that emerged from the dialogue
during the workshop is that developing and nurturing a true partnership
between the research team and the relevant education communities is criti-
cally important to success in carrying out any research in schools, including
randomized field trials. In each of the three studies featured at the work-
shop, researchers were able to gain entry to the schools, to ensure coopera-
tion in faithfully carrying out the interventions, and to make progress to-
ward mutual goals only by establishing trust and encouraging open
communication. Their experiences suggest that it is nearly impossible for
27
OCR for page 28
28 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION
researchers to conduct randomized field trials in schools unless both re-
searchers and education service providers take time to understand each oth-
ers' goals and develop a study design that will help both parties to reach
them.
It is very difficult for researchers to walk into any type of
school setting cold turkey and say we would like to engage in a
randomized field trial study.
Olatokunbo (Toks) Fashola, Johns Hopkins University
All three studies showed the value of partnering (see Boxes 3-1, 3-2,
and 3-3). For example, the Baltimore After-School Program Study was fa-
cilitated by the existing relationship between Fashola (the chief researcher)
and key staff members of the Child First Authority (CFA) after-school pro-
gram where the study was conducted. The Power4Kids study is based on a
formal partnership between Myers and the research team and the Allegh-
eny Intermediate Unit, a group that works with a consortium of school
districts outside Pittsburgh that is participating in the study.
The Baltimore Whole-Day First-Grade Program Study demonstrates
most clearly the value of taking the time to identify the education
community's goals and interests. In their presentation, Kellam and Chinnia
described how their partnership helped both the education community
and the research team meet their goals. Kellam asserted that when a part-
nership is in place based on "mutual self-interests at multiple levels," then
"consent sounds like a silly word"--illustrating how key implementation
tasks such as recruitment are facilitated by the relationship. Chinnia de-
scribed some of the "self-interests" that led to the long-term partnership
between the Baltimore City Schools and the American Institutes for Re-
search. She explained that randomized field trials helped to meet several of
the school system's goals, including intervening early in elementary school
to enhance and maintain student achievement, identifying best practices
for instruction and classroom management, and promoting parent involve-
ment in students' progress. She noted that the current study could help to
sustain best practices in a whole-day first-grade program, and that the goal
of creating and sustaining whole-day first-grade programs is included in
the Baltimore City Public School System's master plan.
OCR for page 29
HOW CAN RANDOMIZED FIELD TRIALS BE CONDUCTED? 29
In order to do this type of research, especially randomized
field trials, it's important that we have very strong partner-
ships. Not only do we have partnerships within the school
system but also with the community at large. Some of this
[partnership building] has taken two to three years of plan-
ning.
Linda Chinnia, Baltimore City Public School System
Workshop speakers also suggested that in large, heavily minority urban
school districts, researchers must be sensitive to issues of race and power
when seeking to develop such partnerships. For example, Jones said that,
partly because of the lack of diversity in the research community, some
minorities do not trust researchers because they feel that "poor and minor-
ity groups are the most evaluated and researched populations while they
have no input into the process." Lewis expressed similar views, saying, "We
are difficult to work with, yes we are, but many of us have been burned, so
we have reason to be difficult." As Dickersin put it in her closing remarks,
creating "culturally competent" research teams who have experience work-
ing with urban schools is critically important to the success of the research.
Some workshop speakers argued that creating racially and ethnically
diverse research teams can be an important step toward enhancing cultural
competence, building trust, and developing partnerships. Lewis suggested
that, to accomplish this goal, research organizations could, for example,
find competent black researchers through the American Educational Re-
search Associations' special interest groups focusing on urban and minority
education (American Educational Research Association, 2003). In addi-
tion, she proposed that these more diverse research teams collaborate closely
with researchers employed by urban school districts. She said that several
large urban school districts (including Houston and Atlanta) have outstand-
ing research staff who would welcome the opportunity to collaborate in
randomized field trials. Jones echoed these sentiments, suggesting that such
efforts can help to break down historical power dynamics between research-
ers (often white) and students and education professionals in urban school
districts (often nonwhite), which might otherwise pose barriers to estab-
lishing mutual trust. Questioning the capacity of the current cadre of re-
searchers to develop partnerships with inner-city schools in particular, Lewis
OCR for page 30
30 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION
argued that researchers are frequently unfamiliar with inner-city schools
and unsure of how to work with them.
Many of the researchers are not skilled in working with people
in urban centers.
Sharon Lewis, Council of the Great City Schools
Finally, establishing close partnerships can help schools not only to
identify but also to implement scientifically based best practices. These best
practices may in turn help schools make the adequate yearly progress re-
quired by the No Child Left Behind Act. Developing and nurturing this
partnership between the researchers and professionals working in the study
sites facilitates all other implementation tasks--including the three de-
scribed next.
ENSURING INTERNAL VALIDITY
Dickersin also pointed out that several workshop speakers emphasized
the need to spend time in the early design and planning phases to ensure
internal validity of the study. Indeed, in debating the merits of randomized
field trials in education, many scholars have focused on the trade-off that is
necessary between internal validity--that is, the extent to which it can be
concluded that the treatment led to the effect, or difference, between one
group and another on a particular outcome--and external validity--the
extent to which the findings of a particular study hold in other times, places,
and populations. For example, critics argue that the strict protocols of the
studies, which are required to maximize internal validity (e.g., program
options are consistently and comprehensively implemented in both the ex-
perimental and control groups), do not reflect typical school operations,
and thus the usefulness of the results for real life is questionable (Cronbach
et al., 1980; National Research Council, 2002).
Several workshop speakers and audience members raised questions
about a particularly important aspect of comparative studies, including ran-
domized field trials, in ensuring internally valid results: how to measure
and account for the implementation of the experimental and control treat-
ments. Gueron argued that it is important for investigators to monitor and
to understand how the interventions are being applied during the study to
OCR for page 31
HOW CAN RANDOMIZED FIELD TRIALS BE CONDUCTED? 31
assist with the interpretation of the comparison of outcomes between the
groups. She cited the collection of "equivalent and high-quality data" on
process and contextual factors in both groups as essential to the success of a
randomized field trial. Shavelson, too, urged researchers to spend the time
and money required to observe and characterize implementation, since how
a program plays out in schools can often be quite different from what was
planned or expected at the outset of the study. Kellam, too, argued that it is
extremely important to measure implementation in the control group "with
the same intensity" as in the experimental group.
Two speakers offered specific strategies, based on early planning, that
helped to ensure that the intervention is carried out faithfully and that the
randomized field trial would yield internally valid results. In the Baltimore
After-School Program Study, Fashola arranged for the CFA to hire new
non-first-grade teachers who were not a part of the regular CFA program to
provide one-on-one tutoring to the experimental group of first graders.
Using new teachers helped to ensure that the existing CFA did not provide
similar tutoring to students in the control group. In the Power4Kids study
in the Pittsburgh area, Myers described a similar approach. He asked each
participating school to nominate one teacher to provide the remedial read-
ing interventions, replacing these teachers in their usual positions with long-
term substitutes for the entire school year. These teachers were given spe-
cific training and materials to help ensure that they carry out the alternative
reading programs as designed. In this way, Myers argued that they are un-
likely to "accidentally" provide the remedial reading tutorials to students
not in their experimental groups, because they will not be acting as regular
classroom teachers. Researchers employing randomized field trials can also
use observational methods to detect factors that might contaminate the
results of the study. For example, in the Comer study described by Kelly,
many ethnographers were hired to study and characterize program imple-
mentation, a major factor in the researchers' ability to identify threats to
internal validity.
RECRUITING SUFFICIENT NUMBERS OF
STUDY PARTICIPANTS
As summarized in Chapter 3, Dickersin reiterated that recruiting study
participants is critical to ensure sufficient sample sizes and to be able to
draw valid and reliable conclusions from a randomized field trial in turn.
The implementation of this step, too, depends in large part on the broader
OCR for page 32
32 IMPLEMENTING RANDOMIZED FIELD TRIALS IN EDUCATION
partnership between researchers and educators. According to several work-
shop speakers, success in recruiting study participants can be aided by key
intermediaries. At the highest level, a board including school, community,
and research officials--such as the Baltimore City Community and Institu-
tional Board, which oversees the long-term research program there--can
help to both oversee and develop support for the study. When many school
districts are involved in a large-scale study, an intermediary such as the
Allegheny Intermediate Unit led by Durno, who has strong working rela-
tionships with the schools, can help to forge partnerships. At the level of
the individual school, a designated site coordinator can help to communi-
cate with teachers and parents and ensure that random selection and ex-
perimental and control processes flow smoothly, and that differences be-
tween the experimental and control groups are preserved over the course of
the study. In describing the use of such a strategy, Durno put it this way,
"the on-site coordinator must be an excellent communicator who can meet
the school on that school's terms." Chinnia trained a community organizer,
who was already trusted by parents, to help explain the study and win the
support of students and parents. Now in a full-time position as a "family
classroom partnership aide," the coordinator has won consent among 100
percent of parents when recruiting students to participate in the most re-
cent generation of randomized field trials.
One of the challenges that we experienced in working with
the schools is the difference in the climate and the culture of
each school.
Donna Durno, Allegheny Intermediate Unit
PLANNING FOR IMPLEMENTATION
When educators consider the possibility of participating in a random-
ized field trial, workshop speakers suggested that the pivotal factor is often
whether the study is likely to lead to an actual improvement in their own
schools. Researchers can help educators reach this objective by designing
the study to support future implementation of an intervention if it has
been shown as effective and, after the study is completed, by working with
the school and funders to provide the effective intervention to all students.
The Baltimore Whole-Day First-Grade Program Study plans for and sup-
OCR for page 33
HOW CAN RANDOMIZED FIELD TRIALS BE CONDUCTED? 33
ports future implementation of the intervention by including analysis of
the cost-effectiveness of the whole-day program, in light of its potential to
reduce the future costs associated with drug abuse and school failure.
In all three of the studies described at the workshop, research teams
provided materials and training to administrators and teachers, at no cost
to the schools. Workshop speakers describing these studies suggested that
investigators wishing to conduct a successful randomized field trial should
consider whether they have adequate funding to provide such services. With
adequate funding, researchers can form a research partnership that will help
schools reach their goals, which include implementing (not just studying)
educational interventions if they are found to be successful. Providing a
proven program to all students following the study ensures that the study
results are used and useful, not only by policy makers, but also by class-
room teachers.
LOOKING AHEAD
Taken as a whole, the workshop presentations and discussions sug-
gested that the outlook for continuing and expanding randomized field
trials in schools, based on strong community-research partnerships, is good.
In her concluding remarks, Dickersin said that one of the most important
things she had learned from the workshop was that some educators were
"very positive about participating" in randomized field trials "if the barriers
. . . can be overcome." She referred to an earlier session, in which she had
asked education representatives how they would respond if she proposed to
conduct a randomized field trial on an issue important to their schools,
with adequate resources and a culturally sensitive research team. In response,
Lewis said, "I would jump at it," while Jones said that her students (study-
ing to become teachers and education researchers) would also want to be
partners in the research.
The three studies featured at the workshop also indicate that when
designed to help meet their goals, educators and school officials would
welcome the opportunity to participate in, and help carry out, randomized
field trials. In presenting each case, representatives of the education com-
munities described why they viewed participating in such studies as benefi-
cial for their schools. Their enthusiasm at the workshop appeared similar to
what Gueron said she had encountered among state and local welfare offi-
cials who have willingly participated in repeated randomized field trials
over the years, because "they actually believe it makes a difference and . . .
can bring visibility and resources to their community."
Representative terms from entire chapter:
field trials
