Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 29
n
Ha~ninn Ha
~~a situ since
~es:
Y1
Would the substantial investment required to build a
new education research and development infra-
structure pay off? Proof, of course, cannot be pro-
vided by arguments made in advance of such an
experiment. But our case for such an investment rests on evi-
dence that: (a) the collaborative research and development ef-
fort in school settings that we propose is feasible; (b) there are
cases in which this type of research and development has been
carried out with productive results that are directly applicable
to improving classroom practice; and (c) this type of work, even
when highly successful, has been difficult to sustain without the
proposed research and development infrastructure.
To support these claims, we provide a set of illustrative
cases below. The cases differ substantially in their details, but all
were pioneering efforts; the individuals involved had to find
paths through unfamiliar, and at times difficult, terrain. The
researchers involved had to become a jack-of-all-trades, able to
function in the separate worlds of research and practice, to
design and conduct research, develop and maintain partner-
ships, and continue to raise substantial funds. And the efforts
continue only so long as the individuals who undertook them
do not tire and their funders do not shift focus. There are no
railroads or highways that have been built in these pioneer's
footsteps. Without a supporting infrastructure, their paths did
not become well-traveled roads, and the settlements they cre-
ated are unlikely to become permanent.
WHY INFRASTRUCTURE MATTERS
nttarQ
29
OCR for page 30
BOSTON READING STUDY
In 1996 the Boston public school system, with substantial
foundation support, introduced a whole school reform program
focused on primary grade literacy. Participating schools were
required to adopt a structured model for literacy instruction
from a menu of four options: Balanced Early Literacy (BEL),
Developing Literacy First (DLF), Literacy Collaborative (LC),
and Success for All (SFA). In the 1998-1999 school year, 66
schools were involved in the effort. The variation in the pro-
grams adopted, as well as the number of schools involved,
provided an opportunity to learn a great deal about which
programs produce which results, whether the program results
differ for children with different demographic and primary lan-
guage characteristics, and how teachers and administrators use
the programs and their professional development components.
A research study was designed to examine these questions
in Boston. In 199S, with initial funding from Harvard Universit~v's
Interfaculty Initiative, Lowry Hemphill and Terrence Tivnan
began pilot work in two schools that later expanded to a study
of 16 schools 4 schools using each of the 4 models. The study
was designed to examine how students perform on the separate
skills that are known to contribute to success in early reading:
word reading, word attack, phonemic awareness, writing, and
reading comprehension (Tivnan, 2002~.
While individual program effects are being studied in this
ongoing program of research, so are the differences in instruc-
tional practices between and within programs. Data on student
performance are collected, along with data on student charac-
teristics and characteristics of the instructional program. Impor-
tantly, data will be collected over multiple years. Since reading
is a skill that is still emerging in first grade, results of the
program at the end of second and third grade will be critical to
an informative program assessment.
Early findings already suggest that many important lessons
are being learned about the programs and their role in teaching
and learning, as well as about research on classroom practice.
With regard to the programs themselves, their implementation
has made a measurable difference. The biggest gains were in
decoding and in reading sight words. No program could be
30
STRATEGIC EDUCATION RESEARCH PARTNERSHIP
OCR for page 31
deemed best overall. As one might expect, the programs that
spent more time on developing phonemic awareness skills
showed better results in tests of this skill, while programs that
spent more time on language development showed relative
strength in vocabulary and reading comprehension. More im-
portantly, however, the differences in outcome for the same
program in different classrooms were larger than the differ-
ences between programs. This is not an effect that can be attrib-
uted to the school, since large differences exist within a school
as well. The research team is observing teachers' instructional
practices in order to develop and test hypotheses about the
contributors to differences in teacher effectiveness.
The preliminary results point to another important issue:
the programs as a group post substantial gains in bringing
students up to grade level in word reading, but they make far
less progress in narrowing the yawning gap in reading compre-
hension and vocabulary. While the programs differ somewhat
in their emphasis on these skills, none provides an adequate
response to the existing disparities. For public policy and for the
direction of future research and development dollars, this is a
very important finding.
W O U ~ D S E R P M A K E A D ~ F F E R E N C E ~
The research now under way in these Boston schools is an
example of the type of research that can help inform education
practice and policy. Indeed, the research team is working with
teachers in the schools to provide professional development
based on what is being learned about effective instructional
practices. The schools involved in the project are eager for input
from the researchers both for understanding the program out-
comes and for informing instructional practice.
But while this case provides optimism regarding the poten-
tial of collaborations between researchers and practitioners, it
also points to the inefficiencies and disincentives of carrying out
this work in the absence of a supportive infrastructure. Without
a formal organizational arrangement between the research team
and Boston public schools, the researchers themselves must
negotiate arrangements with each of the 16 schools. Those ar-
rangements depend on personal relationships, and with each
personnel change, the relationship must be renegotiated. The
researchers also needed to obtain the parental permissions re-
WHY INFRASTRUCTURE MATTERS
31
OCR for page 32
quired to collect information on individual students for this
type of research.
At the same time, the researchers have needed to secure the
funds required to carry the project forward. The funding has
come in one- or two-year commitments, requiring persistent
attention to negotiating the next research grant or contract. In
the early stages of the project, the researchers were also re-
quired to negotiate with the developers of each of the four
programs regarding the outcome measures that would be used
to assess program impact.
If SERP were in existence, funding, research protocols and
instruments, and the terms of access to schools would still need
to be negotiated, and permission to collect data on individual
students would still need to be obtained. But an organization
could develop the capacity to do these much more efficiently by
institutionalizing the knowledge and skills involved and mak-
ing routine what otherwise must be reinvented by each
research team. Many outstanding researchers would be unable
or unwilling to undertake the Herculean efforts of the Boston
research team, discouraging this type of badly needed investi-
gation.
The role of a SERP enterprise is not just facilitative. It would
develop and steer a program that could make more of research
findings. The Boston researchers are keenly aware of the oppor-
tunities lost by working in isolation. While many other jurisdic-
tions around the country are using the same literacy programs
in different contexts, the lack of a coordinated effort means that
they are learning less about the features of those contexts that
contribute to outcomes. The confidence in particular outcomes
would be strengthened or called into question if results from
other sites could be compared. But such a comparison requires
an extent of coordination in research design that does not now
exist.
Hemphill and Tivnan write up their results and present
them at research conferences. They point out, however, that the
operative norms at these meetings are not those of a network of
researchers engaged in an effort to collectively advance a field,
but rather those of a professional competition that minimizes
the opportunity for productive collaboration. In their view, the
role of a SERP network in fostering an environment in which
the operative norms are those of productive collaboration would
make a significant contribution to the productivity of work like
32
STRATEGIC EDUCATION RESEARCH PARTNERSHIP
OCR for page 33
theirs. The benefits of that collaboration, of course, would ex-
tend beyond researchers. A critical role for SERP would be one
of supporting collaboration and shared knowledge among the
school systems and teachers undertaking reforms to address
similar problems (based in part on phone conversation with
Lowry Hemphill, November 2002~.
· ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
RECIPROCAL TEACHING
As the Boston reading study results confirm, many students
who successfully learn to read nonetheless do poorly at reading
comprehension. Instructional approaches to improving reading
comprehension primarily fall under the category of strategy
instruction (RAND, 2002a). The skills taught in strategy instruc-
tion generally target improved recall of text, teaching students
to attend to headings, to outline or map the text in graphic form,
and to reread for specific information or structural cues. While
these strategies do improve recall, particularly for low-achiev-
ing students (National Institute of Child Health and Human
Development, 2000), they focus on surface features of the text.
They can therefore be mastered successfully without the stu-
dent understanding the meaning of the text or integrating the
new knowledge from the text with their existing understand-
ings.
Reciprocal Teaching (RT) is a technique developed by
Annemarie Palincsar and Anne Brown two decades ago to en-
gage students more deeply in understanding the meaning of
text. The active processes of making sense of a text involved in
skilled reading comprehension are taught to students explic-
itly.1 The teacher initially models four strategies: questioning
unclear content, summarizing meaning paragraph by paragraph,
clarifying comprehension problems, and predicting what will
come next. Students practice the strategies with guidance from
the teacher, and, as their skill increases, the teacher increases the
demands. Gradually the role of the teacher diminishes as stu-
dents become more competent and sophisticated in the ques-
tioning and monitoring role. In groups, students ask each other
For more detail, see the companion report, Learning and Instruction: A
SERP Research Agenda.
WHY INFRASTRUCTURE MATTERS
33
OCR for page 34
questions, practicing aloud the type of dialogue that will even-
tually become internal.
The technique has been used to improve listening compre-
hension among young children, as well as to improve reading
comprehension once children become fluent readers. Children
who are exposed to the reciprocal teaching intervention per-
formed better than control children on several dimensions, in-
cluding the quality of summaries and questions and scores on
criterion tests of comprehension (Palincsar and Brown, 1984~.
Gains were maintained over time, generalized to classroom
comprehension tests, and transferred to novel tasks involving
summarizing, questioning, and clarifying.
W O U ~ D S E R P M A K E A D I F F E R E N C E ~
Reciprocal Teaching has demonstrable effects on a problem
that is at the heart of effective education. The ability to compre-
hend text unlocks knowledge in all fields. But without an infra-
structure to nurture the program through further stages of de-
velopment and integration into the classroom, Reciprocal
Teaching has largely remained a small-scale effort in the hands
of a dedicated researcher whose work has moved into new
areas. The absence of any infrastructure to carry the program
forward has repercussions that Annemarie Palincsar describes
vividly:
Believe it or not, after all these many years, I still get requests (at
least two a month) to conduct professional development regard-
ing reciprocal teaching. I always feel badly saying no (in part
because a very important reason for doing the research was to
inform practice), and I really have no one to whom I can refer the
school personnel with confidence that the version of RT that they
will describe/demonstrate is consistent with the original RT....
Because there is currently nothing like SERF, I had no systematic
way to disseminate RT on a large scale nor to engage in profes-
sional development that would reach the numbers of schools and
districts that have requested it.
· · ~
This is what I did do.... I authored a facilitator's manual (that has
been sent to literally thousands of folks who have requested it).
I also prepared a videotape that provides an overview of RT,
excerpts of a teacher implementing RT, and debriefing conversa-
tions with students who have learned how to use RT. I purchased
a tape-to-tape video recorder and when people ask for this video
tape, I ask them to send me a blank VHS tape in a stamped, self-
34
STRATEGIC EDUCATION RESEARCH PARTNERSHIP
OCR for page 35
addressed padded envelope and I make them a copy of this tape. I
have lost count of how many copies I have made but I should be
embarrassed to send any more since it is so dated!
What role might a SERP have played? Well, first, it would have
been wonderful to have had high-quality video and audio of
teachers working in different contexts (demographically as well as
content-wise) using RT in a manner that reflects the principles on
which it was designed.
Second, it would have been so satisfying to never have had to say
"no" to a professional development request because there was a
network of teacher leaders or professional development person-
nel who would either work onsite or who would have offered
teaching institutes to support educators and educational leaders
to learn about RT (and other forms of comprehension instruc-
tion).
Third, it would have been ideal to be able to have supported
conversations with other researchers who were similarly investi-
gating class-wide models of comprehension instruction to talk
about how our work was complementary and where it differed
and the implications of these differences for professional develop-
ment. For example, what support can we provide educators who
are trying to choose among QtA [Questioning the Author] (Beck
et al., 1997), Collaborative Reasoning with Text (Chine and
Anderson and colleagues), SAIL (Pressley et al., 1989) and RT?
How might we have done our research more synergistically so
that we might have learned more from our respective programs
of research? I can imagine an entity like SERP playing such a role.
PaTincsar describes an effort by a major commercial publish-
ing company to conduct professional development on RT, with
neither her permission (which is not required), nor her input.
Her understanding is that the version of RT taught in the train-
ing program is fundamentally different from hers in critical
dimensions. But no one at the company has returned her phone
calls. She writes:
Fourth. . . it would have been terrific to have some means of
working more closely with commercial endeavors, so that these
efforts do not undermine the research and development the
publisher is trying to disseminate.
A major national goal expressed in the No Child Left Behind
legislation is effective reading instruction for all children. Mil-
lions of dollars are earmarked to support the effort. But without
WHY INFRASTRUCTURE MATTERS
35
OCR for page 36
a research and development infrastructure, one of the few inter-
ventions that has been demonstrated to improve reading com-
prehension outcomes will reach children only if a teacher learns
about the program, contacts Annemarie PaTincsar and mails a
self-addressed, stamped enveloped with a blank tape inside
(Annemarie Palincsar, Charles Waigreen Professor of Reading
and Literacy, School of Education, University of Michigan, per-
sonal communication, November 2002~.
· ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
CREATING THE COGNITIVE TUTOR
The Cognitive Tutor Algebra I is one of a set of "cognitive
tutors" developed at Carnegie Mellon University to teach aige-
bra and geometry. Of great relevance to the SERP vision, the
tutors are a good illustration of how to make the transition from
the laboratory to the classroom, as well as the nature of the
partnership between researchers and teachers that have made
the program a success.
The work at Carnegie Mellon began as a project to see
whether a computational theory of thought, called ACT (Ander-
son, 1983), could be used as a basis for delivering computer-
based instruction in algebra. The ACT theory of problem-
solving cognition is the basis for modeling students' algebra
knowledge. These models are capable of generating almost any
sensible solution to an algebra problem. They are embedded in
a computer program that can then identify the particular ap-
proach a student is taking to a solution.
The cognitive models enable two sorts of instructional re-
sponses that are individualized to students:
1. By a process called mode! tracing, the program will
infer how a student is going about problem solving
and generate appropriate help and instruction when
a student is pursuing an unproductive or incorrect
strategy.
2. By a process called knowledge tracing, the program
will infer where a student falls in the learning trajec-
tory (what knowledge has been mastered and what
is insecure) and select instruction and problems
appropriately.
36
STRATEGIC EDUCATION RESEARCH PARTNERSHIP
OCR for page 37
Developing cognitive models that accurately reflect student
competences and developing appropriate instructional responses
is very much an iterative process. The success of the tutors
depends on a design-test-redesign effort in which models are
assessed for how well they capture competence and in which
instructional responses are assessed for effectiveness.
In controlled trials, the curriculum performs well. It was
found that students could go through the algebra curriculum
with the Tutor in a third of the time normally required. In
carefully managed classrooms, students would show about a
standard deviation (approximately one letter grade) improve-
ment in achievement (Anderson et al., 1995~. In real classroom
situations, the impact of the tutors tends not to be as large,
varying from O to 1 standard deviation across 13 evaluations.
Another third-part~v evaluation focused on the social conse-
quences of the tutors; it documented large motivational gains
resulting from the active engagement of students and their suc-
cessful experiences on challenging problems (Schofield et al.,
1990~.
Unlike many such small-scale success stories in cognitive
science, this project was able to grow to the point at which the
cognitive tutors now are used in 33 of the 100 largest school
districts in the United States and are interacting with about
200,000 students yearly. A number of features were critical to
making this successful transition:
1. While the ACT theory provided the foundation for
the program, there was a concerted effort to identify a
curriculum that educators wanted taught in the class-
room. In particular, there was a major effort to teach a
curriculum that was in compliance with the National
Council of Teachers of Mathematics standards (Na-
tional Council of Teachers of Mathematics, 1989~.
2. A curriculum was designed that teachers would ac-
cept and could implement. The curriculum design
was largely the product of teachers with experience in
urban classrooms. To meet their needs, a full-year
curriculum was developed rather than an enrichment
program to be inserted into an existing curriculum.
And the computer tutors were used as a support rather
than a replacement for the teachers. In this curriculum
WHY INFRASTRUCTURE MATTERS
37
OCR for page 38
students spend 40 percent of their time with the com-
puter tutors and 60 percent of their time with other
activities. These classroom activities help them transi-
tion to their lessons with the tutor, as well as to transi-
tion from the tutor to things that they will have to do
in the real world.
3. A structure was set up for supporting the use of the
curriculum and tutors. Before introducing the tutors
into a classroom, it has been important to provide
professional development time to enable teachers to
prepare for the change they are about to experience. A
center at Carnegie Mellon was set up for responding
to teacher and school problems. As the adoptions
grew, a separate company (Carnegie Learning Corpo-
ration) was created to perform this function and main-
tain and adapt the materials.
4. Ultimately, such a curriculum must be financially self-
sustaining, and the program was developed from the
beginning with a plausible financial model in mind.
In particular, by offering a full grade 9-11 curriculum,
it was possible to earn the kind of income from sales
that is necessary to sustain the activity.
W O U ~ D S E R P M A K E A D I F F E R E N C E ~
The Cognitive Tutor Algebra I represents a success of the
type that is rare in K-12 education. Its developers, however,
point to the problems raised when an effort like this is under-
taken without a research infrastructure:
Once leaving the laboratory, there have been only haphazard
efforts to evaluate the curriculum as it has multiplied through the
school systems. It is only now that our tutors are about to receive
their first adequate third-party evaluation and this is only because
of the funding from the Hewlett Foundation. There are natural
38
reasons to avoid rigorous evaluation of material. Early in the
development of a program a negative evaluation may make it
difficult to get the next round of funding. Once a product be-
comes commercial as ours did, there is even less incentive for
such evaluations because in addition to bringing potentially bad
news, they cost and so threaten the need to meet the next
month's payroll. Mechanisms need to be set up to both require
and fund rigorous formative evaluations in the development of
curricula, and impartial third-party evaluations of curricula once
they start to be disseminated.
STRATEGIC EDUCATION RESEARCH PARTNERSHIP
OCR for page 39
And despite its success, Anderson argues that the Cognitive
Tutor Algebra I has room for improvement in some very impor-
tant dimensions.
Early in the development of the algebra tutor, a decision was
made to place a heavy emphasis on contextualizing algebra to help
students make the transition to the formalism. This has been
successful and there are fewer students dropping out. However,
as a consequence the curriculum does not achieve the fluency in
symbol manipulation and abstract analysis expected for high-
achieving students. There is no reason why the cognitive tutors
could not be extended to these topics but they have not (from
Learning and Instruction: A SERP Research Agenda, Box 3.3~.
· ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
THE COGNITIVE TUTOR ALGEBRA I
IN AN OKLAHOMA SCHOOL
DISTRICT
Pat Morgan, mathematics coordinator for the Moore Inde-
pendent School District in Oklahoma, knew Carnegie Learning
Corporation's Cognitive Tutor Algebra I had been rated favor-
ably by the U.S. Department of Education. She thought it was
worth a try in her school district but knew from experience that
her teachers were likely to balk at being asked to do something
new. She decided that, to get their support, she would need to
show them that the new program worked better than their
current program. Her plan was to introduce the Cognitive Tutor
Algebra I in a subset of the algebra classrooms, and compare the
results to those in the classrooms that continued to use McDougal
Littell's Heath algebra I text.
With a Goals 2000 grant from the U.S. Department of Educa-
tion, Pat was able to purchase the program and pay teachers to
attend training workshops during the summer of 2000. In Sep-
tember, the Cognitive Tutor Algebra I was introduced in five
middle schools in the district. Students in the honors algebra
class were not involved in the study. Other students had al-
ready been assigned more or less randomly to classes, and
teachers who had undergone training were asked to teach both
traditional classes and Cognitive Tutor Algebra I classes so that
the effect of the teacher could be separable from the effect of the
program. In order to compare outcomes, Pat decided to use a
WHY INFRASTRUCTURE MATTERS
39
OCR for page 41
it somewhat. She would like to understand the consequences of
the adaptations. But with tight budgets, she cannot afford to
have ETS administer the end-of-year test anymore. The school
district is now also experimenting with Carnegie Learning's
Cognitive Tutor Algebra II and with the Cognitive Tutor Geom-
etry on a small scale, but they are not conducting a similar
study. While Pat would like to continue to study program ef-
fects to better inform her decisions, her job as the math coordi-
nator is very demanding, and she cannot undertake the effort to
raise funds and conduct additional studies on her own again.
Even more than the money, she says, she needs help so that the
unfamiliar job of researcher does not fall fully on her shoulders.
Pat Morgan's research design was better than she knew. She
had no background in research. She just believed from experi-
ence that the teacher is the most important contributor to stu-
dent achievement, so she decided that she needed to have the
same teachers using both curricula. Students who were not
assigned to honors algebra had been placed without tracking
into other sections of algebra before a decision was made re-
garding which teachers would introduce the new curriculum in
which of their classes, so random assignment was happenstance.
Once the data were collected, Pat wasn't sure how to make the
best use of them, so she called Carnegie Learning and found
help on that score. She provided one of the best tests of the
curriculum to date, as well as a valuable source of information
for other school districts considering the program, although
that outcome was not by design.
A research and development infrastructure prepared to sup-
port efforts like this and to guide research design could make
more commonplace what is now the outcome of the combina-
tion of happenstance and extraordinary effort by a very dedi-
cated and insightful school administrator. It could also make
the very instructive finding from one school district easily avail-
able to other districts, so that thousands of other Pat Morgans
could persuade their teachers of the value of trying a new
instructional approach (Pat Morgan, mathematics coordinator,
Moore Independent School District, Moore, Oklahoma, personal
communication, November 2002~.
WHY INFRASTRUCTURE MATTERS
4
OCR for page 42
LINKING RESEARCH AND PRACTICE
WITH EASE
One Wednesday morning in 199S, Catherine Snow arrived
in her office to find a large box. It contained many sheaves of
test data, summarized in a few tables, and a note that read
something like this:
I collected these data in the Title I program of the White Bear
Lake Schools we designed an intervention program based on
your findings from the Home-School Study of Language and
Literacy Development. The results seem to suggest it worked. But
I don't know how to do the right analyses, so I am sending you
the data.
Snow got in touch with the source of the note and the data,
Gail Jordan, who was then Title I director in White Bear Lake.
Gail is a gifted curriculum designer and teacher educator, with
a commitment to using research results to inform practice. Gail
had taken seriously the correlational findings reported by Snow
and her colleagues suggesting that preschool and kindergarten-
age children in Tow-income families did better in literacy learn-
ing if they had had rich linguistic interactions with their par-
ents. The helpful interactions that the researchers described
included telling stories, reading books and engaging in discus-
sion about them, giving explanations, using rich vocabulary,
and engaging in pretend play. Relying heavily on the research
findings, Gail figured out how to teach parents to engage in
these sorts of interactions and how to design activities that
kindergarten teachers could assign as homework that would
provide occasions for the parent. She also met with the research
team before designing the program.
With a Ph.D. under her belt, Gail also designed and carried
out a random assignment study randomly assigning kinder-
garten classrooms to treatment and control conditions, pretest-
ing all the children in order to be able to control for initial status,
and incorporating ways of assessing how many of the activities
parents engaged in. She was engaging in precisely the kind of
problem-oriented, practice-embedded research that is needed
to improve education, and she successfully designed and car-
ried out a very sophisticated study. She did not, however, know
how to analyze the results or write up the findings for broader
42
STRATEGIC EDUCATION RESEARCH PARTNERSHIP
OCR for page 43
dissemination. Fortunately, she passed along the data, rather
than just leaving them in the corner of her office.
Snow had some uncommitted funds for doctoral student
support available, so she hired Michelle Porche to analyze the
dataset. Michelle's analysis confirmed, as Gail's preliminary
look at the findings had suggested, that the group that had
received the intervention showed greater gains in language
skills over the course of their kindergarten year and, further-
more, that those gains were greatest for the children who had
started with the weakest language skills. Gail Jordan subse-
quently visited Snow and Porche, and they worked together on
writing up the paper. It was published in Reading Research Quar-
terly as "Project EASE: Easing Children's Transition to Kinder-
garten Literacy Through Planned Parent Involvement." It won
the International Reading Association award for the best paper
published in that journal in the year 2000. Porche also helped
Jordan build a web site describing EASE and providing re-
sources for those who wanted to replicate it; EASE is now being
used widely in Tow-income districts in the United States, and it
has been incorporated into a state literacy reform initiative in
Ohio (where it is again being evaluated).
WO U ~ D S E R P H AV E MA D E A D I F F E R E N C E ~
The outcome in the EASE case was a very positive one from
all standpoints. The important work done both by a practitioner
and by researchers came together in a way that allowed both
practice, and the knowledge base, to advance. But while the
success is worthy of celebration, it is disquieting that so many of
the critical events were a matter of chance, as Snow makes clear:
Many events had to converge to enable EASE to be disseminated
and implemented outside White Bear Lake. First, Gail Jordan is
more focused on the possibilities of research than many practitio-
ners though of course in an ideal world every educational
innovation initiated locally would be subjected to a systematic
evaluation. Second, Gail and I had met before, so she felt she
could send me the data. Third, I happened to have some uncom-
mitted funds that could support analysis of the data. Fourth, I
happened to have a doctoral student interested in parent involve-
ment and in literacy, who could thus easily be recruited to be
involved in this project. Fifth, the findings were of sufficient
interest that Reading Research Quarterly was willing to publish
them. Sixth, the intervention was designed in such a way that it
WHY INFRASTRUCTURE MATTERS
43
OCR for page 44
was feasible for others to replicate it without much adaptation. If
any one of these factors had been different, this valuable educa-
tional intervention with its potential to improve children's literacy
success would never have seen the light of day.
Gail Jordan also notes how easily the value of this work could
have been compromised.
It was a challenge to keep the integrity of the project because the
planning team had no research background and there were many
times that there were confounding suggestions made (like making
experimental classrooms full day and control classrooms half day).
There were also concerns about the amount of testing required. .
. . It would have been wonderful to have a research team guide us
in those key decisions.... Our success was truly due to the
kindness of strangers, specifically the research team at Harvard
(Gail Jordan, personal communication, December 2002~.
Surely there are many other cases in which locally designed
innovations remain local and person-specific because some or
all of the chance events were absent. The purpose of SERP is to
make what is now an extraordinary outcome much more com-
monplace. It would do so by providing the infrastructure to
connect practitioners like Gail Jordan to senior researchers like
Catherine Snow and more junior researchers like Michelle
Porche. It would provide support for the design of an interven-
tion to assure replicability, as well as for the design of program
evaluation to ensure rigor, relying less on extraordinary capaci-
ties of those in Gail lordan's role.
In a world with a well-functioning SERP organization,
progress would not require that the developer of the interven-
tion have the capacity to design a random assignment trial on
her own, as was true in this case. And SERP would have exten-
sive capacity for data analysis so that high-quality data collec-
tion efforts, to be used, would not require the good fortune of a
researcher with financial and human capital to spare. Finally,
SERP would engage the effort to make the EASE results avail-
able more widely to those who are unlikely to read a report of
published research (Catherine Snow, Henry Lee Shattuck Pro-
fessor of Education, Harvard Graduate School of Education,
personal communication, December 2002~.
44
STRATEGIC EDUCATION RESEARCH PARTNERSHIP
OCR for page 45
· ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
CONSORTIUM ON CHICAGO
SCHOOL RESEARCH
Perhaps the most powerful evidence to suggest the possibil-
ity and the value of research on practice, conducted in school
settings as collaborative efforts among researchers, practitio-
ners, and policy makers, comes from the Consortium on Chi-
cago School Research. The consortium began in 1990 under the
leadership of Anthony Bryk as an effort to study the impact of a
major school reform effort passed into law in Chicago in 1988.
The mission of the consortium is to undertake research of high
technical quality that can inform education policy making and
school improvement efforts.
The 1988 Chicago school reform decentralized authority and
accountability in the schools. It established local school councils
(LSCs) for that purpose, comprised of the school's principal,
two teachers, and six elected parents.3 The LSCs approved the
budget and held authority over the principal's contract. All
schools were required to develop, implement, monitor, and
update annually a school improvement plan (SIP), with LSC
participation and oversight (Consortium on Chicano School Re-
search, 2003~.
The consortium's initial task was to study what happened in
the wake of the reform. From the outset, this required compre-
hensive data collection. Since 1991 the consortium has con-
ducted biannual surveys of students, teachers, and principals.
Through an agreement with the Chicago Public Schools, its
archives include the following:
· Test score data for the Iowa Tests of Basic Skills
(ITBS), tests of academic proficiency, the Illinois Goal
Assessment Program, and the Illinois Standards Achieve-
ment Test (beginning in the late 1980s);
· Administrative history information (as of 1992~;
· Grade files from all high school students (as of
1993~.
As part of a five-year grant from the Chicago Annenberg
Research Project, the consortium also collected extensive data at
3In high schools, a student representative joined the group.
WHY INFRASTRUCTURE MATTERS
45
OCR for page 46
24 schools on teacher assignments, samples of student work,
and extensive, in-depth interviews and classroom observations.
As the reforms in Chicago evolved, including a shift of
authority from school boards to the mayor in 1994-1995 on a
temporary basis, the consortium has continued to amass the
data that will allow the effects to be teased out over time. The
availability of this rich dataset has spawned many studies over
the decade of importance for education policy. They have ad-
vanced understanding of schools as organizations and of the
conditions that foster school improvement, providing critical
insights on the effects of high school size, intraschoo! teacher
relationships, and the cognitive demands placed on students by
teachers.
The consortium has also worked with schools to provide
them with data, and an approach to interpreting it, that give
schools greater insight into their own functioning and perfor-
mance. For example, the consortium's work allows each school
with an eighth grade class to look at how its graduates perform
over the course of the next five years. But the exchange between
researchers and teachers is bidirectional. Important findings in
studies of relational trust among school personnel have origi-
nated from the insights of teachers who felt that this played a
major role in the performance of a school's students.
Many of the studies done by the consortium could serve to
illustrate research that has provided critical knowledge and
insights for policy makers, practitioners, or both. For example,
the enactment of legislation in 1996 ending social promotion in
the Chicago Public Schools set minimum scores in math and
reading that students must achieve on the ITBS in grades 3, 6,
and ~ in order to be promoted. Students who failed to meet the
cutoff were required to attend a summer school program and
retake the test at program completion. Students who fail again
to meet the standard are retained in grade.
The Chicago policy was designed to address problems faced
by all school districts. Many students are having difficulty in
later grades, particularly high school, because they lack basic
skills. Teachers are being asked to teach to higher standards. But
many believe that the students who appeared in their class-
rooms do not have the skills to move on to more advanced
work.
The "theory of action" in the legislation, according to the
consortium research team, is three pronged (Roderick et al.,
46
STRATEGIC EDUCATION RESEARCH PARTNERSHIP
OCR for page 47
1999~. First, before students are tested, they, their parents, and
their teachers face new incentives. When students are confronted
with the prospect of being retained in grade, they are motivated
to work harder, and their parents are motivated to monitor the
student's performance more closely. Teachers are sent a strong
message to focus attention on students who are not mastering
basic skills and to emphasize those skills in their teaching. To
improve the opportunity to succeed, students who are at risk of
failure are given extended instructional time through Light-
house, an after-school program begun in 1997.
The second prong provides an opportunity for a second
chance. If a student fails to meet the minimum standard at the
end of the school year, the summer bridge program offers addi-
tional, more focused instruction. The theory is that many stu-
dents who fad! initially can be brought up to speed with this
additional opportunity. Finally, a second failure to meet the
standard is met with retention in grade. The theory here is that
students who repeat the material yet again will master it and
move on to the next grade better prepared.
A research team at the consortium set out to test each of
these theories. A full analysis will require data collection over a
longer period of time to ascertain Tong-term effects, but results
from the first few years are very informative (Roderick et al.,
199S, 1999~. Using 1995 data as a reference point, the policy
raised the number of students meeting minimum standards in
sixth and eighth grades (by 20 percent and 21 percent respec-
tively, during the first year), and efforts both during the school
year and the summer bridge program contributed. For third
graders, there was no measured improvement during the school
year, but some improvement after the Summer Bridge program.
For all three grades results improved somewhat each year from
1997 to 1999. The students with the weakest skills at the start
gained most. Between 1995 and 1997 the proportion of high-risk
students who were able to meet the cutoff score rose from 4 to
34 percent among sixth graders and from 12 to 49 percent among
eighth graders.
The picture is bleaker for students who were retained in
grade. They did not do better than students who were previ-
ously socially promoted; only one-fourth of the eighth graders
and one-third of the sixth and third graders stayed in the system
and passed the test cutoff at the end of the repeated year. Re-
sults for third graders were particularly troublesome, since the
WHY INFRASTRUCTURE MATTERS
47
OCR for page 48
program appeared to do harm to performance outcomes. Third
graders below the cutoff had on average improved 1.5 grade
equivalents (GEs) when socially promoted, but only 1.2 GEs
when retained. For eighth graders, one-year dropout rates were
higher with retention than with social promotion. Clearly, a
policy that produces positive benefits for some students (those
who meet the minimum standard) imposes very real costs on
others (those who are retained).
The work of the consortium raised important policy ques-
tions, some of which have already sparked a response. Given
the poor results for third graders, the additional support pro-
vided through the Lighthouse and Summer Bridge programs
was extended to first and second graders whose performance
was below grade level. And a search for more effective ways of
addressing the needs of children who fad! to meet the standard
even after a summer program is now under way. The consor-
tium report indicates: "CPS [Chicago Public Schools] has con-
tinued to experiment with alternatives to retention and with
directing resources to students in the second [retained] year. At
present, students in the retained year are provided with sub-
stantial extra resources through Lighthouse, reduced class sizes,
and extra instructional support in schools hit hard by retention.
In our subsequent work, we will be looking specifically at how
these various interventions in the retained year. . . may shape
students' learning" (Roderick et al., 1999:57~.
The ability to follow students from one year to the next
allowed for further insight. Third grade students who improved
enough in the summer program to be promoted made gains in
the next year at about the same inadequate pace as in the previ-
ous year, leaving them at risk of falling behind by the time of the
next test. Students were not on a different learning trajectory;
they simply were given a one-time boost from the summer
program. The report authors write: "taken together, one inter-
pretation of the findings of this report is that the CPS social
promotion policy has worked to reveal a core problem the
adequacy of instruction during the school year. If this is indeed
a problem, then the ultimate success of this policy will depend
upon whether the extra program efforts and extra efforts on the
part of students are matched with an increase in the capacity of
teachers to build early literacy and numerary and to diagnose
and address students' problems when they are not progressing"
(Roderick et al., 1999:57~.
48
STRATEGIC EDUCATION RESEARCH PARTNERSHIP
OCR for page 49
Future work planned at the consortium will include analy-
sis of the instructional impact of the program the extent to
which the focus on raising ITBS scores in reading and math
constrict instructional opportunities as well as a cost analysis.
Providing the Lighthouse and Summer Bridge programs has
positive outcomes, but it is expensive. Would that money be
more or less productive if it were invested in raising the quality
of instruction during the regular school day? Given the empha-
sis nationwide on using high-stakes testing approaches to rais-
ing standards, the potential policy implications of this research
in Chicago are vast.
W O U ~ D S E R P M A K E A D ~ F F E R E N C E ~
The work of the consortium is of tremendous importance
not only to Chicago, but also to all schools particularly those
in large, urban school districts. Despite the success of the con-
sortium in conducting high-quaTity social science research that
is directly applicable to education policy and practice, its Tong-
term viability is in question. Its existence thus far has depended
on three foundations that are based in Chicago. While all have
been generous in their support, foundations do not typically
fund long-term efforts. They expect that if an enterprise is suc-
cessful, it will generate the capacity to be self-sustaining. And as
the foundation leadership changes (as it has recently in all three
of the supporting foundations), new ideas that bear the mark of
the new president can overshadow ideas that emerged under
previous leadership. At the same time that enthusiasm wanes
for an enterprise that is no longer new, cost pressures begin to
rise. Initially the work of the consortium drew on Ph.D. candi-
dates who could be used at relatively Tow wages for purposes of
helping to found something new and important. But to keep
employees as they enter their Tong-term career paths, and as the
institution becomes more established, will require more com-
petitive salaries.
The consortium's leadership is uncertain about future fund-
ing. At the same time, however, they are encouraged by the
expanding possibilities of the work they are undertaking. Their
presence in the schools is more secure and welcomed. With
permission from the school system, they will begin to collect
new classroom-level data that will allow student performance
to be studied in relation to individual programs and teachers.
WHY INFRASTRUCTURE MATTERS
49
OCR for page 50
This will create a much more powerful ability to study learning
and instruction. A growing number of researchers have become
interested in the uniquely rich data collected by the consortium
and they are using it to productive ends. That so clearly valu-
able an effort finds itself in such a precarious position today
speaks strongly to the inadequacy of the existing education
research and development infrastructure.
Moreover, while the consortium's effort to work at the inter-
section of research, policy, and practice has been impressive,
Bryk notes that they are "the only game in town." Chicago's
approach to school reform is very different from approaches
taken in many other districts and states. But we are not learning
how the different approaches compare, because there are no
comparison sites. "If there were a federation of consortia like
ours," says Bryk, "the power of this work would be greatly
magnified. We could really make some progress" (Anthony
Bryk, Professor, University of Chicago, Department of Sociol-
ogy, personal communication, December 2002~.
A federation of field sites could lend power to education
research and development in another respect as well: different
field sites could begin with entirely different foci. The Chicago
consortium began as an effort to draw on one of the three
resources we highlighted in Chapter 1: natural experimenta-
tion. The experiment began with a policy change (decentraliza-
tion). The study of the effects of that change quickly took re-
searchers into issues of school organization (professional
communities), teacher learning (professional development), and
instruction (authentic intellectual work). The questions asked,
however, were shaped by the framing question: What are the
effects of the policy reform?
Other efforts that draw on different resources that take as
their point of departure the insights from disciplines related to
student learning or teacher learning, for example would prob-
ably Took quite different. If the leading questions pertained to
how students improve their reading comprehension or how
they develop mathematical knowledge, the research agenda
would have a stronger focus on the components of knowledge
development and conceptual change. When those questions are
pursued in the school context, teacher knowledge and learning,
as well as the organization and policy influences on the class-
room, are likely to play an important role in the research as well.
50
STRATEGIC EDUCATION RESEARCH PARTNERSHIP
OCR for page 51
With different lead questions and different research exper-
tise, the areas in which our understanding progresses are likely
to be very different. While one agenda would be expected to
yield insights about the locus of decision-making authority in
the school district, the other is more likely to advance the knowI-
edge base on effective reading comprehension instruction. The
synergistic effects of the different efforts, brought together by
the networking efforts of a SERP infrastructure, would lend a
power to education research and development and its ability to
inform policy and practice that is nowhere present today.
CONCLUSION
The above set of cases makes clear the possibility of con-
ducting rigorous research on and for educational practice. But
they also highlight the difficulty of undertaking and sustaining
those efforts in the absence of a new research and development
infrastructure. SERP would facilitate such efforts in the future
by
· Providing a place for researchers and practitioners
interested in research to link up;
· Providing institutional support for negotiating col-
laborations between researchers, school administrators,
and teachers;
· Providing program steering and stable funding to
allow successful efforts to be carried forward;
· Providing research and data collection protocols to
limit the role of happenstance in the production of high-
qualit~v outcomes; and
· Providing regular opportunities for those involved
in education research and development in different sites
to learn from and build on each other's work.
WHY INFRASTRUCTURE MATTERS
51
Representative terms from entire chapter:
education research
