Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
n Ha~ninn Ha ~~a situ since ~es: Y1 Would the substantial investment required to build a new education research and development infra- structure pay off? Proof, of course, cannot be pro- vided by arguments made in advance of such an experiment. But our case for such an investment rests on evi- dence that: (a) the collaborative research and development ef- fort in school settings that we propose is feasible; (b) there are cases in which this type of research and development has been carried out with productive results that are directly applicable to improving classroom practice; and (c) this type of work, even when highly successful, has been difficult to sustain without the proposed research and development infrastructure. To support these claims, we provide a set of illustrative cases below. The cases differ substantially in their details, but all were pioneering efforts; the individuals involved had to find paths through unfamiliar, and at times difficult, terrain. The researchers involved had to become a jack-of-all-trades, able to function in the separate worlds of research and practice, to design and conduct research, develop and maintain partner- ships, and continue to raise substantial funds. And the efforts continue only so long as the individuals who undertook them do not tire and their funders do not shift focus. There are no railroads or highways that have been built in these pioneer's footsteps. Without a supporting infrastructure, their paths did not become well-traveled roads, and the settlements they cre- ated are unlikely to become permanent. WHY INFRASTRUCTURE MATTERS nttarQ 29
BOSTON READING STUDY In 1996 the Boston public school system, with substantial foundation support, introduced a whole school reform program focused on primary grade literacy. Participating schools were required to adopt a structured model for literacy instruction from a menu of four options: Balanced Early Literacy (BEL), Developing Literacy First (DLF), Literacy Collaborative (LC), and Success for All (SFA). In the 1998-1999 school year, 66 schools were involved in the effort. The variation in the pro- grams adopted, as well as the number of schools involved, provided an opportunity to learn a great deal about which programs produce which results, whether the program results differ for children with different demographic and primary lan- guage characteristics, and how teachers and administrators use the programs and their professional development components. A research study was designed to examine these questions in Boston. In 199S, with initial funding from Harvard Universit~v's Interfaculty Initiative, Lowry Hemphill and Terrence Tivnan began pilot work in two schools that later expanded to a study of 16 schools 4 schools using each of the 4 models. The study was designed to examine how students perform on the separate skills that are known to contribute to success in early reading: word reading, word attack, phonemic awareness, writing, and reading comprehension (Tivnan, 2002~. While individual program effects are being studied in this ongoing program of research, so are the differences in instruc- tional practices between and within programs. Data on student performance are collected, along with data on student charac- teristics and characteristics of the instructional program. Impor- tantly, data will be collected over multiple years. Since reading is a skill that is still emerging in first grade, results of the program at the end of second and third grade will be critical to an informative program assessment. Early findings already suggest that many important lessons are being learned about the programs and their role in teaching and learning, as well as about research on classroom practice. With regard to the programs themselves, their implementation has made a measurable difference. The biggest gains were in decoding and in reading sight words. No program could be 30 STRATEGIC EDUCATION RESEARCH PARTNERSHIP
deemed best overall. As one might expect, the programs that spent more time on developing phonemic awareness skills showed better results in tests of this skill, while programs that spent more time on language development showed relative strength in vocabulary and reading comprehension. More im- portantly, however, the differences in outcome for the same program in different classrooms were larger than the differ- ences between programs. This is not an effect that can be attrib- uted to the school, since large differences exist within a school as well. The research team is observing teachers' instructional practices in order to develop and test hypotheses about the contributors to differences in teacher effectiveness. The preliminary results point to another important issue: the programs as a group post substantial gains in bringing students up to grade level in word reading, but they make far less progress in narrowing the yawning gap in reading compre- hension and vocabulary. While the programs differ somewhat in their emphasis on these skills, none provides an adequate response to the existing disparities. For public policy and for the direction of future research and development dollars, this is a very important finding. W O U ~ D S E R P M A K E A D ~ F F E R E N C E ~ The research now under way in these Boston schools is an example of the type of research that can help inform education practice and policy. Indeed, the research team is working with teachers in the schools to provide professional development based on what is being learned about effective instructional practices. The schools involved in the project are eager for input from the researchers both for understanding the program out- comes and for informing instructional practice. But while this case provides optimism regarding the poten- tial of collaborations between researchers and practitioners, it also points to the inefficiencies and disincentives of carrying out this work in the absence of a supportive infrastructure. Without a formal organizational arrangement between the research team and Boston public schools, the researchers themselves must negotiate arrangements with each of the 16 schools. Those ar- rangements depend on personal relationships, and with each personnel change, the relationship must be renegotiated. The researchers also needed to obtain the parental permissions re- WHY INFRASTRUCTURE MATTERS 31
quired to collect information on individual students for this type of research. At the same time, the researchers have needed to secure the funds required to carry the project forward. The funding has come in one- or two-year commitments, requiring persistent attention to negotiating the next research grant or contract. In the early stages of the project, the researchers were also re- quired to negotiate with the developers of each of the four programs regarding the outcome measures that would be used to assess program impact. If SERP were in existence, funding, research protocols and instruments, and the terms of access to schools would still need to be negotiated, and permission to collect data on individual students would still need to be obtained. But an organization could develop the capacity to do these much more efficiently by institutionalizing the knowledge and skills involved and mak- ing routine what otherwise must be reinvented by each research team. Many outstanding researchers would be unable or unwilling to undertake the Herculean efforts of the Boston research team, discouraging this type of badly needed investi- gation. The role of a SERP enterprise is not just facilitative. It would develop and steer a program that could make more of research findings. The Boston researchers are keenly aware of the oppor- tunities lost by working in isolation. While many other jurisdic- tions around the country are using the same literacy programs in different contexts, the lack of a coordinated effort means that they are learning less about the features of those contexts that contribute to outcomes. The confidence in particular outcomes would be strengthened or called into question if results from other sites could be compared. But such a comparison requires an extent of coordination in research design that does not now exist. Hemphill and Tivnan write up their results and present them at research conferences. They point out, however, that the operative norms at these meetings are not those of a network of researchers engaged in an effort to collectively advance a field, but rather those of a professional competition that minimizes the opportunity for productive collaboration. In their view, the role of a SERP network in fostering an environment in which the operative norms are those of productive collaboration would make a significant contribution to the productivity of work like 32 STRATEGIC EDUCATION RESEARCH PARTNERSHIP
theirs. The benefits of that collaboration, of course, would ex- tend beyond researchers. A critical role for SERP would be one of supporting collaboration and shared knowledge among the school systems and teachers undertaking reforms to address similar problems (based in part on phone conversation with Lowry Hemphill, November 2002~. · ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ RECIPROCAL TEACHING As the Boston reading study results confirm, many students who successfully learn to read nonetheless do poorly at reading comprehension. Instructional approaches to improving reading comprehension primarily fall under the category of strategy instruction (RAND, 2002a). The skills taught in strategy instruc- tion generally target improved recall of text, teaching students to attend to headings, to outline or map the text in graphic form, and to reread for specific information or structural cues. While these strategies do improve recall, particularly for low-achiev- ing students (National Institute of Child Health and Human Development, 2000), they focus on surface features of the text. They can therefore be mastered successfully without the stu- dent understanding the meaning of the text or integrating the new knowledge from the text with their existing understand- ings. Reciprocal Teaching (RT) is a technique developed by Annemarie Palincsar and Anne Brown two decades ago to en- gage students more deeply in understanding the meaning of text. The active processes of making sense of a text involved in skilled reading comprehension are taught to students explic- itly.1 The teacher initially models four strategies: questioning unclear content, summarizing meaning paragraph by paragraph, clarifying comprehension problems, and predicting what will come next. Students practice the strategies with guidance from the teacher, and, as their skill increases, the teacher increases the demands. Gradually the role of the teacher diminishes as stu- dents become more competent and sophisticated in the ques- tioning and monitoring role. In groups, students ask each other For more detail, see the companion report, Learning and Instruction: A SERP Research Agenda. WHY INFRASTRUCTURE MATTERS 33
questions, practicing aloud the type of dialogue that will even- tually become internal. The technique has been used to improve listening compre- hension among young children, as well as to improve reading comprehension once children become fluent readers. Children who are exposed to the reciprocal teaching intervention per- formed better than control children on several dimensions, in- cluding the quality of summaries and questions and scores on criterion tests of comprehension (Palincsar and Brown, 1984~. Gains were maintained over time, generalized to classroom comprehension tests, and transferred to novel tasks involving summarizing, questioning, and clarifying. W O U ~ D S E R P M A K E A D I F F E R E N C E ~ Reciprocal Teaching has demonstrable effects on a problem that is at the heart of effective education. The ability to compre- hend text unlocks knowledge in all fields. But without an infra- structure to nurture the program through further stages of de- velopment and integration into the classroom, Reciprocal Teaching has largely remained a small-scale effort in the hands of a dedicated researcher whose work has moved into new areas. The absence of any infrastructure to carry the program forward has repercussions that Annemarie Palincsar describes vividly: Believe it or not, after all these many years, I still get requests (at least two a month) to conduct professional development regard- ing reciprocal teaching. I always feel badly saying no (in part because a very important reason for doing the research was to inform practice), and I really have no one to whom I can refer the school personnel with confidence that the version of RT that they will describe/demonstrate is consistent with the original RT.... Because there is currently nothing like SERF, I had no systematic way to disseminate RT on a large scale nor to engage in profes- sional development that would reach the numbers of schools and districts that have requested it. · · ~ This is what I did do.... I authored a facilitator's manual (that has been sent to literally thousands of folks who have requested it). I also prepared a videotape that provides an overview of RT, excerpts of a teacher implementing RT, and debriefing conversa- tions with students who have learned how to use RT. I purchased a tape-to-tape video recorder and when people ask for this video tape, I ask them to send me a blank VHS tape in a stamped, self- 34 STRATEGIC EDUCATION RESEARCH PARTNERSHIP
addressed padded envelope and I make them a copy of this tape. I have lost count of how many copies I have made but I should be embarrassed to send any more since it is so dated! What role might a SERP have played? Well, first, it would have been wonderful to have had high-quality video and audio of teachers working in different contexts (demographically as well as content-wise) using RT in a manner that reflects the principles on which it was designed. Second, it would have been so satisfying to never have had to say "no" to a professional development request because there was a network of teacher leaders or professional development person- nel who would either work onsite or who would have offered teaching institutes to support educators and educational leaders to learn about RT (and other forms of comprehension instruc- tion). Third, it would have been ideal to be able to have supported conversations with other researchers who were similarly investi- gating class-wide models of comprehension instruction to talk about how our work was complementary and where it differed and the implications of these differences for professional develop- ment. For example, what support can we provide educators who are trying to choose among QtA [Questioning the Author] (Beck et al., 1997), Collaborative Reasoning with Text (Chine and Anderson and colleagues), SAIL (Pressley et al., 1989) and RT? How might we have done our research more synergistically so that we might have learned more from our respective programs of research? I can imagine an entity like SERP playing such a role. PaTincsar describes an effort by a major commercial publish- ing company to conduct professional development on RT, with neither her permission (which is not required), nor her input. Her understanding is that the version of RT taught in the train- ing program is fundamentally different from hers in critical dimensions. But no one at the company has returned her phone calls. She writes: Fourth. . . it would have been terrific to have some means of working more closely with commercial endeavors, so that these efforts do not undermine the research and development the publisher is trying to disseminate. A major national goal expressed in the No Child Left Behind legislation is effective reading instruction for all children. Mil- lions of dollars are earmarked to support the effort. But without WHY INFRASTRUCTURE MATTERS 35
a research and development infrastructure, one of the few inter- ventions that has been demonstrated to improve reading com- prehension outcomes will reach children only if a teacher learns about the program, contacts Annemarie PaTincsar and mails a self-addressed, stamped enveloped with a blank tape inside (Annemarie Palincsar, Charles Waigreen Professor of Reading and Literacy, School of Education, University of Michigan, per- sonal communication, November 2002~. · ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ CREATING THE COGNITIVE TUTOR The Cognitive Tutor Algebra I is one of a set of "cognitive tutors" developed at Carnegie Mellon University to teach aige- bra and geometry. Of great relevance to the SERP vision, the tutors are a good illustration of how to make the transition from the laboratory to the classroom, as well as the nature of the partnership between researchers and teachers that have made the program a success. The work at Carnegie Mellon began as a project to see whether a computational theory of thought, called ACT (Ander- son, 1983), could be used as a basis for delivering computer- based instruction in algebra. The ACT theory of problem- solving cognition is the basis for modeling students' algebra knowledge. These models are capable of generating almost any sensible solution to an algebra problem. They are embedded in a computer program that can then identify the particular ap- proach a student is taking to a solution. The cognitive models enable two sorts of instructional re- sponses that are individualized to students: 1. By a process called mode! tracing, the program will infer how a student is going about problem solving and generate appropriate help and instruction when a student is pursuing an unproductive or incorrect strategy. 2. By a process called knowledge tracing, the program will infer where a student falls in the learning trajec- tory (what knowledge has been mastered and what is insecure) and select instruction and problems appropriately. 36 STRATEGIC EDUCATION RESEARCH PARTNERSHIP
Developing cognitive models that accurately reflect student competences and developing appropriate instructional responses is very much an iterative process. The success of the tutors depends on a design-test-redesign effort in which models are assessed for how well they capture competence and in which instructional responses are assessed for effectiveness. In controlled trials, the curriculum performs well. It was found that students could go through the algebra curriculum with the Tutor in a third of the time normally required. In carefully managed classrooms, students would show about a standard deviation (approximately one letter grade) improve- ment in achievement (Anderson et al., 1995~. In real classroom situations, the impact of the tutors tends not to be as large, varying from O to 1 standard deviation across 13 evaluations. Another third-part~v evaluation focused on the social conse- quences of the tutors; it documented large motivational gains resulting from the active engagement of students and their suc- cessful experiences on challenging problems (Schofield et al., 1990~. Unlike many such small-scale success stories in cognitive science, this project was able to grow to the point at which the cognitive tutors now are used in 33 of the 100 largest school districts in the United States and are interacting with about 200,000 students yearly. A number of features were critical to making this successful transition: 1. While the ACT theory provided the foundation for the program, there was a concerted effort to identify a curriculum that educators wanted taught in the class- room. In particular, there was a major effort to teach a curriculum that was in compliance with the National Council of Teachers of Mathematics standards (Na- tional Council of Teachers of Mathematics, 1989~. 2. A curriculum was designed that teachers would ac- cept and could implement. The curriculum design was largely the product of teachers with experience in urban classrooms. To meet their needs, a full-year curriculum was developed rather than an enrichment program to be inserted into an existing curriculum. And the computer tutors were used as a support rather than a replacement for the teachers. In this curriculum WHY INFRASTRUCTURE MATTERS 37
students spend 40 percent of their time with the com- puter tutors and 60 percent of their time with other activities. These classroom activities help them transi- tion to their lessons with the tutor, as well as to transi- tion from the tutor to things that they will have to do in the real world. 3. A structure was set up for supporting the use of the curriculum and tutors. Before introducing the tutors into a classroom, it has been important to provide professional development time to enable teachers to prepare for the change they are about to experience. A center at Carnegie Mellon was set up for responding to teacher and school problems. As the adoptions grew, a separate company (Carnegie Learning Corpo- ration) was created to perform this function and main- tain and adapt the materials. 4. Ultimately, such a curriculum must be financially self- sustaining, and the program was developed from the beginning with a plausible financial model in mind. In particular, by offering a full grade 9-11 curriculum, it was possible to earn the kind of income from sales that is necessary to sustain the activity. W O U ~ D S E R P M A K E A D I F F E R E N C E ~ The Cognitive Tutor Algebra I represents a success of the type that is rare in K-12 education. Its developers, however, point to the problems raised when an effort like this is under- taken without a research infrastructure: Once leaving the laboratory, there have been only haphazard efforts to evaluate the curriculum as it has multiplied through the school systems. It is only now that our tutors are about to receive their first adequate third-party evaluation and this is only because of the funding from the Hewlett Foundation. There are natural 38 reasons to avoid rigorous evaluation of material. Early in the development of a program a negative evaluation may make it difficult to get the next round of funding. Once a product be- comes commercial as ours did, there is even less incentive for such evaluations because in addition to bringing potentially bad news, they cost and so threaten the need to meet the next month's payroll. Mechanisms need to be set up to both require and fund rigorous formative evaluations in the development of curricula, and impartial third-party evaluations of curricula once they start to be disseminated. STRATEGIC EDUCATION RESEARCH PARTNERSHIP
And despite its success, Anderson argues that the Cognitive Tutor Algebra I has room for improvement in some very impor- tant dimensions. Early in the development of the algebra tutor, a decision was made to place a heavy emphasis on contextualizing algebra to help students make the transition to the formalism. This has been successful and there are fewer students dropping out. However, as a consequence the curriculum does not achieve the fluency in symbol manipulation and abstract analysis expected for high- achieving students. There is no reason why the cognitive tutors could not be extended to these topics but they have not (from Learning and Instruction: A SERP Research Agenda, Box 3.3~. · ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ THE COGNITIVE TUTOR ALGEBRA I IN AN OKLAHOMA SCHOOL DISTRICT Pat Morgan, mathematics coordinator for the Moore Inde- pendent School District in Oklahoma, knew Carnegie Learning Corporation's Cognitive Tutor Algebra I had been rated favor- ably by the U.S. Department of Education. She thought it was worth a try in her school district but knew from experience that her teachers were likely to balk at being asked to do something new. She decided that, to get their support, she would need to show them that the new program worked better than their current program. Her plan was to introduce the Cognitive Tutor Algebra I in a subset of the algebra classrooms, and compare the results to those in the classrooms that continued to use McDougal Littell's Heath algebra I text. With a Goals 2000 grant from the U.S. Department of Educa- tion, Pat was able to purchase the program and pay teachers to attend training workshops during the summer of 2000. In Sep- tember, the Cognitive Tutor Algebra I was introduced in five middle schools in the district. Students in the honors algebra class were not involved in the study. Other students had al- ready been assigned more or less randomly to classes, and teachers who had undergone training were asked to teach both traditional classes and Cognitive Tutor Algebra I classes so that the effect of the teacher could be separable from the effect of the program. In order to compare outcomes, Pat decided to use a WHY INFRASTRUCTURE MATTERS 39
standardized, end-of-course assessment developed by the Edu- cational Testing Service (ETS).2 At the end of the school year, scores were obtained for 220 students who received traditional algebra instruction and 224 who received instruction with the Cognitive Tutor Algebra I. The data were turned over to the Carnegie Learning Corpora- tion for analysis. Students who received traditional instruction earned a mean scaled score of 15.1 (with a standard deviation of 5.5~; students instructed with Cognitive Tutor earned 16.7 (with a stand deviation of 5.7~. This indicates a benefit from the Cog- nitive Tutor Algebra I that is significant (F (1,442) = S.S, p < 0.01~. When student performance was broken down by category- minimal, basic, proficient, and advanced the nature of the gains is more apparent. The Cognitive Tutor Algebra I reduced substantially the number of students who performed at the lowest levels and almost doubled the percentage who were proficient. It had no effect, however, on those who scored at the advanced level. Grade point averages were higher for students who used the tutor, as were scores on a student attitude survey. With the exception of the honors algebra classes, teachers in all algebra classes in the school district now use the Carnegie Learn- ing curriculum willingly. W O U ~ D S E R P H A V E M A D E A D ~ F F E R E N C E ~ The introduction of the Cognitive Tutor Algebra I in the Moore school district suggests that a well-designed test of whether a curriculum improves student outcomes can stimulate productive change. It also suggests that the conditions for such a study, including random assignment and intrateacher pro- gram comparisons, can be created in normal school settings. But for school district personnel to manage such an effort without the support of a research infrastructure requires an extraordinary effort. While Pat Morgan undertook that effort, it was not one she could sustain. The answers provided by the study were helpful, but Pat and her math teachers would like to know whether the gains for the Cognitive Tutor Algebra I stu- dents are sustained in higher-level math courses. And she ob- serves that teachers using the curriculum are beginning to adapt 2The test scores are reported on a 0-50 scale, and the national mean scores for the select group of students who take this test is 18. 40 STRATEGIC EDUCATION RESEARCH PARTNERSHIP
it somewhat. She would like to understand the consequences of the adaptations. But with tight budgets, she cannot afford to have ETS administer the end-of-year test anymore. The school district is now also experimenting with Carnegie Learning's Cognitive Tutor Algebra II and with the Cognitive Tutor Geom- etry on a small scale, but they are not conducting a similar study. While Pat would like to continue to study program ef- fects to better inform her decisions, her job as the math coordi- nator is very demanding, and she cannot undertake the effort to raise funds and conduct additional studies on her own again. Even more than the money, she says, she needs help so that the unfamiliar job of researcher does not fall fully on her shoulders. Pat Morgan's research design was better than she knew. She had no background in research. She just believed from experi- ence that the teacher is the most important contributor to stu- dent achievement, so she decided that she needed to have the same teachers using both curricula. Students who were not assigned to honors algebra had been placed without tracking into other sections of algebra before a decision was made re- garding which teachers would introduce the new curriculum in which of their classes, so random assignment was happenstance. Once the data were collected, Pat wasn't sure how to make the best use of them, so she called Carnegie Learning and found help on that score. She provided one of the best tests of the curriculum to date, as well as a valuable source of information for other school districts considering the program, although that outcome was not by design. A research and development infrastructure prepared to sup- port efforts like this and to guide research design could make more commonplace what is now the outcome of the combina- tion of happenstance and extraordinary effort by a very dedi- cated and insightful school administrator. It could also make the very instructive finding from one school district easily avail- able to other districts, so that thousands of other Pat Morgans could persuade their teachers of the value of trying a new instructional approach (Pat Morgan, mathematics coordinator, Moore Independent School District, Moore, Oklahoma, personal communication, November 2002~. WHY INFRASTRUCTURE MATTERS 4
LINKING RESEARCH AND PRACTICE WITH EASE One Wednesday morning in 199S, Catherine Snow arrived in her office to find a large box. It contained many sheaves of test data, summarized in a few tables, and a note that read something like this: I collected these data in the Title I program of the White Bear Lake Schools we designed an intervention program based on your findings from the Home-School Study of Language and Literacy Development. The results seem to suggest it worked. But I don't know how to do the right analyses, so I am sending you the data. Snow got in touch with the source of the note and the data, Gail Jordan, who was then Title I director in White Bear Lake. Gail is a gifted curriculum designer and teacher educator, with a commitment to using research results to inform practice. Gail had taken seriously the correlational findings reported by Snow and her colleagues suggesting that preschool and kindergarten- age children in Tow-income families did better in literacy learn- ing if they had had rich linguistic interactions with their par- ents. The helpful interactions that the researchers described included telling stories, reading books and engaging in discus- sion about them, giving explanations, using rich vocabulary, and engaging in pretend play. Relying heavily on the research findings, Gail figured out how to teach parents to engage in these sorts of interactions and how to design activities that kindergarten teachers could assign as homework that would provide occasions for the parent. She also met with the research team before designing the program. With a Ph.D. under her belt, Gail also designed and carried out a random assignment study randomly assigning kinder- garten classrooms to treatment and control conditions, pretest- ing all the children in order to be able to control for initial status, and incorporating ways of assessing how many of the activities parents engaged in. She was engaging in precisely the kind of problem-oriented, practice-embedded research that is needed to improve education, and she successfully designed and car- ried out a very sophisticated study. She did not, however, know how to analyze the results or write up the findings for broader 42 STRATEGIC EDUCATION RESEARCH PARTNERSHIP
dissemination. Fortunately, she passed along the data, rather than just leaving them in the corner of her office. Snow had some uncommitted funds for doctoral student support available, so she hired Michelle Porche to analyze the dataset. Michelle's analysis confirmed, as Gail's preliminary look at the findings had suggested, that the group that had received the intervention showed greater gains in language skills over the course of their kindergarten year and, further- more, that those gains were greatest for the children who had started with the weakest language skills. Gail Jordan subse- quently visited Snow and Porche, and they worked together on writing up the paper. It was published in Reading Research Quar- terly as "Project EASE: Easing Children's Transition to Kinder- garten Literacy Through Planned Parent Involvement." It won the International Reading Association award for the best paper published in that journal in the year 2000. Porche also helped Jordan build a web site describing EASE and providing re- sources for those who wanted to replicate it; EASE is now being used widely in Tow-income districts in the United States, and it has been incorporated into a state literacy reform initiative in Ohio (where it is again being evaluated). WO U ~ D S E R P H AV E MA D E A D I F F E R E N C E ~ The outcome in the EASE case was a very positive one from all standpoints. The important work done both by a practitioner and by researchers came together in a way that allowed both practice, and the knowledge base, to advance. But while the success is worthy of celebration, it is disquieting that so many of the critical events were a matter of chance, as Snow makes clear: Many events had to converge to enable EASE to be disseminated and implemented outside White Bear Lake. First, Gail Jordan is more focused on the possibilities of research than many practitio- ners though of course in an ideal world every educational innovation initiated locally would be subjected to a systematic evaluation. Second, Gail and I had met before, so she felt she could send me the data. Third, I happened to have some uncom- mitted funds that could support analysis of the data. Fourth, I happened to have a doctoral student interested in parent involve- ment and in literacy, who could thus easily be recruited to be involved in this project. Fifth, the findings were of sufficient interest that Reading Research Quarterly was willing to publish them. Sixth, the intervention was designed in such a way that it WHY INFRASTRUCTURE MATTERS 43
was feasible for others to replicate it without much adaptation. If any one of these factors had been different, this valuable educa- tional intervention with its potential to improve children's literacy success would never have seen the light of day. Gail Jordan also notes how easily the value of this work could have been compromised. It was a challenge to keep the integrity of the project because the planning team had no research background and there were many times that there were confounding suggestions made (like making experimental classrooms full day and control classrooms half day). There were also concerns about the amount of testing required. . . . It would have been wonderful to have a research team guide us in those key decisions.... Our success was truly due to the kindness of strangers, specifically the research team at Harvard (Gail Jordan, personal communication, December 2002~. Surely there are many other cases in which locally designed innovations remain local and person-specific because some or all of the chance events were absent. The purpose of SERP is to make what is now an extraordinary outcome much more com- monplace. It would do so by providing the infrastructure to connect practitioners like Gail Jordan to senior researchers like Catherine Snow and more junior researchers like Michelle Porche. It would provide support for the design of an interven- tion to assure replicability, as well as for the design of program evaluation to ensure rigor, relying less on extraordinary capaci- ties of those in Gail lordan's role. In a world with a well-functioning SERP organization, progress would not require that the developer of the interven- tion have the capacity to design a random assignment trial on her own, as was true in this case. And SERP would have exten- sive capacity for data analysis so that high-quality data collec- tion efforts, to be used, would not require the good fortune of a researcher with financial and human capital to spare. Finally, SERP would engage the effort to make the EASE results avail- able more widely to those who are unlikely to read a report of published research (Catherine Snow, Henry Lee Shattuck Pro- fessor of Education, Harvard Graduate School of Education, personal communication, December 2002~. 44 STRATEGIC EDUCATION RESEARCH PARTNERSHIP
· ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ CONSORTIUM ON CHICAGO SCHOOL RESEARCH Perhaps the most powerful evidence to suggest the possibil- ity and the value of research on practice, conducted in school settings as collaborative efforts among researchers, practitio- ners, and policy makers, comes from the Consortium on Chi- cago School Research. The consortium began in 1990 under the leadership of Anthony Bryk as an effort to study the impact of a major school reform effort passed into law in Chicago in 1988. The mission of the consortium is to undertake research of high technical quality that can inform education policy making and school improvement efforts. The 1988 Chicago school reform decentralized authority and accountability in the schools. It established local school councils (LSCs) for that purpose, comprised of the school's principal, two teachers, and six elected parents.3 The LSCs approved the budget and held authority over the principal's contract. All schools were required to develop, implement, monitor, and update annually a school improvement plan (SIP), with LSC participation and oversight (Consortium on Chicano School Re- search, 2003~. The consortium's initial task was to study what happened in the wake of the reform. From the outset, this required compre- hensive data collection. Since 1991 the consortium has con- ducted biannual surveys of students, teachers, and principals. Through an agreement with the Chicago Public Schools, its archives include the following: · Test score data for the Iowa Tests of Basic Skills (ITBS), tests of academic proficiency, the Illinois Goal Assessment Program, and the Illinois Standards Achieve- ment Test (beginning in the late 1980s); · Administrative history information (as of 1992~; · Grade files from all high school students (as of 1993~. As part of a five-year grant from the Chicago Annenberg Research Project, the consortium also collected extensive data at 3In high schools, a student representative joined the group. WHY INFRASTRUCTURE MATTERS 45
24 schools on teacher assignments, samples of student work, and extensive, in-depth interviews and classroom observations. As the reforms in Chicago evolved, including a shift of authority from school boards to the mayor in 1994-1995 on a temporary basis, the consortium has continued to amass the data that will allow the effects to be teased out over time. The availability of this rich dataset has spawned many studies over the decade of importance for education policy. They have ad- vanced understanding of schools as organizations and of the conditions that foster school improvement, providing critical insights on the effects of high school size, intraschoo! teacher relationships, and the cognitive demands placed on students by teachers. The consortium has also worked with schools to provide them with data, and an approach to interpreting it, that give schools greater insight into their own functioning and perfor- mance. For example, the consortium's work allows each school with an eighth grade class to look at how its graduates perform over the course of the next five years. But the exchange between researchers and teachers is bidirectional. Important findings in studies of relational trust among school personnel have origi- nated from the insights of teachers who felt that this played a major role in the performance of a school's students. Many of the studies done by the consortium could serve to illustrate research that has provided critical knowledge and insights for policy makers, practitioners, or both. For example, the enactment of legislation in 1996 ending social promotion in the Chicago Public Schools set minimum scores in math and reading that students must achieve on the ITBS in grades 3, 6, and ~ in order to be promoted. Students who failed to meet the cutoff were required to attend a summer school program and retake the test at program completion. Students who fail again to meet the standard are retained in grade. The Chicago policy was designed to address problems faced by all school districts. Many students are having difficulty in later grades, particularly high school, because they lack basic skills. Teachers are being asked to teach to higher standards. But many believe that the students who appeared in their class- rooms do not have the skills to move on to more advanced work. The "theory of action" in the legislation, according to the consortium research team, is three pronged (Roderick et al., 46 STRATEGIC EDUCATION RESEARCH PARTNERSHIP
1999~. First, before students are tested, they, their parents, and their teachers face new incentives. When students are confronted with the prospect of being retained in grade, they are motivated to work harder, and their parents are motivated to monitor the student's performance more closely. Teachers are sent a strong message to focus attention on students who are not mastering basic skills and to emphasize those skills in their teaching. To improve the opportunity to succeed, students who are at risk of failure are given extended instructional time through Light- house, an after-school program begun in 1997. The second prong provides an opportunity for a second chance. If a student fails to meet the minimum standard at the end of the school year, the summer bridge program offers addi- tional, more focused instruction. The theory is that many stu- dents who fad! initially can be brought up to speed with this additional opportunity. Finally, a second failure to meet the standard is met with retention in grade. The theory here is that students who repeat the material yet again will master it and move on to the next grade better prepared. A research team at the consortium set out to test each of these theories. A full analysis will require data collection over a longer period of time to ascertain Tong-term effects, but results from the first few years are very informative (Roderick et al., 199S, 1999~. Using 1995 data as a reference point, the policy raised the number of students meeting minimum standards in sixth and eighth grades (by 20 percent and 21 percent respec- tively, during the first year), and efforts both during the school year and the summer bridge program contributed. For third graders, there was no measured improvement during the school year, but some improvement after the Summer Bridge program. For all three grades results improved somewhat each year from 1997 to 1999. The students with the weakest skills at the start gained most. Between 1995 and 1997 the proportion of high-risk students who were able to meet the cutoff score rose from 4 to 34 percent among sixth graders and from 12 to 49 percent among eighth graders. The picture is bleaker for students who were retained in grade. They did not do better than students who were previ- ously socially promoted; only one-fourth of the eighth graders and one-third of the sixth and third graders stayed in the system and passed the test cutoff at the end of the repeated year. Re- sults for third graders were particularly troublesome, since the WHY INFRASTRUCTURE MATTERS 47
program appeared to do harm to performance outcomes. Third graders below the cutoff had on average improved 1.5 grade equivalents (GEs) when socially promoted, but only 1.2 GEs when retained. For eighth graders, one-year dropout rates were higher with retention than with social promotion. Clearly, a policy that produces positive benefits for some students (those who meet the minimum standard) imposes very real costs on others (those who are retained). The work of the consortium raised important policy ques- tions, some of which have already sparked a response. Given the poor results for third graders, the additional support pro- vided through the Lighthouse and Summer Bridge programs was extended to first and second graders whose performance was below grade level. And a search for more effective ways of addressing the needs of children who fad! to meet the standard even after a summer program is now under way. The consor- tium report indicates: "CPS [Chicago Public Schools] has con- tinued to experiment with alternatives to retention and with directing resources to students in the second [retained] year. At present, students in the retained year are provided with sub- stantial extra resources through Lighthouse, reduced class sizes, and extra instructional support in schools hit hard by retention. In our subsequent work, we will be looking specifically at how these various interventions in the retained year. . . may shape students' learning" (Roderick et al., 1999:57~. The ability to follow students from one year to the next allowed for further insight. Third grade students who improved enough in the summer program to be promoted made gains in the next year at about the same inadequate pace as in the previ- ous year, leaving them at risk of falling behind by the time of the next test. Students were not on a different learning trajectory; they simply were given a one-time boost from the summer program. The report authors write: "taken together, one inter- pretation of the findings of this report is that the CPS social promotion policy has worked to reveal a core problem the adequacy of instruction during the school year. If this is indeed a problem, then the ultimate success of this policy will depend upon whether the extra program efforts and extra efforts on the part of students are matched with an increase in the capacity of teachers to build early literacy and numerary and to diagnose and address students' problems when they are not progressing" (Roderick et al., 1999:57~. 48 STRATEGIC EDUCATION RESEARCH PARTNERSHIP
Future work planned at the consortium will include analy- sis of the instructional impact of the program the extent to which the focus on raising ITBS scores in reading and math constrict instructional opportunities as well as a cost analysis. Providing the Lighthouse and Summer Bridge programs has positive outcomes, but it is expensive. Would that money be more or less productive if it were invested in raising the quality of instruction during the regular school day? Given the empha- sis nationwide on using high-stakes testing approaches to rais- ing standards, the potential policy implications of this research in Chicago are vast. W O U ~ D S E R P M A K E A D ~ F F E R E N C E ~ The work of the consortium is of tremendous importance not only to Chicago, but also to all schools particularly those in large, urban school districts. Despite the success of the con- sortium in conducting high-quaTity social science research that is directly applicable to education policy and practice, its Tong- term viability is in question. Its existence thus far has depended on three foundations that are based in Chicago. While all have been generous in their support, foundations do not typically fund long-term efforts. They expect that if an enterprise is suc- cessful, it will generate the capacity to be self-sustaining. And as the foundation leadership changes (as it has recently in all three of the supporting foundations), new ideas that bear the mark of the new president can overshadow ideas that emerged under previous leadership. At the same time that enthusiasm wanes for an enterprise that is no longer new, cost pressures begin to rise. Initially the work of the consortium drew on Ph.D. candi- dates who could be used at relatively Tow wages for purposes of helping to found something new and important. But to keep employees as they enter their Tong-term career paths, and as the institution becomes more established, will require more com- petitive salaries. The consortium's leadership is uncertain about future fund- ing. At the same time, however, they are encouraged by the expanding possibilities of the work they are undertaking. Their presence in the schools is more secure and welcomed. With permission from the school system, they will begin to collect new classroom-level data that will allow student performance to be studied in relation to individual programs and teachers. WHY INFRASTRUCTURE MATTERS 49
This will create a much more powerful ability to study learning and instruction. A growing number of researchers have become interested in the uniquely rich data collected by the consortium and they are using it to productive ends. That so clearly valu- able an effort finds itself in such a precarious position today speaks strongly to the inadequacy of the existing education research and development infrastructure. Moreover, while the consortium's effort to work at the inter- section of research, policy, and practice has been impressive, Bryk notes that they are "the only game in town." Chicago's approach to school reform is very different from approaches taken in many other districts and states. But we are not learning how the different approaches compare, because there are no comparison sites. "If there were a federation of consortia like ours," says Bryk, "the power of this work would be greatly magnified. We could really make some progress" (Anthony Bryk, Professor, University of Chicago, Department of Sociol- ogy, personal communication, December 2002~. A federation of field sites could lend power to education research and development in another respect as well: different field sites could begin with entirely different foci. The Chicago consortium began as an effort to draw on one of the three resources we highlighted in Chapter 1: natural experimenta- tion. The experiment began with a policy change (decentraliza- tion). The study of the effects of that change quickly took re- searchers into issues of school organization (professional communities), teacher learning (professional development), and instruction (authentic intellectual work). The questions asked, however, were shaped by the framing question: What are the effects of the policy reform? Other efforts that draw on different resources that take as their point of departure the insights from disciplines related to student learning or teacher learning, for example would prob- ably Took quite different. If the leading questions pertained to how students improve their reading comprehension or how they develop mathematical knowledge, the research agenda would have a stronger focus on the components of knowledge development and conceptual change. When those questions are pursued in the school context, teacher knowledge and learning, as well as the organization and policy influences on the class- room, are likely to play an important role in the research as well. 50 STRATEGIC EDUCATION RESEARCH PARTNERSHIP
With different lead questions and different research exper- tise, the areas in which our understanding progresses are likely to be very different. While one agenda would be expected to yield insights about the locus of decision-making authority in the school district, the other is more likely to advance the knowI- edge base on effective reading comprehension instruction. The synergistic effects of the different efforts, brought together by the networking efforts of a SERP infrastructure, would lend a power to education research and development and its ability to inform policy and practice that is nowhere present today. CONCLUSION The above set of cases makes clear the possibility of con- ducting rigorous research on and for educational practice. But they also highlight the difficulty of undertaking and sustaining those efforts in the absence of a new research and development infrastructure. SERP would facilitate such efforts in the future by · Providing a place for researchers and practitioners interested in research to link up; · Providing institutional support for negotiating col- laborations between researchers, school administrators, and teachers; · Providing program steering and stable funding to allow successful efforts to be carried forward; · Providing research and data collection protocols to limit the role of happenstance in the production of high- qualit~v outcomes; and · Providing regular opportunities for those involved in education research and development in different sites to learn from and build on each other's work. WHY INFRASTRUCTURE MATTERS 51
