Click for next page ( 4


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 3
of methodological questions to intense consideration of the structure of public education in the United States. As the symposium planners intended, the presentations and discussion focused not on achieving group consensus, but on unearthing a variety of views on a complex topic. (See Appendix C for a list of the papers presented.) Clearly the day and a half allotted for the symposium did not allow for an exhaustive discussion of either the strengths and weaknesses of TIMSS or its many implications for policy makers. Moreover, because of time constraints, a number of important points were raised but not elaborated during the discussion. This summary report is an additional component of the effort to foster dialogue in the education research and policy communities. It describes the major elements of TIMSS, presents some of the discus- sion that took place at the symposium, and explores the themes that emerged from it. Because TIMSS is so complex, the steering com- mittee charged with planning the symposium decided to devote con- siderable symposium time to explication of the structure of the study and a few of its principal findings. This document follows that lead. The next section, "What Is TIMSS?," provides a description of the study and of the presentations made by the TIMSS researchers. The following two sections summarize, respectively, the questions and critiques that presenters raised about the study itself and the major policy issues that were addressed. The last section summarizes the major ideas that emerged at the symposium. WHAT IS TIMSS? As its name indicates, TIMSS is the third in a series of investiga- tions of mathematics and science learning conducted under the aus- pices of the International Association for the Evaluation of Educa- tional Achievement. IEA is an international consortium of research institutions in more than 40 countries. Although individual govern- ments may fund their countries' participation in IEA activities, the organization is run by an assembly of country representatives. The first IEA study, of mathematics, was conducted in the 1960s; the second mathematics study was done in the 1970s. IEA has also conducted studies of learning in a variety of other subjects. Although the structure and composition of IEA's studies have evolved some since the 1960s, their purpose to describe and explain differences in student achievement has remained the same. More specifically, the organizers of the study described the pur- pose of TIMSS in this way: "to learn more about mathematics and science curricula and teaching practices associated with high levels of student achievement, in order to improve the teaching and the learning of mathematics around the world" (Robitaille and Garden, 1996:15). Study planners recognized that to accomplish this goal they would need to collect a variety of different kinds of data. First, they needed the kind of common measure of achievement used in previous studies numbers that would represent the varying degrees RESULTS OF THE THIRD INTERNATIONAL MATHEMATICS AND SCIENCE STUDY 3

OCR for page 3
to which students around the world have learned the body of math- ematics and science knowledge deemed (through international con- sensus) essential. This was obtained by means of an achievement test (described in greater detail below). All of the other components of TIMSS were designed to provide data that can help explain variations in performance on the achievement test: these included a detailed look at the content of mathematics and science curricula and text books around the world, as well as investigations of student attitudes and experiences, teaching practices and school resources, and many other factors that affect achievement (these other components of the study are described below). The challenge for TIMSS researchers, and for others wishing to use the data for additional analyses, is to make full use of this combination of information about the education practices and contexts that influence student learning. The scope of TIMSS is unprecedented in several ways. Though many international comparative assessments have been conducted, none has assessed student learning in two subjects in so many countries at the same time. Those involved in the planning and design of the study paid considerable attention to the experience gained in the study's predecessor, the Second International Mathematics Study (SIMS) (McKnight et al., 1987; Medrich and Griffith, 1992~. They addressed many of the criticisms leveled at SIMS, both by adhering to strict sampling procedures and by expanding the scope of the design for TIMSS to include the collection of an extensive variety of contextual data (Rotberg, 1990; Bracey, 1996; Third International Mathematics and Science Study, 1996~. In addition, the designers of TIMSS incor- porated research methods from several different disciplines in a ground- breaking effort to link different kinds of data. Essentially, several distinct studies were conducted, each investigating questions about mathematics and science learning from a different perspective. The combination of different research methods raised a variety of issues and questions, some of which are addressed below ("Critiques and Mathodological Issues". (See Appendix D for a bibliography of TIMSS reports and resources.) The different components of the study grew out of three basic questions that it was designed to answer: What are students in each nation expected to learn? What, and how, are students actually taught? What do students actually learn? TIMSS researchers used the terms "intended, implemented, and achieved curricula," respectively, to re- fer to these three basic questions (Robitaille and Garden, 1996~. The Achievement Study The core of TIMSS is an assessment of student achievement in mathematics and science, administered to students at ages 9 (Popula- tion 1), 13 (Population 2), and 17 (Population 3~. The achievement results, of course, provide the data on the achieved curriculum what students have actually learned. The content to be tested in each sub 4 LEARNING FROM TIMSS:

OCR for page 3
ject and at each age level was determined through a sometimes con- tentious consensus process involving all of the participating coun- tries. The resulting framework document, which guided the develop- ment of the test questions, reflects many compromises; it does not reflect the actual curriculum in any one country, and each country is free to conduct further analyses on just those questions that covered the material taught to its own students (International Association for the Evaluation of Educational Achievement, 1996a, 1996b). The test itself is similar to other large-scale assessments that are used in the United States, such as the National Assessment of Educa- tional Progress (NAEP). It is a combination of multiple-choice ques- tions and open-ended exercises that ask students to generate solutions to problems or to answer questions in their own words. The open- ended exercises are scored using guidelines that describe several cat- egories of responses and assign scores to them. In each country the test was administered to a sample of classes of students approxi- mately 3,750 students per country at each grade level (Third Interna- tional Mathematics and Science Study, 1996~. The samples were chosen so that various groups were adequately represented and each country's overall population characteristics were reflected. Each stu- dent answered only a portion of the questions meant for his or her grade level; various subsets of the questions were printed in different test booklets so that an appropriate number of students in each sample would take each possible combination of questions. Consequently, data could be reported on the entire content domain covered by the test although each student sat for only 60 or 90 minutes of testing. The complex item sampling design made it possible for researchers to report on the performance of different population groups and on student performance for different types of questions and different content areas. The sampling procedure also made possible the so- called "horse race" results, which rank the performances of partici- pating countries. Results are being reported for nations and, in the United States, for three states and one consortium of school dis- tricts.~ Forty-one countries participated in the assessment of middle- school, or Population 2, students (13-year-olds); these results were released shortly before the symposium. Twenty-six nations partici- pated in the elementary school, or Population 1, portion (9-year-olds), results for which were released in June 1997. Data for Population 3, students at the end of secondary school (17-year-olds), are scheduled for release in February 1998.2 No individual scores are available. iThe three states, Colorado, Illinois, and Minnesota, and the First in the World Schools, a consortium in the northwest suburbs of Chicago, provided funds for their participation as "mini-nations" in order to learn how their own students compare to others internationally. NCES has made it possible for other states or districts who wish to administer TIMSS locally to do so. 2Symposium participants repeatedly stressed the importance of recognizing, when drawing interpretations from TIMSS, that different groups of nations participated in different portions of the project. See note 5 on page 17 for the numbers of countries participating in each major component. RESULTS OF THE THIRD INTERNATIONAL MATHEMATICS AND SCIENCE STUDY s

OCR for page 3
Because the results are based on the performance of representative samples of students in each country, they actually, as TIMSS researchers explained, "represent a range within which the nation's actual average would most likely fall if all students were tested" (TIMMS U.S. Na- tional Research Center, 1996~. Thus, the U.S. achievement results were presented in terms of three bands groups of countries that per- formed better than the United States did, at approximately the same level as the United States, or worse than the United States. By pre- senting the results this way, researchers hoped to discourage observ- ers from focusing on slight differences that might be inappropriately magnified if numerical scores were simply listed in rank order. More than 20 countries also chose to include a set of performance assessment tasks for Populations 1 and 2; these were simple experi- ments using standardized materials provided in kits. The tasks were too expensive and time-consuming to include for the entire testing population, but they are expected to yield data on skills not easily measured by paper-and-pencil assessments (National Center for Edu- cation Statistics, 1996~. Testing of Population 3 students also ad- dressed two "specialist" subpopulations: students enrolled in advanced mathematics or physics courses. Background Questionnaires At the time the assessments were administered, students, teachers, and school officials were also asked to fill out background question- naires designed to elicit important information about the contexts in which student learning occurs. These questionnaires collected data on students' and teachers' backgrounds, school structures and resources, students' and teachers' attitudes about mathematics and science, teachers' pedagogical beliefs and practices, classroom coverage of various math- ematics and science topics, and other variables. Responses to these questions can then be correlated with achievement data to reveal asso- ciations between various factors and student performance. Although such associations cannot support specific causal inferences, they can call attention to factors that are associated with success and identify promising areas for further study. Quality Control The planners for TIMSS took great care to ensure the quality of the data collection, and independent observer Edward Haertel com- mented on the high quality of the sampling and data collection in the paper he presented at the symposium. The research team paid particu- lar attention to the sampling in part because SIMS, its predecessor, was criticized for using sampling methods that may have distorted the international comparisons. An entire volume documenting the quality control procedures used in TIMSS has been published (Third Interna- tional Mathematics and Science Study, 1996), but it is worth noting one strategy in particular. Because the sampling rules were so rigor 6 LEARNING FROM TIMSS:

OCR for page 3
ous and complicated, not all countries were able to meet all of them, but the data collected from these countries were still of value. The TIMSS research team defined several levels of compliance, which were clearly indicated in the main ranking tables. Thus, readers could see easily that comparisons between nations with differing lev- els of compliance should be made with caution and with an under- standing of the nature of these differing levels. Albert Beaton, TIMSS study director, presented a brief summary of the study at the symposium and highlighted a few of the key results from the Population 2 data, the first data to be released and the only data available at the time of the symposium.3 He began by noting that, while there has been worldwide interest in the country rankings, members of the press had not really addressed the more complex findings of the study or the issues and the questions they raise. For example, Beaton showed a table depicting results for the 41 Population 2 countries, similar to those used in the published reports. He explained that a reporter from a national news magazine had declined to publish it on the grounds that it was too complicated. Perhaps the most striking finding for Beaton was that all of the re- porting countries show a connection between socioeconomic factors and performance. In every one of the 41 countries, he explained, "there is a relationship between the number of books in the home and school performance." There was a similarly clear relationship across countries tested between parents' levels of education and student per- formance. Other factors explored in TIMSS did not demonstrate such clear relationships: for example, class size shows some rela- tionship to achievement, except that Korea, whose performance was second only to Singapore's, averages more than 40 students per class. Beaton presented some other key findings: There are differences in performance on particular content areas covered by the assessment that are consistent with differences in curricula across countries. . U.S. seventh-grade students ranked higher among nations than did U.S. eighth-grade students. Beaton remarked that this finding is important because it supports the overall achievement differences that were found. That is, differences between grades within a nation cannot be explained away by a large national difference, which would have affected performance at both grades equally. Within most countries and overall, boys had significantly higher mean science achievement than did girls in both the seventh and eighth grades. Gender differences in mathematics achievement were small or nonexistent; differences that did exist favored boys. There is a large difference in average science and mathemat- ~cs achievement between the top-performing and bottom-performing . . 3Population 2 covered the two school grades containing the largest numbers of 13- year-olds, grades seven and eight in the United States. RESULTS OF THE THIRD INTERNATIONAL MATHEMATICS AND SCIENCE STUDY 7

OCR for page 3
countries. Despite this large difference, when countries were ordered by average achievement, there were only small differences in achieve- ment between each country and the ones closest to it. In science, students generally had the most difficulty with the chemistry items. In mathematics, the questions that stood out as most difficult called for multistep problem solving and applications. In both mathematics and science, country performance in dif- ferent content areas seemed to correspond to curricular emphasis. Beaton was the first of many at the symposium to point out that the TIMSS data has, not surprisingly, failed to produce a "silver bul- let" that will magically transform mathematics and science education. As Beaton put it: "Wouldn't it be nice to just find that all we have to do is something simple, you know, increase the school year, for ex- ample? . . . We have been poring over the data . . . and there is just no simple answer." For every likely looking connection between achieve- ment and a variable such as amount of homework or class size, TIMSS showed counterexamples. Beaton and his colleagues concluded that, while each probably has an effect, none by itself made a major differ ence. The Curriculum Study As even casual observation reveals, there are substantial differ- ences among the education systems and curricula in use in the partici- pating nations. The purpose of the curriculum study was to find a way to make sense of these differences and to make it possible to explore the relationship between curriculum and achievement results. More specifically, researchers hoped that by looking systematically at which topics are covered at which levels around the world, and at performance expectations, they could gain understanding of differ- ences in student performance on particular skills and segments of the content that were tested. This study, of course, primarily explored what study planners called the intended curriculum. Undertaking a thorough comparison among the curricula of 46 countries was complicated by the fact that there is no common way of even describing curricula. The solution to this problem was a proce- dure called topic trace mapping, by which researchers in each country collected information about topic coverage in various documents and translated it into a common format. Using formally defined "docu- ment analysis procedures" as guides, the national researchers took the most widely used textbooks in their respective countries, as well as national and regional curriculum guides, and analyzed the documents section by section to determine the extent to which material included in the TIMSS frameworks was covered. A total of 491 curriculum guides and 638 textbooks were analyzed. The researchers also asked education experts within each country to respond to questionnaires 8 LEARNING FROM TIMSS:

OCR for page 3
designed to support the document analyses (Schmidt et al., 1997; TIMSS U.S. National Research Center, 1996~. William Schmidt, who directed the curriculum study, described some of the team's findings and conclusions, focusing primarily on the issues addressed in A Splintered Vision, the curriculum analysis results for the United States (Schmidt, McKnight, and Raizen, 1997~.4 For him, the study's most valuable product is what he sees as resolu- tion of the debate over whether school curricula truly make a differ- ence in student learning. For him it is clear that teaching matters, and he argues that the "somewhat disappointing" achievement results for the United States reflect the weaknesses in the U.S. curricula. His conclusion is that many other factors such as length of time spent in school and assignment of homework that have been blamed for poor student performance in the United States are side issues. He explained that his research has shown that "there is a tremendous amount of variability across these countries in terms of the way in which mathematics or science is taught." He suggested that further exploration of the relationship between achievement and topic cover- age in the curriculum will clarify the picture of student learning con- siderably. Specifically, Schmidt argued that no intellectually coherent vi- sion guides mathematics and science curriculum development in the United States. Because responsibility for curriculum decisions rests with states and localities, there is variation among the curricula used within U.S. borders, just as there is among those of different nations. Some of this variation reflects differing educational goals and phi- losophies, while some of it is, in effect, coincidental. Schmidt pre- sented a few specific findings to illustrate his points: . . Both science and mathematics textbooks in the United States include far more topics than was typical for other countries at all three grade levels. This is true even for science texts devoted to particular topics, such as earth science or physical science. Mathematics curricula in the United States consistently cover far more topics than is typical in other countries. In science, the tendency toward breadth is similar, though less pronounced. . Topics remain in both the mathematics and science curricula for more years in the United States than in all but a few other TIMSS countries. The U.S. practice is to introduce many more topics than do other countries in grades one and two and then to repeat these 4William Schmidt served as both the principal investigator for the curriculum study and the national coordinator for the U.S. portion of the achievement study. He also served as the project director for the Survey of Mathematics and Science Oppor- tunities (SMSO). This study, conducted in advance of TIMSS, produced a set of classroom observations in six countries that were designed primarily to identify important themes and issues to be explored in the TIMSS background questionnaire. His presentation drew on all of these sources. RESULTS OF THE THIRD INTERNATIONAL MATHEMATICS AND SCIENCE STUDY 9

OCR for page 3
topics through grade seven. Schmidt emphasized this point by noting that although new elements called for by science standards have gen- erally been added to the curriculum, little has been removed to make room for the new. American teachers, Schmidt argued, are sent into their classrooms with a mandate to teach using curricula that reflect few decisions about priorities, are fragmented, and are poorly integrated with one another. Teachers, he said, are armed with textbooks that are simi- larly laden with a jumble of topics. The curricula are, in his words, "a mile wide and an inch deep." How do teachers handle this situation? Schmidt argued that the instructional decisions made by U.S. teachers mirror the inclusive approach of the tools they are given. Teachers cover more topics, he suggested, but spend less time and emphasis on each than do many of their international counterparts. Instead of "telling a story" about a particular topic, allowing enough time for students to learn it and move on, he argued, U.S. teachers tend to keep reintroducing topics that have not yet been mastered. Schmidt concluded that the U.S. educational vision is splintered because the U.S. system has many actors and is characterized by "dis- persed control," as Richard Elmore later put it. For Schmidt, this system is responsible for the seriously inadequate sets of curricula currently in use. The incoherence of the curricula, he argued, has impeded student learning. The Three-Country Qualitative Studies Germany, Japan, and the United States participated in additional studies, sponsored by the United States, in order to augment their understanding of the achievement results. These studies, a videotape analysis and a set of case studies, were devised to explore both in- struction and the cultural contexts within which the learning and teaching of mathematics take place. They involved methodologies rarely used in conjunction with large-scale assessments of achievement, and, in the case of the videotape study, of technology developed specifically for TIMSS. James Stigler and Harold Stevenson, the principal re- searchers for the videotape study and the case studies, respectively, each described their methods and some key findings. Videotape Study The primary goal of the videotape study was to capture and then analyze entire mathematics lessons taught to a subsample of the Popu- lation 2 (grades seven and eight) students. Lessons were taped in a total of 231 classrooms across the three participating countries. Teachers were asked to make no changes in their normal classroom routines for the videotaping sessions. Standardized camera procedures and other protocols were developed for the data collection. The thousands of 10 LEARNING FROM TIMSS:

OCR for page 3
hours of tape were digitized, and computer software was developed for analyzing them. Thus it has been possible for researchers to scan quickly through the material on a computer and to search it in various ways. In addition, the tapes were transcribed and translated and then coded for the occurrence of various events, teaching strategies, and content elements. The coding made it possible for researchers to analyze the lessons quantitatively and to explore such issues as amounts of time spent on seatwork and classwork, discussing and doing home- work, and non-lesson activity. The tapes were also analyzed by math- ematicians for mathematics content. In addition to making possible the exploration of questions about teaching practice, as well as specific questions raised by data from the achievement tests and the background questionnaires, the video- tapes have two other important uses. First, as symposium partici- pants who watched just a few short segments emphasized, the oppor- tunity to observe a lesson on tape is far more powerful than any verbal description can be. It is clear that the tapes themselves, as well as the experience gained in collecting them, will be an extremely valuable resource for teacher training, as well as for research. Sec- ond, the digitized tapes are a permanent, unchanging resource. Fu- ture research can be conducted using these tapes as a record of teacher practice at a particular time, as research questions change. Apart from the interesting technical issues Stigler and his team faced, the videotape study produced some interesting conclusions about variations in teacher practice among the three participating nations. The report on the study had not been released at the time of the symposium, but Stigler discussed several of its key findings. Perhaps most important was Stigler's conclusion that the majority of prescrip- tions about teacher practice that have been generated by the research community in recent years have not been implemented in U.S. class- rooms. Stigler argued that the relatively large-scale videotape study has made it possible for the first time to look at what teachers are actually doing in the classroom and to compare that with their verbal descriptions of what they believe they are doing. Citing the notion of problem solving, for example, a traditional mathematics skill that is carefully redefined in the National Council of Teachers of Mathematics (NCTM) standards, Stigler pointed out that the understandings teachers and others have of what it means in practice vary to an alarming degree. He described a lesson he had observed, in which students solved a series of traditional word prob- lems as a group. Their teacher had spoken enthusiastically about the "amazing problem solving" the students were doing, believing that she had fully responded to this aspect of the revised standards. Stigler made the further point that major shifts in education policy often occur without the benefit of any, or sufficient, data about the extent to which the current policy has actually been implemented in the classroom. This point is relevant to a question that many have RESULTS OF THE THIRD INTERNATIONAL MATHEMATICS AND SCIENCE STUDY 11

OCR for page 3
asked about TIMSS whether TIMSS achievement results could be seen as a measure of the impact of the NCTM standards, which were published in 1989, on student learning. Stigler's conclusion from the videotapes is that the question is moot since the NCTM reforms have, by and large, not been implemented in U.S. classrooms. Stigler also reviewed some of what the videotapes revealed about differences among the three nations. He noted that the sample sizes chosen, 100 teachers for Germany, 81 for the United States, and 50 for Japan, partly reflected expectations about how much teaching styles were likely to vary within each country. Surprisingly, Stigler found that teachers in both Japan and the U.S. were remarkably consistent. In general, Stigler's portrait of typical approaches to lessons in Japan and the United States (his presentation focused on these two nations) is likely to cause concern in the U.S. education community, and that impression was strongly reinforced by the videotapes he showed. The Japanese lesson showed a teacher who pushed his students to grapple with a series of problems and to come up with alternative solutions. The teacher communicated respect for his students' abili- ties to cope with challenging material, and he guided the students skillfully from the alternative solutions to a more general understand- ing of the concept the lesson covered. In contrast, the U.S. teacher seemed to lead his students by the hand through an explanation of a concept, and he telegraphed his expectation that the students would have trouble applying the concept in a challenging problem by warn- ing them repeatedly about a particular problem as they began their seatwork. Then, before they had had time to attempt that problem, he stopped them and led them through it step by step. The U.S. lesson was also interrupted more than once, both by conversation about school schedules and other issues unrelated to the lesson and by an announcement over the public address system. These two excerpts were chosen by Stigler to represent what he and his team had judged to be typical of the lessons he saw in the two countries, and they raise issues that are familiar to many in the policy and research communities. For Stigler, the videotapes from Japan and the United States painted a consistent picture of two different ap- proaches to teaching. He noted that the questionnaires administered to the teachers who participated in the videotape study (these were different from the questionnaires administered with the achievement tests) revealed very different expectations for the outcome of a lesson: 70 percent of Japanese teachers reported that their goal was to get the students to understand a concept; similar percentages of U.S. (and German) teachers reported a goal of getting students to be able to do a certain kind of problem. In Stigler's view, the Japanese lessons gen- erally "tell a story" and provide students with the opportunity to struggle with and explore the concept the teacher is presenting. In contrast, the videotapes show relatively less development of concepts in the U.S. lessons, which Stigler characterized as focusing on short-term goals. 12 LEARNING FROM TIMSS:

OCR for page 3
To support his conclusions, Stigler explained a few of his spe . a. I. . ClilC IlnC .lngS: . While the proportions of time spent on incliviclual work ant! work as a class are roughly the same in the two countries, Japanese teachers tend to switch between the two much more frequently than do U.S. teachers. . The U.S. teachers pay far more attention to homework than clo their Japanese counterparts, allotting significant chunks of class time for going over previous homework or allowing students to begin new assignments. leaving relatively less time for instruction. . a, , a, The U.S. lessons were interrupted by non-mathematics-relatec} activities significantly more frequently than were the Japanese les- sons. This fincling reinforced for Stigler the sense that in Japanese society the lesson is regarclec! as a coherent, sustained inquiry into a topic while in the U.S. it is regarclec! more as an episode or a practice session. Japanese teachers generally focus on just one topic cluring each lesson; U.S. teachers average close to two topics. The participants' responses to the brief videotape excerpts were extremely lively, ant! many remarkoc! on how convincingly the ex- cerpts seemec! to illustrate particular arguments about teaching prac- tice. Some of the issues raiser} both in the papers preparer} for the symposium ant} by participants about ways of using ant} unclerstanci- ing this kind of data are explorer} below ("Critiques ant} Methoci- ological Issues"). Case Studies While the primary focus of the videotape study was on teacher practice, the case studies concluctec} by Harold Stevenson explorer} in cletail the contexts that shape the experiences of students ant} teach- ers. Like other parts of TIMSS, this stucly was clesignec! to provide data to help account for some of the variations in student perfor- mance, in this case by examining contextual influences. Previous studies have shown that differences in curriculum ant} education structure can provide insights about performance, but other kinds of informa- tion are also neecleci. How do teachers in different places think about teaching, learning, ant! curriculum? How have they been preparer! ant! what kinds of support clo they receive? What factors in ant! out of school affect students' motivation to learn? What are students' attitudes about mathematics ant} its value? While the contexts that shape learning can be explorer} through written questionnaires, the case studies were an opportunity to make cross-cultural comparisons in far greater detail ant! to investigate subtler issues than a coclec! questionnaire could permit. Through this project researchers intenclec! to produce thorough analyses case studies of eclucation-relatec} factors in three distinct cultures, the United States, RESULTS OF THE THIRD INTERNATIONAL MATHEMATICS AND SCIENCE STUDY 13

OCR for page 3
Germany, and Japan. The studies were structured around three basic issues. The first was how content and performance standards in the three countries compare. Through this comparison, researchers hoped to explore the ways in which each country deals with individual dif- ferences among students. The second was the role of school in ado- lescents' lives. The last was the ways that training, certification, and support for teachers' continuing professional development affect their working lives. Stevenson began by explaining that the study was a sort of hybrid, devised for TIMSS, between the methods of anthropological ethnog- raphy and the interview approach characteristic of psychology. The result was what he termed "a descriptive study" "a description of what you would find if you were in these particular cultures." The basic plan was to identify and train individuals who were familiar with each of the three societies, fluent in the requisite language, and skilled in observation and interview techniques, and to send them into the field to collect information. With the help of country experts, sites were chosen that would broadly reflect national characteristics, and researchers assigned to each country spent 2-3 months collecting data. The researchers spent the bulk of their time interviewing parents, students, and teachers and observing classroom lessons. They visited homes, schools, and education ministries. The result was hundreds of hours of audiotape, which was transcribed and translated. As in the videotape study, the material was entered into a computer and coded so that researchers could search it efficiently, but the data were not analyzed statistically; rather, they were synthesized into detailed de- scriptions, organized around the explicit questions that guided the study. Like the report on the videotape study, the case study reports had not been released at the time of the symposium, but Stevenson high- lighted some of the insights that have emerged. One important focus of his presentation was on ways in which detailed knowledge of cul- tural contexts can significantly alter discussions about a particular issue. His choice of an example homework was inspired by his concerns about the ways in which symposium participants had dis- cussed the relationship between homework and achievement results. He noted that in Japan there are four possible translations for the term, none of which corresponds to our notion of the word. The Japanese terms describe a variety of activities one might do outside of class study, work on practice questions, or do an assignment, for example. They reveal that ways of categorizing such activity differ in the two cultures. To further illustrate the point, Stevenson noted that the amount of homework done by German students varies signifi- cantly, depending on the type of school they attend. Consequently, a mean for homework done in Germany would have very little value. It is only through interviews, Stevenson maintained, that researchers were able to discover what kinds of out-of-school studying students in each 14 LEARNING FROM TIMSS: