Cover Image

PAPERBACK
$34.75



View/Hide Left Panel

3
Impact

Determining the appropriate mix of different kinds of international comparative education studies requires not only an awareness of their scope and cost, but also a better understanding of the actual benefits and impact of different types of studies in the U.S. education community and the time frame necessary for that impact to be manifest. The United States, like many other countries, has participated in large-scale, cross-national education surveys assuming that comparative studies will, at some point, have a positive impact on its education system. Investments in smaller scale international comparative work by individuals and private groups are often similarly motivated.

Major investments in international comparative studies in education, however, are proceeding without analysis of these assumptions with respect to individual studies or programs of study. As in other areas of education research, gauging the actual impact of these studies—in terms of findings used, changes in student achievement, policies debated, understanding expanded—is difficult. Nonetheless, this chapter illustrates the many ways international comparative studies of all three types have had impact on the U.S education system and, with systematic effort, could have more. Our aim is not to use cost-benefit analysis to determine which studies should be funded and which dropped; rather, it is to understand when studies do in fact have impact and to reinforce them in those areas. Of course, the impact studies would also be valuable points of reference when new investment decisions are to be made as well.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education 3 Impact Determining the appropriate mix of different kinds of international comparative education studies requires not only an awareness of their scope and cost, but also a better understanding of the actual benefits and impact of different types of studies in the U.S. education community and the time frame necessary for that impact to be manifest. The United States, like many other countries, has participated in large-scale, cross-national education surveys assuming that comparative studies will, at some point, have a positive impact on its education system. Investments in smaller scale international comparative work by individuals and private groups are often similarly motivated. Major investments in international comparative studies in education, however, are proceeding without analysis of these assumptions with respect to individual studies or programs of study. As in other areas of education research, gauging the actual impact of these studies—in terms of findings used, changes in student achievement, policies debated, understanding expanded—is difficult. Nonetheless, this chapter illustrates the many ways international comparative studies of all three types have had impact on the U.S education system and, with systematic effort, could have more. Our aim is not to use cost-benefit analysis to determine which studies should be funded and which dropped; rather, it is to understand when studies do in fact have impact and to reinforce them in those areas. Of course, the impact studies would also be valuable points of reference when new investment decisions are to be made as well.

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education Recommendation 2: The United States should conduct systematic analyses of costs of the expensive Type I and Type II studies (including financial cost, respondent or participant burden, accommodating dosing shortcomings, etc.) and benefits (services received, information provided, topics studies, etc.) so that a more complete picture of impact can inform future program and funding decisions. These analyses should be “internationally comparative” in that they compare impact in the United States with impact from the same or similar studies in other countries. DEFINING IMPACT The goals of the U.S. education system range from teaching such fundamental skills as reading, writing, and arithmetic to helping students begin a rewarding career and develop physically and emotionally (Cremin, 1990:42). With such a broad target, it might seem hard for education research to miss the mark. Nonetheless, the impact of education research—both domestic and international—remains difficult to measure. Weiss (1998:331) defines impact as “the net effects of a program (i.e., the gains in outcomes for program participants minus the gain for an equivalent group of non-participants). Impact may also refer to program effects for the larger community.” Measuring the impact of domestic or international education research findings on the U.S. education system, then, involves looking for costs and benefits for participants in different parts of the system and thinking about the system as a whole. Take TIMSS as an example. Wiseman and Baker (2002) report that many classroom teachers who have encountered TIMSS perceive the results as implicitly critical of U.S. teachers, even as a challenge to their professionalism. In contrast, for state and national policy makers, TIMSS has been a more constructive experience, as many have been able to use poor performance on TIMSS 1995 and TIMSS 1999 to lobby for more funds for education. Similarly, for much of the public, TIMSS 1995 is history, but researchers only recently began to publish the results of secondary analyses addressing one or more of the myriad questions the study has raised. Finally, the influence of TIMSS 1995 on individual schools and school districts varied; although some higher achieving U.S. schools launched major reform efforts in response to the performance of the United States in TIMSS, the same data did not mobilize lower achieving U.S. schools into action (Wiseman and Baker, 2002). International education research in the United States, like domestic research, suffers from a lack of infrastructure that could make research matter. Where do the results of research flow into the education system? Is the system too loosely coupled (Weick, 1976) to be influenced even by the

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education strongest research findings? Raizen (2002) recounts how the United Kingdom and Singapore analyzed their performance on the Second International Science Study (SISS) and used the results to reform their science curriculum. Although the data do not allow an attribution of causality, she notes that both countries improved their performance dramatically in TIMSS; in contrast, Hong Kong undertook neither analyses of the SISS results nor curriculum reform and did not improve its performance in science between SISS and TIMSS. Raizen observes that, although many complained about the shallowness of U.S. mathematics and science curricula after the TIMSS results were released in 1996, there is little evidence of subsequent efforts to reduce coverage and deepen conceptual learning. Plank (2002), looking at the impact of international models on the U.S. debate on school choice and privatization, identifies the fragmented state of education decision making in the United States as one of the chief impediments to the effective use of education research about other countries. Since decentralized education decision making is highly prized in the United States, the task of getting policy-relevant findings to the decentralized level will continue to challenge both domestic and international education researchers for the foreseeable future. Because of this fragmented decision making, international and domestic education studies are more likely to have an impact at the school or classroom level if design and analysis teams include representatives from the state and local levels and if the results can be disaggregated by state and locality. For example, TIMSS was a curriculum-based test administered to a national sample of classrooms, producing a distribution of scores around a national mean. Curriculum policy in the United States tends to be made at the state or local level, and benchmarking studies were attached to both TIMSS 1995 and TIMSS 1999, the latter allowing more than two dozen states, districts, and consortia to administer TIMSS as though their jurisdictions were countries. This located TIMSS results closer to the level at which the effects of curriculum decisions could be compared. Type I studies tend to be the most expensive in terms of direct costs, as noted earlier, involving many individuals in the conduct of surveys and employing experts to organize and analyze them. But less often taken into account is that this type of study has high indirect costs as well and these should be taken into consideration when studies are planned and when impact is evaluated. These may include the burden on schools that sacrifice classroom and administrator time in order to participate in international assessments but typically gain little or nothing from them; the maintenance of the sophisticated infrastructure—data process-

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education ing centers, analysis units, training programs—needed to implement technically sound studies; the diplomatic capital needed to persuade a significant number of comparable countries to participate in a given assessment, with the desired level of sampling and measurement rigor; the opportunity cost, since adding studies reduces the amount of time that a limited number of experts in international large-scale assessment can spend on any single study; and undermining policy reform when studies are undertaken more frequently than policy reforms can produce change, creating the sense that policy reforms are not working. Recommendation 2.1: U.S. funders should move away from piecemeal, ad hoc funding of international studies, and toward incorporating explicit considerations of relative cost, benefit, and impact in both the planning and the proposal review processes. The flow of findings from domestic and international education studies into the U.S. education system is affected by factors that create a social and political environment more or less open to change. For example, after the mid-1980s many factors aided the rapid acceptance of the Reggio Emilia model by some in the U.S. early childhood education community (see Box 3-1). In the 1990s, an engaging traveling exhibition on the Reggio Emilia model exposed the public and teachers to a new way of organizing early childhood education in Italy. Many Americans who earlier had heard of another Italian preschool model, the Montessori method, were open to the possibility that the United States might have more to learn from Italy about early childhood education. The end of the cold war in the early 1990s also brought a greater level of comfort with ideas derived from groups who earlier might have been dismissed as European leftists. Interested scholars and educators from the United States would have found it easier to travel to Italy to observe Reggio Emilia than to travel to Japan or China. Implementing the model in the United States did not involve changing legislation or even getting permission from a school board, since most early childhood education programs in the United States are privately rather than publicly organized. Early childhood educators interested in improving their instruction methods were in a position to try various models at will, knowing that their students would not face either mandatory exit exams from kindergarten or entry exams for elementary school. University-based scholars in the United States had studied the Reggio Emilia model in Italy and other countries and offered their expertise to some Reggio Emilia experiments in the United States.

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education BOX 3-1 The Reggio Emilia Approach to Early Childhood Education In the early 1960s, the parents in Reggio Emilia, a small, wealthy city in northern Italy, began developing an early childhood education program designed to take children seriously and to focus on nurturing collaboration and critical thinking as a means to prepare children for life in a democratic society. Inspired by a strong sense of purpose and civic engagement, Loris Malaguzzi, an Italian early education specialist, joined in the effort to open the first municipal preschool in 1963. The Reggio Emilia philosophy of early childhood education is based on these key features: The role of the environment as teacher: The classroom, school, and its surroundings encourage discovery, involvement, and an interest in beauty and the environment. Visual arts as means of representation: Because of a belief that creativity is not a separate faculty but rather a way of thinking and responding to the world, collaborative partnerships are formed between classroom teachers and art educators to promote children’s multiple means for expression. Documentation as assessment and advocacy: Documentation of student progress promotes adult (teacher and parent) understanding of and interest in children’s development. Long-term project work or progettazione: Teachers guide children in gaining new insights through long-term projects that emerge from the child’s interests. Teacher as researcher: Reggio Emilia educators use a Deweyian approach to scientific inquiry to guide classroom practice, i.e., posing hypotheses about children’s learning, creating experimental conditions to test those hypotheses, collecting data (e.g., artifacts of children’s work and transcripts of adult-child conversations), analyzing those data, and posing new hypotheses. Education as relationship (adults and children, home and school): The child, parents, and teachers all actively participate in the education process. Efforts are made to promote parent participation through building respectful home-school relationships. In the 1980s, Reggio Emila’s influence began extending in other countries, particularly in the United States. Since 1986, delegations of American early childhood educators have traveled to Reggio Emilia to observe the approach firsthand and to share those experiences with others at U.S. regional, state, and national conferences. Conference presentations and delegations helped to generate greater interest

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education in Reggio Emilia and led to the establishment of formal and informal networks of teachers and teacher educators who began establishing their own study groups. The American media also helped get the word out through special features on public television and Newsweek’s declaration that Reggio Emilia preschool was “the best in the world” (Hinckle, 1991). Its widespread appeal stems in part from the guidance the approach gives teachers on how contemporary theories of how children learn can be translated into classroom practice. Reggio Emilia also affirms personal and professional values that take seriously both young children and early childhood educators. SOURCE: Adapted from New (2002). In addition, the model encouraged teachers and parents to undertake their own “action research,” thereby legitimizing their role in and perhaps their commitment to developing their own programs. Further reinforcing the professional legitimacy of the program, Reggio Emilia now has its own set of sessions at the annual meetings of the National Association of Educators of Young Children and is considered a major curriculum model and educational approach in the literature of professional early childhood education (New, 2002). In this case, systematic scholarly studies did not play a role in spread of this innovation. Rather, Reggio Emilia-related activities offer an opportunity for comparative education researchers to respond to a practitioner-generated agenda by assessing the Reggio Emilia experience in the United States and how it can usefully, and perhaps necessarily, diverge from the program in Italy. While the Reggio Emilia model has been influential among specialists in early childhood education, the extensive research on school choice and privatization in several other industrialized countries, widely reported in the academic and popular presses and synthesized in studies commissioned by the U.S. government, has not played a major role in the debate on choice and privatization in the United States (see Box 2-2). Plank (2002) relates this impotence in part to the equivocal findings of this research, and in part to the level of the education community at which this evidence might be used. Unlike the Reggio Emilia model, which could be adopted in whole or in part by educators at a classroom or a school level, the issue of choice opens up prospects of sweeping change in the allocation of control and resources at the district and state levels. Consequently, this animates powerful interest groups who may well be less interested in whether choice “works” in other countries than whether it challenges sta-

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education tus quo interests. In the policy arena, international education research suffers the same fate as domestic research: strong evidence can be ignored in the formation of policy, and weak evidence can be deployed to support strong policy conclusions (Plank, 2002). The popularity of Liping Ma’s book Knowing and Teaching Elementary Mathematics (see Box 3-2) also provides another contrast that may shed some light on the lack of U.S. interest in international choice and privatization research. Fang (2002) argues that because Ma was writing as an outsider to U.S. education about a non-U.S. practice, her warm reception by partisans on both sides of the U.S. “math wars”1 may be related in part to her perceived status as a neutral party. Ma, nonetheless, was able to explicitly compare Chinese with U.S. teachers, and the reception for her book by U.S. readers indicates it was hard to dismiss her work as another international study about an irrelevant country. Despite the many similarities, international comparative education research does differ from domestic education research in several ways. First, comparative research demands attention to national as well as local contexts. The range of differences in schooling in various countries can hardly be imagined by many educators who have not spent significant time outside the United States. For secondary users to make sense of data about student achievement in other countries, comparative researchers must provide much background information and rich descriptions of context, elevating the importance of parallel case studies and of background variables. For these reasons, international research is likely to be more expensive than comparable domestic research. Finally, international educational research findings lend themselves more readily than domestic findings to rhetoric that plays to Americans’ deep-rooted concerns about the relative competitiveness of the U.S. labor force, the economy, and the future of their children. The rhetoric surrounding international studies sometimes can lead to overestimates of the strength of reported findings and of their potential impact on classroom practice. The following discussion of impact is loosely organized around five paths through which education research may influence the education system. Four of the paths lead to classroom practice: educational materials; pre- and in-service training for educators; education policies; and changed knowledge, attitudes, or practices among the public (National Research Council, 2002b). The fifth path leads to the domestic research community. 1   The term “math wars” refers to controversies beginning in the early 1990s among various groups of mathematics educators and mathematicians regarding the proper emphasis on teaching (1) basic computational skills and (2) problem solving and advanced topics in mathematics.

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education BOX 3-2 Knowing and Teaching Elementary Mathematics Knowing and Teaching Elementary Mathematics: Teachers’ Understanding of Fundamental Mathematics in China and the United States by Liping Ma (1999) is based on a comparative study of knowledge of mathematics by Chinese and U.S. mathematics teachers. Ma interviewed teachers in both countries, and she paints a picture of their subject-matter knowledge about standard topics of elementary mathematics. Ma depicts the differences in mathematics understanding between the two groups, with Chinese teachers tending to have a much more profound understanding of fundamental mathematics, that is, knowledge that enables teachers to go beyond computational accuracy and awareness of concepts to teach mathematical concepts to students. Ma describes factors that support the development of Chinese teachers’ mathematical knowledge that do not exist in the United States, despite the fact that Chinese mathematics teachers typically receive less education than their U.S. counterparts. She concludes the book with suggestions for changes in teacher preparation and mathematics education to enable U.S. teachers to deepen their mathematics knowledge, such as enhancing the interaction between the study of mathematics as a subject and ways of learning how to teach it. With 31,000 copies sold by the beginning of 2002, its publisher describes Knowing and Teaching Elementary Mathematics as a “runaway bestseller.” Much of the book’s popularity flows from its focus on issues of current concern in the U.S. education community and also from the clarity of its argument. About the time of the book’s release, the National Council of Teachers of Mathematics was in the process of updating its 1989 mathematics curriculum, assessment, and professional development standards. Both sides in the so-called math wars,* no longer able to discuss dispassionately what constitutes basic skills and conceptual understanding in mathematics or how best to teach them, found evidence and arguments in the book to support their positions. Even though the findings were not entirely new, Liping Ma presented them in a way that resonated with both sides. Ma’s life story contributed significantly to the development of her research and ultimately her book. Ma grew up in urban Shanghai, but during the Cultural Revolution, after receiving only eight years of formal school *   Between 1992 and 2002, with funding from the National Science Foundation (NSF) and NCES, the board published four reports on the results, use, secondary analysis, and effects of TIMSS, all of them with many caveats and recommendations about the implications of TIMSS for policy and practice. In addition, the board sponsored a public symposium and published an edited volume, based on the symposium, on methodological advances in the methodology of cross-national achievement studies (including TIMSS) and a report on the methodology and use of videotapes in cross-national studies, prompted by the TIMSS Videotape Classroom Study.

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education ing, was sent to teach in a rural village. In Jiangxi, she taught for seven years and then served as the school’s principal and later as the county’s education superintendent. After the Cultural Revolution, Ma studied at the East China Normal University, a national university that plays a leading role in teacher preparation, where she read world classics in education, such as Confucius, Rousseau, and Zahkob. Her passion for education led her to study education research and teaching at Michigan State University. She began work on a teacher research project in the National Research Center for Teacher Education to help pay for her education. While coding interviews with U.S. elementary teachers about mathematics, she noticed some differences between U.S. and Chinese teachers’ understanding of mathematics. The center provided a $1,000 grant for Ma to return to China and collect data. These data later became the foundation for her dissertation at Stanford University and for the 1999 book. The impact of the book has been widespread yet disparate, affecting the research community more than practitioners. The mathematics community played an important role in disseminating the book, which has been adopted at the university level for mathematics courses but has had limited impact in teacher preparation programs. The Conference Board of the Mathematical Sciences drew on Ma’s ideas about teachers’ deep, profound, and connected mathematical knowledge to describe the content and pedagogical knowledge it recommended that U.S. mathematics teachers acquire during preservice training. The book’s ideas have affected some school districts and teachers. The California Subject Matter Project includes a professional development component for mathematics that helps teachers develop the profound understanding Ma describes. The Mid-Atlantic Eisenhower Consortium for Mathematics and Science Education at Research for Better Schools has also promoted Ma’s ideas through its professional development workshops. The far-reaching influence of Ma’s work illustrates how small-scale comparative studies can have a powerful impact on educational research and practice. SOURCES: Fang (2002); Ma (2002). EVIDENCE OF IMPACT The examples scattered throughout this chapter illustrate many ways in which comparative studies can change classroom practice and ideas about education. Examining intellectual histories and antecedents of these cases, the board found a surprising degree of connection among these studies. These connections suggest that the impact of international studies is sometimes recursive. For example, TIMSS 1995 played a role in

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education opening U.S. educators’ eyes to instructional methods used in other countries, particularly in mathematics and science, and particularly in high-performing countries, such as Singapore (see Box 2-2). The Learning Gap (Stevenson and Stigler, 1992), which compares mathematics education in China, Japan, and the United States, as well as the relatively high performance of Asian countries on TIMSS 1995, prepared some in the U.S. mathematics community to absorb Liping Ma’s insights about the superior knowledge base of elementary mathematics teachers in China and to seize the Singapore mathematics curriculum. The TIMSS videotapes, which were widely used in professional development, and The Teaching Gap (Stigler and Hiebert, 1999), a book based on the TIMSS Videotape Classroom Study, motivated educators to take a closer look at Japanese lesson study. The impact of studies and innovations on U.S. education policy and practice, however, has been mixed, as described below. Educational Materials Curriculum documents, lesson plans, textbooks, and computer programs are among the most easily transported, interpreted, and analyzed means of changing what goes on in classrooms. Their appropriate interpretation, however, requires a deeper understanding of the education system in which those materials play only a supporting role. Different curricula assume different levels of expertise on the part of the teachers, different frequency and length of classes, different preparation and extra-curricular support for learning, and a host of other factors. The first and second IEA mathematics and science studies were among the first large-scale achievement surveys to attempt to seek a connection between the curriculum topics teachers said they taught and student performance on achievement surveys. The First International Mathematics Study (FIMS) introduced the concept of “opportunity to learn”; the Second International Mathematics Study (SIMS) contributed the concept of “enacted curriculum”; and it was in connection with TIMSS curriculum analysis that Americans learned to think about their K-12 curricula as “a mile wide and an inch deep” (Dossey, 2002). These changes have stimulated U.S. educators to reexamine their own and other countries’ curricula, although not necessarily to the extent of the United Kingdom and Singapore, mentioned above, and not necessarily combined with efforts to derive hypotheses from the TIMSS data about what sorts of curricula might be more effective. The United States was not alone in its lack of attention to and support for this task; Postlethwaite (1999:60) writes, “It was the crude achievement results that had an impact in terms of awakening the [U.K. education] ministry personnel to the shortcomings of a region or country.”

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education Wiseman and Baker (2002:i) assert that American response to TIMSS has been more reactive than interpretive. When TIMSS 1995 placed U.S. students near the mean in international mathematics and science assessments, some U.S. educators made a direct connection between curriculum and performance and promptly acquired copies of the curriculum used in the highest performing country, Singapore (see Box 2-2). Singapore’s textbooks, written for English-language schools, were accessible to English-speaking Americans and appeared to offer a more sequential, structured approach to teaching mathematics than most U.S. textbooks at the time. Meanwhile, a few years after TIMSS was released, Singapore itself changed its textbooks, reducing the content and increasing the emphasis on problem-solving and critical thinking (Lee and Fan, 2002). Several cases highlight the important role played by artifacts—actual textbooks from Singapore, children’s artwork in the Reggio Emilia traveling exhibit—in documenting techniques, as opposed to abstract concepts. Understanding the power of such artifacts, the Council for Basic Education’s Schools Around the World2 program recently began giving a small group of teachers in nine countries the opportunity to compare examples of student work on similar subjects and problems. The TIMSS Toolkit3 includes the official English translation of the Japanese Ministry of Education’s National Course of Study for Mathematics (Peak et al., 2002) and a videotape of actual mathematics lessons in Germany, Japan, and the United States. Finally, in an approach somewhere between adapting another country’s curriculum and trying to learn from another’s artifacts, in 2000, Houghton Mifflin, a U.S. textbook publisher, hired Liping Ma and her collaborator to develop a U.S. mathematics curriculum supplement informed by their research on the different ways U.S. and Chinese teachers communicate profound understanding of fundamental mathematics (Fang, 2002). Teacher Development There are many ways in which teacher development can be influenced by international research and in which teachers can, in turn, inform international research. As described in Box 3-3, The Teaching Gap (Stigler and Hiebert, 1999) combined with the TIMSS Videotape Classroom Study to 2   Council for Basic Education, Washington, D.C. (http://www.c-b-e.org). 3   Attaining Excellence: A TIMSS Resource Kit, commonly known as the “TIMSS Toolkit,” was prepared by the U.S. Department of Education to share some of the highlights of TIMSS with U.S. practitioners. It includes modules on TIMSS as a starting point to examine mathematics assessments, U.S. education, student achievement, teaching, and curricula (U.S. Department of Education, Office of Educational Research and Improvement, 1997).

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education BOX 3-3 Impact of Japanese Lesson Study in the United States Lesson study, or jugyokenkyu, is a process of professional development that evolved during the period of progressive education reform in Japan following World War II, a time when the focus was on teacher-directed work on curriculum development and child-centered education (Yoshida, 1999). The process begins when a group of teachers, either from the same school or from different ones, comes together to work toward a common goal they want to achieve, usually to address a gap they have identified in students’ current knowledge or ability. Working on lesson study involves several steps that incorporate both observation and collaboration, and yield a written record of the insights gained from watching lessons unfold in a real classroom. The goal of the lesson study process is to help teachers become more deliberate and self-aware in their teaching practice and to carry insights gained from group planning into their daily individual lesson planning and teaching practice. The process is designed to be teacher-driven; teachers select the topic to be studied, identify the goal of the lesson, and learn from each other’s experiences and expertise. The history of lesson study in the United States began in the 1980s and 1990s, when the process came to the attention of several U.S. education scholars through their ongoing research into Japanese approaches to teaching. Catherine Lewis became aware of lesson study through her observations in Japanese elementary classrooms (see Box 2-3). James Stigler became aware of lesson study during his research on an NSF-funded study using videotapes of Japanese and U.S. teaching. In 1993-1994, Stigler and his graduate student Clea Fernandez directed a lesson study group in the United States with a group of teachers from the University of California Los Angeles Lab School. Makoto Yoshida, a doctoral student of Stigler’s collecting data on lesson study in Japan, served as their main source of information for guiding U.S. teachers through the process. The TIMSS Videotape Classroom Study, with its images from Japanese classrooms, piqued broad interest in Japanese approaches to teaching mathematics. Stigler and James Hiebert published the findings from TIMSS for a broad audience in The Teaching Gap (1999) and, through this work, lesson study emerged both as an important tool for understanding the culture of teaching in Japan and a potentially beneficial strategy in the U.S. education context. To deepen understanding about lesson study and to explore its impact in U.S. schools, in 1999 Fernandez, now a professor at Teachers’ College at Columbia University, and Yoshida began working with Paterson Public School #2 in New Jersey in collaboration with the Mid-Atlantic Eisenhower Consortium and the Japanese School of Greenwich, Connecticut. The first large-scale open house took place in 2000 with 150 attendees from the

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education Association of Mathematics Teachers from across the United States. Over time, a number of lesson-study groups were formed. In 2000, Delaware implemented one of the first statewide initiatives for lesson study; at the same time, Bellevue School District in Washington instituted one of the first district-wide initiatives. At the time of this writing, 25 states are represented in the roster of lesson-study clusters, covering at least 60 school districts and over 200 schools. Those involved have suggested reasons why lesson study might work well in the United States. It is a commonsense idea that fits into U.S. thinking about educational reform. Lesson study professionalizes teaching through its focus on improving teaching practice, with teachers controlling their own professional development through concrete, classroom-based activity. At the same time, obstacles to its adoption in the United States include such logistical concerns as time, bureaucratic details, and money; the need for more in-depth understanding of the process; cultural limitations, such as the idea that peer observation is threatening to many U.S. teachers; and systemic issues, such as the pressure for accountability demonstrable through achievement scores. A tension exists in the United States between the appeal of and resistance to things Japanese. Because of Japan’s higher results on TIMSS and on other international assessments, there is an impulse to emulate successful strategies. Those trying to introduce lesson study here, however, face some resistance from U.S. educators who believe that the cultural differences are too great to allow lesson study to work in the United States. The question remains whether lesson study can be as successful a grassroots movement in the United States as it has been in Japan. SOURCE: Chokshi (2002). For more information, see, for example, Fernandez (2002) and Lewis and Tsuchida (1998). prompt a wave of practitioner-level interest in Japanese lesson study. Chokshi (2002) argues that lesson study caught on among practitioners because it is a commonsense idea, appealing to teachers’ sense of professionalism, teacher-controlled, able to fill gaps in teachers’ skills, and a concrete rather than a theoretical activity. Chokshi also reports, however, that several researchers working with teachers to operationalize and adapt lesson study anticipate that several factors will make it difficult to adapt to the U.S. context beyond the pilot phase: teachers’ general lack of time for professional development not already programmed by the school or district; lack of the necessary expertise in the practitioner community with

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education respect to implementing this approach, leading to a need for experts to assist teachers in organizing and conducting lesson-study activities; cultural limitations, such as U.S. teachers’ reluctance, relative to Japanese teachers, to allow others to observe or evaluate their teaching; current policy emphasis on standards and accountability not amenable to teachers’ experiments to increase opportunity to learn or on gradual efforts to change classroom practice; and lack of long-term perspective among school administrators and teachers, related to the U.S. focus on quick results. Although pre- and in-service professional development programs are important conduits for communicating findings from international comparative education research to teachers, practitioners may on their own initiative seek and try out international innovations that only later attract the attention of scholars and policy makers. At times, scholarly international comparative studies or professional development materials may be the second or third step in the process of introducing an international innovation to the United States, and the early-adapting teachers in the United States may provide some of the primary data for those studies. For example, as with any innovation, teachers or school boards who may have a limited understanding of the context of an international model will make sense of the model in their own terms, adapting it in ways that strengthen or weaken the effects intended by the model’s developers. As communication technologies continue to improve, and more educational materials are posted on the Internet, one can expect to see more direct grassroots borrowing of educational methods from one country to another. The TIMSS 1995 videotapes of mathematics lessons in three countries were powerful demonstrations to U.S. mathematics teachers of how different their teaching approaches could be and motivated many to adapt their instruction (Bunt, 2001). As a result, science educators insisted that science classrooms be included along with mathematics classrooms in the TIMSS-R Videotape Study. Both the Organisation for Economic Co-operation and Development (OECD) and the International Association for the Evaluation of Educational Achievement (IEA) are in the process of launching new, large, cross-nationally comparative studies of teacher education. These studies also are likely to increase teachers’ exposure to and opportunities to experiment with teaching practices from other countries. Policy The demand for TIMSS 1995 originated with policy makers who wanted to know how U.S. students compared academically with their

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education peers in other countries. In reviewing just a few of many U.S. activities undertaken following the release of the TIMSS data in 1996, Wiseman and Baker (2002) note several impacts in the United States, which include increasing the impetus for reform, rather than increasing reform capacity, per se; stirring policy makers into action at the national and state levels but discouraging educators at lower levels in the education system; providing a benchmark for the education system; increasing the quantity of domestic education research; and contributing to an improved understanding of the basic nature of the education system. At the same time, Wiseman and Baker also conclude that response to TIMSS has tended to be direct rather than interpretive, reactive rather than reflective, concentrated in high-performing jurisdictions, and having more impact on professional development than on scholarly or policy analysis. These conclusions are consistent with the work of Weiss (1991), who argues that research presented as data is usually too dense and indirect to inform policy makers who are not already in agreement on values and goals, particularly when alternative policies are not sharply drawn and the situation is changing slowly. Indeed, TIMSS was more often simplified into ideas and arguments that Weiss points out are particularly influential when existing policy is in disarray, uncertainty is high, and policy makers are looking for legitimization after decisions have been made in a decentralized policy arena. Participants in a Board on Comparative Studies in Education (BICSE) symposium reflecting on the results of TIMSS (National Research Council, 1999) expressed concern that the results would be translated into quick fixes. They emphasized that the picture of the U.S. education system provided by TIMSS was an incomplete snapshot of the state of U.S. schools at one point in time. The test covered only that part of the U.S. curriculum that was common with the curriculum of other participating countries and did not represent necessarily a balanced picture of the entire U.S. curriculum. U.S. educators who have taken the TIMSS curriculum framework or the test as a basis for their own work have not always been aware of the lack of balance and completeness. Nevertheless, policy makers interviewed in connection with several

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education of the background papers recently prepared for the board said they were convinced that TIMSS has made a significant impact on standards and assessment (Raizen, 2002; Wiseman and Baker, 2002). Rothman (2002) traces the movement toward constructing educational assessments tied to content standards, which began in the United States in the early 1990s when President George H.W. Bush announced the National Education Goals and state governors pledged that the United States would be “first in the world” in mathematics and science by the year 2000. Throughout the 1990s, the National Education Goals Panel, composed of governors, business leaders, administration officials, and members of Congress, monitored the progress toward this national goal using comparisons with international studies. By 2001, the United States had clearly failed to meet the goals and the panel was disbanded. Work on benchmarking standards with international data continues on a state-by-state basis through the nonprofit organizations Achieve and McREL.4 After TIMSS, school districts working with the Council of Great City Schools were more willing to participate in the National Assessment of Educational Progress (Sharon Lewis, personal communication, April 2002), and discussions of a voluntary national test took on new life (Lois Peak, personal communication, April 2002). Much of the story in the preceding paragraph, however, could be construed as policy mischief brought about by government officials determined to increase funding for education and armed with an extremely powerful piece of rhetoric: cross-national comparisons of education performance based on national averages. National comparisons tend to be more valid when countries have national education systems and centralized decision making. In the United States, however, more than 14,000 school districts vary enormously in terms of curriculum, funding, and performance. For example, A Nation at Risk used the mean performance level of U.S. students on SIMS, an international study with notoriously weak sampling standards, to convince Americans that there was a generalized crisis of poor quality in the U.S. school system. This crisis was used to promote standards and led directly to the formation of the National Education Goals Panel. Berliner and Biddle (1995) call this a “manufactured crisis,” on the grounds that the performance of many U.S. students is comparable to those in high-performing countries. They argue the U.S. crisis lies in the low performance of a subset of schools and students, and therefore 4   Achieve and McREL (Mid-continent Research for Education and Learning) have collaborated to “help states implement high academic standards by providing access to comprehensive standards-based resources,” http://frodo.mindseye.com/achieve/achievestart.nsf/OutsideSearch.

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education response to this crisis should be focused on those schools and students, rather than generalized to the entire U.S. education system.5 Nonetheless, U.S. officials used the mean performance level of U.S. students on TIMSS 1995 to impress on Americans once again that U.S. schools in general were not “first in the world.” Mathematics and science educators are less convinced that TIMSS has had significant impact on their fields. Raizen (2002) explains that the U.S. national science standards were completed before TIMSS was released. Dossey (2002) observes that the middle school mathematics standards proposed by Achieve and McREL appear to be directly linked to TIMSS 1995 results, but those results simply reiterate the findings of the earlier FIMS and SIMS studies using more compelling data. Policy makers are sometimes tempted to use international achievement trend studies to evaluate policy reforms (National Research Council, 2002a). However, synchronizing the attention span of the political world (1 year or less), the current cycle between international education assessments (3 or 4 years), the time needed to create a critical mass of peer-reviewed research findings (5 to 10 years), and the cycle of education reform (perhaps decades) remains a problem. Nonetheless, understanding the interaction of these four different cycles, and the funding cycles that constrain them, is a central imperative for all education research in the United States, domestic or international. Four years after TIMSS 1995, U.S. students did not manifest a significant gain on TIMSS 1999, and the results were released with little fanfare. Was it too early to detect any effects of reform? Was there not a critical mass of U.S. education jurisdictions engaged in a common reform effort, on a common schedule? Did different jurisdictions with different performances effectively cancel out each other’s effects? Even had the United States made significant gains, one journalist suggested, officials might be reluctant to showcase international education studies in which U.S. students perform well, such as reading and civic education, given that requests for budget increases are easier to defend when they are tied to crises, rather than to strong performances (Berliner and Biddle, 1995; Rothman, 2002). Recommendation 2.2: U.S. funders should give highest priority, of all the possible impact studies that might be undertaken, to an impact analysis of TIMSS 1995. This analysis should address TIMSS in terms of (a) its efforts to serve as both an indicator and a research study and (b) its impact on the U.S. education system. 5   A recent report on educational disadvantage in 24 industrialized countries, using data from the Programme for International Student Assessment 2000 and TIMSS 1999, consistently places the United States in the bottom third of all countries (United Nations Children’s Fund, 2002).

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education The Public Like teachers, the public both influences and is influenced by international research. Some examples illustrate the complex balance between research and public perceptions. In her review of three Japanese instructional practices currently used in the United States—the Suzuki method of music instruction, the Japanese system of martial arts instruction, and the kumon method of mathematics instruction—Peak (2002) identified five stages in the U.S. institutionalization of these innovations: initial interest, try out existing materials, exchange master teachers, adopt a few techniques, and create a full U.S. version. These five stages echo some of the innovations described in this report, including the Reggio Emilia traveling exhibit containing artifacts of young children’s classroom work and the lesson-study scholars linking Japanese teachers with U.S. teachers. Now that TIMSS shows students in less affluent and less developed Asian countries consistently outperforming their peers in the United States, the U.S. public may be more open to ideas and practices from other countries than it has been in the recent past. Such receptivity was harder to sustain as the U.S. economy prospered in the late 1990s, when the Asian countries that had performed particularly well on TIMSS were struggling through economic downturns. This caused some to question the often-asserted link between high performance on international mathematics and science assessments and high national performance in the global economy. In trying to understand the impact of TIMSS 1995 on discourse in the U.S. education community, several individuals6 who participated in the TIMSS 1995 release and public relations process attributed great importance to releasing the data with much fanfare and embedding them in a coherent “story line” already developed by one research group granted early access to the data. Such fanfare for a single story line, however, runs the risk of becoming a “policy trap,” as noted by LeTendre et al. (2001:1): The early release of TIMSS reports, combined with pre-existing political agendas and educational reform movements, created conditions where univariate distributions were interpreted as “hard data,” first in the media and later in policy debates, eventually undermining the ability of later, secondary analysis to impact reform initiatives. Such policy traps are not unique to TIMSS (Wineburg, 1997a, 1997b) and were avoided by several other countries participating in TIMSS by 6   Larry Suter, National Science Foundation, and William Schmidt, Michigan State University. Personal communications, 2001-2002.

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education planning secondary analysis at the time the study was designed. By comparison, in the United States, according to David Baker, “secondary is often perceived as second-class analysis” (personal communication, April 17, 2002). Studies that reveal a perceived problem in the U.S. education system tend to galvanize responses from teachers and parents associated with schools that have enough resources to craft and implement a response. By the time the early findings of a large study are challenged or overturned by peer-reviewed secondary analysis, public interest in the study has probably waned. Raizen suggested another effect of TIMSS 1995 and TIMSS 1999 benchmarking studies on the public: with the elite First-in-the-World7 consortium schools performing at the level of the highest performing countries and the Miami-Dade county schools performing at the level of such developing countries as Turkey (Mullis et al., 1998), the benchmarking studies helped to drive home to the public the bifurcation of the U.S. education system into high- and low-performing schools. In this case, such findings help call attention to an aspect of the U.S. school system that is taken for granted by many Americans, and that is a distinguishing factor in comparison with schools in other industrialized countries. Education journalists report that TIMSS 1995 and other international study results in recent years have caused both journalists and the public to be more critical of the U.S. school system (Rothman, 2002). Although TIMSS 1995 is mentioned most frequently in this regard, journalists also mentioned a study of the voucher system in New Zealand (Fiske and Ladd, 2000) and an OECD study on graduation rates in developed countries, which challenged the prevailing notion that the United States educates a greater proportion of its population than any other nation. Recommendation 2.3: U.S. funders should support reviews of the impact of different study methodologies on different audiences. The Research Community Historically, the U.S. education research community owes much to comparative education studies. Although a recent history of education research in the United States (Lagemann, 2000) contains very few references to foreign innovations and research, during the 19th and early 20th centuries, John Dewey and other leading American thinkers took ex- 7   First in the World is a collaboration between 18 school districts in the United States that aim to become first in the world in mathematics and science. http://www.1stintheworld.org.

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education tended study tours of Europe and Russia to observe foreign education systems and practices, and many earned degrees from foreign universities. From 19th century England, U.S. educators borrowed the idea of factory-like classrooms to address urban mass education; from France and Germany, the idea of early childhood education; and from 20th century Japan, the Suzuki method of music instruction, to name just a few of the internationally derived innovations that contributed to the development of the U.S. education system. These and many other popular innovations, however, were instructional and organizational techniques that practitioners could observe, apply, and adapt more or less successfully, without extensive input from formal researchers. Today more than half of the scholarly education researchers in the world are based in the United States, and in the last half of the 20th century Americans placed increasing emphasis on the scientific, quantitative study of education processes. But as education research becomes more technically sophisticated, it often becomes more complex, less accessible to practitioners, and less likely to affect instructional and organizational reform directly. Comparative education research in the past 15 years has, however, had an impact on the direction and abundance of domestic education research in the United States. Wiseman and Baker (2002) cite the importance of A Splintered Vision: An Investigation of U.S. Science and Mathematics Education (Schmidt, McKnight, and Raizen, 1997) in delving into curriculum questions that educational researchers had been asking for decades, but only TIMSS had the data to explore. They also cite the peer-reviewed publications of the Secondary Analysis of TIMSS Project at Pennsylvania State University; of those published between 1997 and 2002, six out of nine appeared in journals with primarily domestic audiences. The availability for secondary analysis of a dataset of the size and quality of TIMSS and the 2001 Civic Education Study has not been lost on younger scholars. Raizen estimates that 20 to 25 percent of American Educational Research Association fellowship proposals are based on research on TIMSS. Other observers attribute increased interest in curriculum studies in the 1990s, particularly the emphasis on mathematics and science, to issues arising from FIMS, SIMS, and TIMSS. The TIMSS videotapes certainly contributed to increased interest in studying teacher preparation and professionalism (Dossey, 2002). The other studies and innovations covered by the cases highlighted in the boxes through this chapter are much smaller, and their impact on research is difficult to track. Some high-profile, practitioner-driven efforts in the United States to try out internationally derived innovations, such as the Singapore mathematics curriculum and the Reggio Emilia early childhood program, may have the effect of getting some practitioner issues on the education research agenda.

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education Foreign graduate students, such as Liping Ma, and foreign-born scholars, such as John Ogbu of Nigeria, can play an important role in introducing new questions and ideas for domestic researchers in U.S. schools of education. Lee Shulman explained that this “comes to the heart of why comparative work is so important.” He views Ma’s book as a dramatic instance of the proposition that international work allows the outsider to see something insiders take for granted and, in the process, make it interesting, problematic, and worthy of investigating for insiders (Fang, 2002:13). IMPLICATIONS This brief, illustrative review contains many ideas for increasing the use and usability of international comparative education studies. Different types of studies have had different effects, within different time frames, on different parts of the education system. Our examples illustrate the need to look at impact at several levels of the education system. In order to reach specific groups or levels in the system most effectively, it may be necessary to incorporate them into the design both of studies and of analyses. The social and political environment affects openness to ideas from abroad; current concerns about globalization among many Americans may be elevating levels of interest in education systems in other countries. Innovations and study findings from other countries that focus on techniques and practices may be more quickly absorbed and adapted than may policy changes. Scholars who can explicitly connect their research on education in other countries to education practices and phenomena in the United States may make more of an impact than those that make no such explicit comparisons with the United States. Foreign scholars may have some comparative advantage in helping Americans talk constructively about education issues that have become polarized in domestic debate. Artifacts and videotapes may have a particularly important role in helping Americans move beyond preconceived notions of the possible and impossible in education. These examples also raise many questions: could international comparative researchers be doing more to work with U.S. practitioners to evaluate practitioner-driven, internationally inspired innovations in U.S. schools? The Reggio Emilia experiments certainly call for this, and the collaboration between lesson-study practitioners and researchers demonstrates how this might be accomplished. In the United States, there is perpetual tension between asynchronous cycles of decentralized politics, research, and education reform; do other countries with decentralized education systems manage these tensions better? Can we match up young researchers attracted by the vast datasets produced by Type I stud-

OCR for page 29
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education ies to states and districts that do not have the time or expertise to analyze their benchmarking data? Recommendation 2.4: U.S. funders should support reviews of the impact of ongoing and completed international comparative studies on the practice of education on a planned and continuing basis to determine how effects can be enhanced. In summary, those looking for ways to balance investments in different types of international comparative education research should consider more factors than cost and topics. These additional factors include the expected and actual benefits of similar, earlier studies; the time frame needed to realize those benefits; negative effects, such as creating an unduly negative picture of the U.S. education systems; and the relative effectiveness of different types of study designs and methodologies in producing “useable” research for different audiences in parts of the education system. These factors are not presently well understood in the education research community and in themselves constitute areas for increased education research funding.