Cover Image

PAPERBACK
$34.75



View/Hide Left Panel

4
New Directions

Despite the emphasis thus far on developing a research agenda more balanced among Types I, II, and III, this chapter contains more specific recommendations for Type I studies than for any other type. This is because the preponderance of the board’s work over the past 10 years has been focused on Type I studies. This sustained focus on Type I studies has impressed on the board not only the importance of Type I studies, but also their limitations and potential abuses and the need for monitoring and evaluating these major investments. Moreover, this concentration has convinced the board of the need for systematic, increased investments in Type II and III studies that can both inform Type I studies and go beyond what they can achieve. Type II and III studies are essential for expanding our understanding about how education can and does work and in exploring questions about context between and within countries that Type I studies cannot yet begin to operationalize. As for ways to improve Type II and III studies, more specific recommendations should be based on more experience with funding such studies on a more systematic and broader scale.

GETTING MORE FROM TYPE I STUDIES

Large-scale, cross-national Type I surveys have dominated U.S. funding for international comparative education research for the past 15 years and, even in a more balanced portfolio, are likely to continue to represent a large proportion of the total budget. Several issues affect the degree of utility that Type I studies can offer; most important are the ways in which



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 51
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education 4 New Directions Despite the emphasis thus far on developing a research agenda more balanced among Types I, II, and III, this chapter contains more specific recommendations for Type I studies than for any other type. This is because the preponderance of the board’s work over the past 10 years has been focused on Type I studies. This sustained focus on Type I studies has impressed on the board not only the importance of Type I studies, but also their limitations and potential abuses and the need for monitoring and evaluating these major investments. Moreover, this concentration has convinced the board of the need for systematic, increased investments in Type II and III studies that can both inform Type I studies and go beyond what they can achieve. Type II and III studies are essential for expanding our understanding about how education can and does work and in exploring questions about context between and within countries that Type I studies cannot yet begin to operationalize. As for ways to improve Type II and III studies, more specific recommendations should be based on more experience with funding such studies on a more systematic and broader scale. GETTING MORE FROM TYPE I STUDIES Large-scale, cross-national Type I surveys have dominated U.S. funding for international comparative education research for the past 15 years and, even in a more balanced portfolio, are likely to continue to represent a large proportion of the total budget. Several issues affect the degree of utility that Type I studies can offer; most important are the ways in which

OCR for page 51
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education they are coordinated and monitored with each other and with other types of studies. The board considers that much has been learned from the conduct of past studies, and in this chapter we lay out our recommendations for improving the conduct of future Type I studies. The emphasis on Type I studies in the 1990s was a logical response to the way that results of earlier surveys of dubious technical quality (U.S. Department of Education, National Center for Education Statistics, 1994) dominated U.S. education policy debates in the late 1980s. Education analysts at the National Center for Education Statistics and the National Science Foundation led the effort to secure the massive funding necessary to raise the quality of the 1990s round of educational achievement surveys in order to establish more rigorous, representative comparisons and raise the quality of discourse about the U.S. education system and its relationship to those in other countries. The board’s report Methodological Advances in Cross-National Surveys of Educational Achievement (National Research Council, 2002a) explores the consensus among leading experts regarding technical improvements in assessing student achievement over the past four decades. In the construction of achievement tests, the report concludes that psychometric advances in differential item functioning, translation procedures, and clearer standards for item statistics represent significant improvements. Best practices in international assessment have been codified to some extent in documents such as the Technical Standards of the International Association for the Evaluation of Educational Achievement (IEA) (Martin et al., 1999) and Measuring Student Knowledge and Skills: A New Framework for Assessment of the Organisation for Economic Co-operation and Development (OECD, 1999). Improvements in sampling and in documentation have generated increased confidence in survey findings. Computer technology has radically increased the amount of data that can be managed by one survey, as well as the level of effort needed to do it. Recent surveys also include better measures of social background (in the Second International Mathematics Study and the Programme for International Student Assessment (PISA), less so in TIMSS), valuable data on opportunity to learn, and more powerful inferences drawn from increasingly complex statistical analyses of achievement data. Room for improvement is also evident in several areas in Type I studies. By discussing this topic, the board does not necessarily imply that some studies were designed or have been implemented badly. Rather, the experience of implementing past studies has brought to light many new issues and dilemmas that now need to be addressed. For example, areas for improvement for TIMSS include: using the study components to strategically inform each other; integrating the mathematics and science portions of the survey; incorporating advances in measurement of oppor-

OCR for page 51
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education tunity to learn, social background, and other independent variables into successive studies; appreciating the importance of distributions of achievement as well as achievement levels; and gaining flexibility in adapting to the needs and findings of secondary analysis. Many of these issues are related to the decision to make TIMSS into a trend study, to be repeated every four years. Others are related to larger issues of coordination, discussed below. Finally, to date, TIMSS has been unable to resolve the sampling and response rate problems that rendered TIMSS 1995 cohort III (12th grade) largely unusable. One of the purposes of the 1990 Framework was to help set technical standards for the few international studies in which the United States had an opportunity to participate. Today, with the improvements in technical standards and the proliferation of studies cited throughout this report, the pivotal issue in the U.S. decision to participate in any given study is the potential impact of the study in light of its costs and the extent to which it fits into a systematic plan. The need for such a plan is highlighted by the current confusing array of Type I studies under way or under development, with similar topics, overlapping implementation schedules, and little reference to each other. This confusion persists because the decision-making process with respect to funding individual studies has been relatively ad hoc. To date, Type I studies lack coordination from a body that is (a) relatively independent of the particular Type I studies themselves, (b) willing and able to identify and prioritize supporting Type II and III studies, (c) able to make choices and to foster channels of communication among various studies, and (d) able to find ways to channel the results of such research into the national educational discourse. Recommendation 3: On a continuing basis, the federal government should plan, coordinate, monitor, and modify studies in the government’s portfolio of international comparative education research. Coordination In many public policy domains, coordination and collaboration are the ideal, not the norm. Competition for resources is fierce; careers are made by investing in one approach and shunning others. While the technical quality of international large-scale research has improved dramatically during the previous decade, with the growth in the number of Type I studies comes an increase in the importance of political issues. Who determines which topics are studied? Who defines the research questions? Who supports which components? Whose agenda is being advanced? Who controls it? How much data will they share? When? To what degree is diplomatic capital needed and deployed? Who decides whether one

OCR for page 51
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education large study is more appropriate than a dozen smaller ones? Methodological improvements do not address these questions. Moreover, if one research group or country does not deal respectfully with others’ concerns early in the design process, then the best technical standards in the world will not guarantee broad agreement on the validity of the final results. For example, to date most Type I studies have not been designed by groups broadly representative of the international comparative education research community and have not integrated practitioner, student, and parent concerns into their work. As a result, Type I study designs often do not reflect critical debates in the social science research community about how knowledge is defined and distributed and how that process reproduces inequality in society. As a result, Type I studies are often of little interest to the scholars most capable of designing the Type III studies that may well explain Type I findings. The three trend studies addressed in the next section—TIMSS, the Progress in International Reading Literacy Study (PIRLS), and PISA—are not the only Type I studies competing for funding from the U.S. government and time in U.S. classrooms. In the past five years there have been two simultaneous cross-national surveys of technology in the classroom,1 and in the first half of 2002 the OECD and the IEA simultaneously circulated three proposals for separate cross-national studies of teaching or teacher professional development. This proliferation of Type I studies with overlapping topics and target groups adds to the testing burden in U.S. schools, rendering it difficult to secure a valid national sample. Moreover, the limited pool of experts to design the tests and to thoughtfully consider the form of old and new measures is also strained. The appropriate mechanism for providing such coordination for Type I studies does not currently exist. In its most expansive form, such a mechanism might be comprised of an independent advisory board broadly knowledgeable about international comparative education research, as well as the policy and practitioner communities, with separate panels for each major Type I study (playing the role of the current technical review panels) and several panels for Type II and III studies. A less ambitious mechanism would focus on Type I outcome studies only and might be a panel that is part of an existing advisory group, such as the National Assessment Governing Board or the Board on Testing and Assessment of the National Research Council. Whether board or panel, this group would serve as a peer review panel to suggest priorities, ensure 1   The OECD’s Information and Computer Technology study and the IEA’s Second International Technology in Education Study.

OCR for page 51
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education that they are implemented, and help make difficult choices for the U.S. government’s international comparative education research program. Recommendation 3.1: The U.S. government should support a broadly knowledgeable body, independent of both the funders and study designers, to oversee coordination of complex and competing large-scale studies. The immediate tasks of this body should be to oversee an impact study of TIMSS, make recommendations on TIMSS and PISA in order to avoid costly and unnecessary duplication, and review proposals for official U.S. participation in new or repeat large-scale, cross-national studies. This body should act in an anticipatory rather than a responsive way, stepping back to look at the qualities and characteristics of these studies and how they fit together, before decisions to participate. It would consider how the various studies fit together conceptually. It would identify gaps in the understanding of education in other countries; of contexts for teaching, learning, and conducting research in those countries; and of international education phenomena. Finally, this body would provide opportunities for policy makers to participate in selecting topics for an agenda and formulating research questions for future studies. Research and Indicator Studies The magnitude of the costs—both direct and indirect—and the need in the case of indicator studies to sustain budgetary commitments over many years make issues of duplication and coordination much more important in Type I than in Type II or III studies. In some cases, Type I research studies can, with careful planning, increase the potential return on the required investment when variables, constructs, and subgroup samples are added that expand secondary analysis opportunities. Such additions, however, carry their own costs in terms of longer questionnaires or additional survey instruments, more complex sampling frames, and increased test burden on the participants and the education systems that serve them. In the case of Type I indicator studies that aim to measure trends over time, the frequency with which the study is repeated and the complexity of the study framework both have the potential to increase costs. The U.S. cost of both types of studies is affected by the degree to which participating countries share the costs of the study. Research and trend studies are not perfectly interchangeable, although sometimes they have been treated as if they are. Restructuring a research study into a trend study severely limits its usefulness as a research study. An international trend study must be carried out using

OCR for page 51
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education very similar procedures over time and with a relatively stable group of countries. By definition, it cannot respond to changing questions and conditions without either compromising its integrity as a measure of trend or incurring the substantial additional and extraordinary costs necessary to bridge studies from old to new procedures. Methodological stasis in indicator studies is a virtue. Research studies, in contrast, are expected to build on one another, using primary and secondary analysis from one study, and from other independent studies, to advance conceptions of what associations might exist, to improve methods of measurement, and to better define important population subgroups. All of these activities involve modifying variables from one research study to the next, and the time needed to do this is usually longer than three or four years. Hence methodological stasis in research studies is a weakness, as are frequent repetitions. This suggests that the methodological imperatives of indicator studies and research studies are basically in conflict, and much is lost when research studies are also made to serve the purpose of indicator studies. Recommendation 3.2: U.S. government sponsors should establish the purposes to be served by each component of individual studies, so that single studies will not be asked to do too much and so that expectations are clear regarding the study’s potential to inform policy, produce databases appropriate for different types of primary and secondary analysis, and serve other specific functions. In the last half of the 1990s, the United States expanded the number of Type I trend studies in which it participates from zero to three: TIMSS, PIRLS,2 and PISA. The first of these studies, TIMSS, is implemented by an organization that had previously specialized in research studies. During the design of TIMSS 1995, IEA members pressed for additional components and variables that made the study look more like a traditional IEA research study and raised expectations that TIMSS data would be useful for a wide variety of secondary analyses. Later, others were disappointed to learn that TIMSS 1999, unlike the research studies more familiar to IEA members, would not address weaknesses recognized relatively early in TIMSS 1995, such as the need for improved assessment of common versus unique content, adjustments for between-country differences in background conditions, and analysis of within-country variance versus central tendency. As a trend study, the work on and refinement of the central 2   PIRLS was first implemented by IEA in 2000 and was scheduled to be repeated in 2003; however, the schedule has changed so that it will be repeated in 2006.

OCR for page 51
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education variables for the TIMSS studies was essentially completed during its original design phase in 1992 and 1993 and could not be changed without disrupting the trend. At the time TIMSS 1995 was being planned, the OECD had just begun designing PISA, and concerns about duplication were theoretical. These two studies were designed to be very different in their goals and sampling. TIMSS is intended to measure how much curricular content 4th-and 8th-grade students have learned; PISA is intended to measure the degree to which 15-year-olds can apply what they have learned in school and elsewhere to real-world situations. In practice, however, the two studies are drawing samples from one common age group in the United States and assessing common topics. Given the lengthy time frame often necessary for the results of major education reforms in the United States to become manifest, trend studies of international student achievement probably provide valid indicators of progress no more than once or twice a decade. All the improvements expected in a research study probably cannot be addressed with only a 4-year interval between large-scale studies; at the same time, a 10-year interval is probably too long for indicator studies. PISA, with minor repeats of two skill areas and a major assessment of one area every three years, provides for a major assessment every nine years and fits in this time frame—provided it remains a relatively streamlined test, not overloaded with independent variables and components that encourage in-depth analysis of determinants of achievement better suited to a large and complex research study. In contrast, TIMSS and PIRLS are currently on four-year cycles, a frequency that may be too short for either an indicator or a research study. Recommendation 3.3: U.S. government sponsors should avoid duplication of studies that create unnecessary costs and demand too much time from respondents. Does the United States need to participate in three Type I trend studies in order to secure two different trends each for mathematics, science, and reading? Are they too costly? Too complicated? Too frequent? Sufficiently different in terms of their purpose and impact to justify participating in all three? More importantly, shouldn’t the United States be participating in a regular Type I research study for one or more of these subjects? In addition to these questions, another is also relevant: What are the other countries participating in these studies willing to do? To date, many years of work have been invested in all these studies, and many difficult compromises have been worked out among the participating countries. These questions probably cannot be addressed by indi-

OCR for page 51
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education viduals and agencies deeply invested in one or more of the studies; they are better addressed by a relatively independent oversight body, such as the body described in Recommendation 3.1. Datasets or Studies? In our view, multiple, timely, primary and secondary analyses could play an important role in stimulating a healthy debate about the proper interpretation of Type I studies, as illustrated in the board’s 1999 report on secondary analysis of TIMSS (National Research Council, 1999). The production of these analyses can be facilitated by the rapid release not simply of quantitative data and codebooks, but also of curriculum guides, student background questionnaires, videotapes, and other artifacts of the study process. Such data, properly archived, allow scholars opportunities to reanalyze the primary data as well as the artifacts that created the data. The cost of such archives, although significant, could help to maximize the return on the investment that Type I studies represent. Nonetheless, few large-scale international studies will assemble datasets that can be used by other researchers for purposes more diverse than those underlying the original design. TIMSS, for example, was not designed to serve as a national education census for each of the participating countries. To promote appropriate use of TIMSS data, the National Science Foundation has sponsored a series of training sessions for researchers interested in analyzing the TIMSS quantitative data. The TIMSS Classroom Videotape Study has adopted a more regulatory approach: before researchers can use the video data, they must be trained and licensed. Costs associated with collecting the data and making them available to secondary researchers need to be weighed against (1) the number of secondary researchers likely to use it, and (2) the potential impact of their work. For example, the videotapes of 8th-grade mathematics classes in the United States, Germany, and Japan have arguably generated as much interest and action as any other component of the TIMSS 1995 study: a single taped episode featuring just two classrooms in each country, one book, and related articles derived from the study have produced substantial buzz. Meanwhile, federal rules intended to protect the confidentiality of research subjects currently limit access to the TIMSS videotapes to a few licensed researchers able to travel to the TIMSS videotape study center, previously at the University of California at Los Angeles and now in Washington, D.C. Consequently, many years after the “release” of these expensive tapes, coded at significant cost, and the funding of a center to make them available to licensed researchers, no secondary research has been published.

OCR for page 51
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education Recommendation 3.4: U.S. government contractors and grantees should provide rapid release of quantitative data and codebooks, curriculum guides, and study artifacts from all types of studies. Federal sponsors shall assure that these materials are archived in such a way that, to the extent possible, scholars have opportunities to reanalyze primary data, and that archives are kept open and available for a decade or more. All study designs, for all types of international studies, need to include explicit plans for both analysis and dissemination. With respect to Type I and II studies, the end-users of data need to be involved at the formulation stage of both studies and databases. For all studies, to the extent that practitioners and policy makers at the state and local levels are intended end-users, their input should be secured at the earliest stages of formulating the design and planning dissemination, and budgets should include line items for local-level analysis and dissemination of results. Doing so will necessitate more collaboration among researchers and state and local officials and practitioners than has been typical in many Type I studies to date. Recommendation 3.5: U.S. government sponsors should set aside funding for data analysis for state and local district participants in future international benchmarking projects. GETTING THE MOST FROM ALL STUDIES Less than three decades ago, many Americans did not believe the U.S. education system had much to learn from countries with smaller economies and lower gross national products, such as Japan and Singapore. That is no longer the case. In recent years, both TIMSS and more general trends in globalization have increased Americans’ awareness of differences in educational achievement across countries that defy economic might. However, people’s understanding of the potential reasons for those differences has not necessarily increased. More Americans may be open to innovations from other countries, but the absolute number of practitioners and policy makers looking for answers to education questions in other countries remains small. Moreover, this search has been hit or miss, with misapprehension of differences in the context of education in the United States and other countries confounding attempts to experiment with and adapt techniques and curricula. How can the portfolio of high-quality Type I, II, and III research be balanced in a way that that will both expand researchers’ basic under-

OCR for page 51
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education standing of educational processes and support the efforts of practitioners, policy makers, and the public to apply this new understanding to the U.S. education system? Accomplishing this implies a more systematic approach to analysis sensitive both to the different questions each group may pose and to the complexities of communicating the results of all types of international comparative studies of education in ways appropriate to each group. Recommendation 4: The U.S. agenda for international comparative studies in education should include a prominent place for interpretive analyses that aim to enhance public understanding of education in other countries. Recommendation 4.1: Analysis plans should be developed as part of study plans so that the sampling plan, the construction and inclusion variables, and links with other datasets will support these uses. International comparative education studies in the United States have a long history of generating new ideas and contributing to general knowledge about education and its potential, but only occasionally have they informed national policy in an immediate, direct, and appropriate manner. As discussed in Chapter 3, the contribution of all three types of studies to efforts to improve U.S. education have been mixed, often because many have lacked mechanisms to help the public, practitioners, and policy makers make sense of and use their findings. Similarly lacking are mechanisms to help with the flow of ideas in the opposite direction, from practitioners to researchers. Practitioners continue to identify and pilot many promising internationally derived ideas, but even the most appropriate ones often founder for lack of critical information about context, such as practitioner support structures or access to extrabudgetary resources. The TIMSS Toolkit represents one effort to disseminate artifacts and ideas—rather than arguments—from the U.S. Department of Education to practitioners at the district and local levels. Peak et al. (2002) note that one of the most popular pieces of the toolkit was a translation of a Japanese curriculum guide, something that brought to life the differences between U.S. and Japanese curriculum planning. Watching the videotapes of Japanese and German classrooms included in the kits also provided an opportunity for practitioners to raise their own questions, such as “Why don’t our classrooms look like that?” Similarly, the visually engaging traveling exhibit of the Reggio Emilia approach resonated with theories already emerging in the early childhood education field in the United

OCR for page 51
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education States and presented an example of one way in which those theories might be translated into classroom practice. The need for more artifacts, ideas, and results from all three types of studies tailored to practitioners, policy makers, and the public is illustrated by the relatively narrow public discourse that persists around the largest most publicized international comparative education studies, TIMSS and its follow-on studies. Despite a major public relations effort mounted by the U.S. Department of Education at the time of the TIMSS release, the majority of the public still does not understand why Japanese and Singaporean students consistently perform better on this assessment than do U.S. students. The response to poor U.S. performance has been to do more assessment and to imitate some easily accessible features of high performing countries’ education systems, such as the curriculum. To the extent that practitioners and policy makers come into contact with artifacts, such as textbooks and curriculum guides, and with practitioners from other countries, and these breathe fresh air into the U.S. system, this can be a constructive process. To the extent, however, that practitioners and policy makers seize on one component of a complex foreign school system and adopt it on a large scale without first piloting it on a smaller scale and studying it in context, the results are likely to be disappointing and costly. While Type I studies have succeeded in increasing public interest in improving U.S. schools, Type II and Type III studies hold the best hope for deepening many Americans’ curiosity about education systems in the rest of the world and envisioning ways in which other ideas might enrich our own. To some foreign observers, the United States seems obsessed with the relationship between mediocre performance on TIMSS today and international economic competitiveness in the future (National Research Council, 1999:22). In contrast, some high-performing countries are looking for ways to use schooling to nurture the sort of creativity and dynamism manifest in American public life, both economic and political, and are prepared to see decreases in their scores on future international assessment to achieve that. No specific entity in the U.S. Department of Education currently has responsibility for getting the message from Type II and III international studies out to the U.S. public. In the case of TIMSS, the National Science Foundation sponsored the work of one researcher to develop a relatively simple narrative about the TIMSS results. As noted earlier, this approach, while ensuring that at least some message gets beyond the research community, also generated a false sense of certainty about what TIMSS has to say, reducing interest in future analyses of TIMSS data. From this experience, the board concludes that communicating the results of complex Type I studies to the American public requires not a simple story line based on

OCR for page 51
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education standard statistical analysis, but an ongoing, public discussion of the meaning and limitations of such studies and the introduction of data from other studies to inform and frame it. Different audiences find different types of data and arguments convincing and useful. Such a debate should address pivotal comparisons, such as those highlighted in Rotberg (1998). For example, what is the relative performance of the United States and high-performing countries in terms of: productivity in science and engineering; research opportunities in institutions of higher education; participation of women and minorities in science and engineering; and access to higher education in science and engineering for low-income students and historically disadvantaged groups. Answering these kinds of questions requires more than doing another, better, large-scale survey. Rather, it requires a series of Type II and III studies to explore differences in meaningful ways that may or may not lead to operationalization of the salient variables in terms that allow quantitative comparisons. Education researchers, whether domestic or international, have struggled to capture and maintain the attention of their intended audiences. The research and public policy communities have not invested many resources in understanding what dissemination strategies work best under what circumstances for which audiences. The idea that one story line is the best way to secure public attention, for example, has not been tested. Three Type I studies (PIRLS, PISA, TIMSS-R) that were recently released with very little fanfare provide an opportunity to experiment with various mechanisms for capturing the attention of various audiences at some time after initial release of large datasets—for example, after secondary analysis has revealed more than one new interpretation of the data. Could such differences be used to foster a healthy debate on education in the United States, or will they frustrate the public and reduce the credibility of international studies? Recommendation 4.2: U.S. funders should support analyses from multiple perspectives as soon as possible after data have been collected so that the public can be exposed to a range of perspectives and interpretations, some complementary (addressing differing questions) and some competing (addressing similar or the same questions).

OCR for page 51
Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education For example, the National Science Foundation has made funds available for researcher-designated secondary analysis studies of TIMSS. Sponsoring more than one initial analysis of primary data, providing funding for secondary analysis, and investing in activities to communicate the findings of these studies to the public, however, will not foster a healthy climate of discussions by themselves. Type I studies still only answer the “what” questions, that is, “at what level are U.S. students performing?” It takes Type II and III studies to answer a higher level of question, that is, “why are U.S. students performing at this level?” and “what can be done to improve that level?” Various hypotheses about how to use different types of international studies to increase and elevate public discourse about education in the United States deserve to be explored and their impact examined. For example, in the previous section we suggested that researchers should team up with practitioners and state and local policy makers in formulating the underlying questions for international studies, and that they return to those audiences to discuss results and dissemination in order to increase the likelihood that these audiences will make use of the results. Is that more or less effective than simply providing state and local officials and practitioners with access to more artifacts, such as videotapes and curriculum guides from other countries, and to more engaging summaries of provocative, detail-rich Type III studies? Under what circumstances? When do international components tied to domestic research studies increase the potential interest of those studies to various audiences and provide more constructive, evidence-based ways to discuss issues polarizing parts of the U.S. education community? Recommendation 4.3: Special activities, publications, and other media should be planned to showcase the results of international studies in formats that are intelligible and engaging for practitioners, the public, and policy makers, and, when necessary, producing reports individually targeted to just one of these audiences. In summary, the funding of multiple primary and secondary analyses and the funding for the preparation of engaging, audience-specific vehicles for communicating the results of those studies are mutually reinforcing strategies for using international studies to promote a more constructive, evidence-based discussion of ways to improve the U.S. education system.