Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
A FRAMEWORK FOR INTERNATIONAL COMPARATIVE STUDIES IN EDUCATION Investment in education is one of the principal means by which individuals and societies improve their well-being. The President and the governors have recognized the importance of education to the future of the country and have adopted a set of national goals for education. International comparative studies of education can assist school teachers and other pro- fessional educators, policy makers, the general public, and the research community in improving education in the United States and in measuring progress toward the realization of the national goals. This document presents a framework for use by the Board on International Comparative Studies in Education in advising the National Center for Education Statistics and the National Science Foundation on U.S. participation in international com- parative studies of education. The framework may also be helpful in identifying areas of research that are neglected. This section of the paper covers four topics: the value of U.S. participation in international comparisons; kinds of com- parative educational studies; measurement of educational achievement; and long-term criteria for U.S. participation. The Value of U.S. Participation in International Studies The most important reason for United States participation in international studies of education is to improve understanding of our own education system, that is, as an extension of and complement to studies within the United States. Since there are no absolute standards of educational achievement or per- formance, comparative studies are vital to policy makers in setting realistic standards and in monitoring the success of educational systems. Through the use of standardized tests, school officials are able to compare the performance of their pupils with some external standard. Studies that compare academic performance 1
2 INTERNATIONAL COMPARATIVE STUDIES IN EDUCATION among schools within a single system provide information to school boards about the relative success of different schools; comparisons over time provide information about improvement or decline over the years. These comparisons, however, are limited by the nature of the reference groups or criteria used: that is, they are usually limited to school systems similar to those being evaluated. Even if schools are doing well when evaluated by local standards, how do boards know how well it is possible to do? Comparisons with other localities are helpful. A natural comparison is with other similar local educational systems within the same state, or with those in other states or the nation as a whole. Such comparisons have been done at the national level for a number of years by means of the National Assessment of Educational Progress (NAEP). In 1990 data were also collected at the state level on a trial basis so that state-by-state comparisons can be made for participating states. Comparisons with other states or the nation as a whole have the advantage of compar- ing between educational systems that are broadly similar. But this advantage is also one of their limitations. International comparisons expand the range of comparison beyond the parochial limits of the U.S. national experience. They provide information on the U.S. level of achievement in relation to the much broader range of the worId's education systems. Recent international comparative studies, for example, have revealed that U.S. pupils could attain a much higher level of achievement in science and mathematics than they currently do. Collection of data at regular intervals from a large and diverse group of countries is thus important for descriptive or · ~ monitoring purposes. Comparative studies can also be helpful in understanding the reasons for observed differences in performance. Studies that explore the relationship between school achievement and such factors as curricula, amount of time spent on school work, teacher training, classroom size, parental involvement, and a host of other possible explanatory variables profit from expanding the range of variation in such factors to the international level. While there is some variation in the characteristics of school systems in the United States, they are not radically different from each other. Schools in different parts of the United States
FRAMEWORK 3 probably differ more in terms of the characteristics of their student bodies than they do in the ways they teach or in their curricula. Thus, the generalizability of the results from U.S. studies are quite limited. Careful international comparative studies can help identify the factors that promote educational achievement and those that do not make a difference. Such studies are difficult to do, however, because there is consider- able uncontrolled variation In variables other than those of policy interest, which may make it difficult to reach sound conclusions. International studies can also be important for issue-centered studies. Sometimes another country will exemplify a particular characteristic with special sharpness that makes it worth an intense comparative study. For example, the study of a country in Asia where student motivation in science and mathematics is extremely high, of a country in Europe where there is a high level of value homogeneity among schools, churches, and families, or of a country where employers participate actively in vocational education might provide insight into ways in which U.S. schools could be improved or into policies that are unlikely to work in the United States even though they may work well in another country. In addition to their value as an extension of internal U.S. evaluation studies of education, international studies of education are important for subsidiary reasons. Instances are increasing in which having an American-only sample is inefficient for the purpose of developing improvements in the effective delivery of education. The issue here is not whether an observed pattern is typical, but rather whether something that exists in another country, but not in the United States, would be useful here. From one nation to another, education as an enterprise contains many similar exigencies and challenges: its methods of finan- cial support; its role in determining what skills are provided at public and private expense; its mechanisms for treating the reaming impaired and the socially underprivileged; its mechanism for rewarding excellence in teaching; and, ultimately, its deci- sion as to what knowledge is most worth having. These are not in any sense American challenges; they are universal. Both local and state U.S. education officials depend on a constant source of good ideas on which to base their man- agement efforts. The range of ideas within a single district
4 INTERNATIONAL COMPARATIVE STUDIES IN EDUCATION usually is less than across many districts, less across districts than across states, and less across states than across nations. New ideas gained from international studies can be tried in the United States to see if they will improve the educational system. But it is critical to know more about input the system variables and the goals for education to understand why the performance of students in different countries differs. International educational research also enhances the research enterprise itself. Many of the advances in the theory and practice of comparative educational research have come from innovations developed during international comparative studies in which the problems of comparison are most challenging. In many countries, working on an international research project helps to disseminate rapidly new models, new computer programs, and new statistical techniques. These in turn become debated locally, improved, and used again by those working on local problems. Finally, international research records the diversity of educational practice. In any enterprise as diverse as education, there are practices and policies that deserve to be chronicled, not just on the grounds of their perceived utility, but on the grounds that they exist: for example, the number of languages taught in the classroom, the prevalence of pen and ink, the memorization of sacred texts, the use of Mark Twain as literature. It cannot easily be said that having information on these issues in different countries is likely to improve the practice of U.S. education, but it is worth knowing what exists in the world and, if the practices die out, what did exist but did not survive. Such knowledge may help educators avoid reinventing a faulty wheel. In sum, international comparative research on education provides an important addition to research within the United States. It increases the range of experience necessary to improve the measurement of educational achievement; it enhances confidence in the generalizability of studies that explain the factors important in educational achievement; it increases the probability of the dissemination of new ideas to improve the design or manage- ment of schools and classrooms; and it increases the research capacity of the United States as well as that of other countries. Finally, it provides an opportunity to chronicle practices and policies worthy of note in their own right.
FRAMEWORK 5 Kinds of Comparative Studies Comparative studies of education can be arrayed on a con- tinuum ranging from theoretically grounded studies intended to build or test complex models of educational systems to de- scriptive studies whose purpose is to monitor or document characteristics of educational systems, practices, or outcomes. More theoretically oriented studies tend to examine relationships among variables and look for causal explanations. For example, they might be designed to examine links between school achievement and such characteristics as curricula, teaching methods, family expectations, and funding levels. These kinds of studies are intended to identify influences on learning and how learning can be improved. They may focus on differences or amount of variation between schools or classes as well as on differences between students as the unit of analysis. These studies are expensive to conduct, but they are essential for policy makers and practitioners in their efforts to improve schools and the achievement of pupils. Less theoretically oriented studies may only collect compara- tive data on test performance, curricula, school calendar, teacher salaries, or other indicators of the educational system. The goal of such studies is to provide useful, precise information on a few simple variables. The power of these studies lies with their rigorous sampling and, hence, their capability to make national estimates of the variables stuclied; the clarity of the findings; and the speed with which findings can be reported. The limitation is that they usually pronde little or no data with which to interpret the reasons for observed differences. Many of these studies consist of ecluca- tional information that can be periodically monitored: the level and variation of teacher salaries; the number and kind of available reading materials; and the level and variation in learning achievement in the more common subjects such as mathematics, science, and reading. Much of this information enrollment, dropout rates, budgetary statistics, etc.can be obtained from official sources and does not require special data collection, although there are often problems in the comparability of official statistics due to differing definitions of data elements. Other data, particularly those on academic achievement, must be gathered by special studies.
6 INTERNATIONAL COMPARATIVE STUDIES IN EDUCATION Descriptive studies measure trends over time, establish the range of variation that exists among countries, or chart the progress of educational reforms. They are of increasing inter- est to governmental policy makers as governments are more concerned with the relationship between investments in human capital and economic performance. From time to time, special studies can focus on a problem, an issue, an exemplary program, or a contrast in educational policy or practice that can be illuminated by the study of schools in a small and selected group of countries. Sometimes countries will represent a "naturally occurring experiment" for example, countries that have different teaching methods, countries that vary in the degree of involvement of parents, or countries that vary in their relation to particular employers or higher educa- tional institutions. Issue-centered studies are likely to use a wider variety of methods than descriptive or explanatory stud- ies and will sometimes take the form of case studies. The Measurement of Educational Achievement The term "educational achievement" is used to refer to skills, knowledge, and understanding that students acquire as a result of their participation in the educational programs of schools. Achievement is usually measured by some sort of test that may be but often is not related to the curriculum being taught in the schools the students attend. Studies of educational achievement may also be concerned with aspects of school systems that have some presumed relation to achievement, such as enrollments and dropout rates, as well as such characteristics of school systems as teacher qualifications, length of the school year, and amount of money spent per pupil. Informative studies of educational achievement often include attention to students' motivation to learn and to expend the effort necessary to perform well on tests. There is no commonly agreed upon measurement scale for educational achievement analogous to the thermometer or the yardstick. In the absence of any common scale, the measure- ment of educational achievement relies on two strategies. The first is an explicitly comparative approach. A test is constructed whose content contains material on the knowledge and skills
FRAME WORK 7 that is thought to encompass the range of material taught in the grades of the students who are being evaluated. Then the tests are given, and norms are constructed on the basis of results obtained from those taking the tests. Results for individual students, classrooms, schools, or school systems are reported as numbers that are compared with the distribution of scores for the whole population taking the test. Thus, results would be reported for a student as "reading at the level of the average fourth grader," when the average for fourth graders is based on the empirical results of a large number of fourth graders who have taken the test. An alternative approach is to decide on a level of knowledge that is expected to be achieved by the average student at some level of development, for example, the fourth grade. This level may be set by teachers, curriculum specialists, school boards, parents, or any group that has responsibility for evaluating educational outcomes. In this form of testing, results are re- ported as the proportion of items that a particular pupil answered correctly or the proportion of students in a particular classroom, grade, or school system that reached a designated criterion level: for example, "80 percent of the students in the fourth grade of a particular school system know the multiplication tables through go, The two approaches yield somewhat different information. The first, sometimes called "norm-referenced testing," shows how particular students or groups of students compare with a reference population, for example, fourth graders. The second approach, sometimes called "criterion-referenced testing," shows how much particular students or groups of students know in relation to a defined body of knowledge. These approaches are not mutually exclusive. Norms can be reported for crite- rion-referenced scales, and any well constructed test can be criterion-referenced by "anchoring" its scales if there are enough items and those items discriminate at various scale points. Neither strategy dominates educational evaluation today, although cri- terion-referenced testing, which is a newer approach, is be- coming more popular. The Iowa Tests of Basic Skills are a familiar example of norm-referenced testing; the National As- sessment of Educational Progress (NAEP) is an example of cri- terion-referenced testing.
8 INTERNATIONAL COMPARATIVE STUDIES IN EDUCATION Most achievement tests, whether norm referenced or crite- r~on referenced, are multiple-choice paper-and-pencil tests. There is increasing interest in both the United States and other countries in performance tests, that is, tests that require pupils to do or write something that demonstrates their ability to solve a problem or perform an activity. For example, students might be given a ruler, a protractor, and a piece of paper and asked to draw an equilateral triangle. Performance testing is particularly valuable if it can become part of the ordinary classroom activity and is not seen as a separate and intrusive activity, as is often the case with achievement testing. Progress has been made in developing performance tests that meet the measurement criteria necessary for valid comparisons, but further developmental work is necessary before they can be easily accommodated in international comparisons. Perfor- mance testing is also more expensive and makes more logisti- cal demands on test administrators, which create further barri- ers to their widespread use. Nonetheless, the board believes that performance testing is a promising methodology that would have considerable value in both public and teacher acceptance of international comparative studies. development. Another issue in measuring educational achievement concerns the way to select the students to be tested. Because the school years are also years of rapid physical and mental growth inde- pendent of any schooling, it is not clear whether students should be tested according to their age or to their years in school. Children start school at different ages; first graders may be 5, 6, or 7 years old, depending on their birthdays, the particular rules of the school system in which they enter school, and parental preference. Grade progression also occurs at different rates across countries. Some of the Nordic countries have policies against repetition. Thus, if one were interested in evaluating achievement at about the transition between "lower" and "middle" school, should one test fourth graders or 9-year olds? in com- paring systems with different age rules for school entry, there may be quite large differences in the average age of students in the fourth grade. Again, there is no consensus on which strat- egy is most appropriate, and different testing and evaluation programs have different decision rules on this issue. We encourage its further
FRAME WORK 9 Population coverage is also an important issue in compara- tive studies. Countries differ in many ways: the legal age of students leaving school, the proportion of students dropping out before completing normal schooling, and the degree of channelling of students into different programs and types of schools. When comparing student performance among countries, it is important that the populations sampled are defined in similar ways and that there is comparable coverage of the populations. The reliable and valid international comparison of educa- tional achievement is not a simple matter. While the theory and methods of achievement measurement are well developed, their application in cross-national studies is neither straightforward nor easy. Such studies are among the most challenging that can be undertaken. They should not be undertaken without adequate resources for detailed planning, for data collection in each of the countries, and, especially, for comparative data analysis at the end of the study. It must be recognized that international comparisons are more expensive than simple comparative studies within one country. Given the importance attached to the results of international comparisons today, it is better to forgo a study altogether than to try to proceed with inadequate funding. Long-Term Needs for U.S. Participation in International Studies The board's concerns embrace the mix of international com- parative studies in which the United States participates as well as the merits of particular studies. Generally speaking, comparative studies supported by the United States should address a range of content areas and grade levels and should encompass quan- titative survey research studies as well as more intensive stud- ies that use a range of qualitative research methods. Although most studies may be limited to paper-an~l-penci} measures of educational achievement, there is also a need for some studies that use performance tests. International educational studies appear to be so important that the United States should plan to participate in the prepa- ratory meetings, obtain the necessary commitments from local and state officials, and set aside sufficient resources to ensure
10 INTERNATIONAL COMPARATIVE STUDIES IN EDUCATION that the data gathering and analytic work will meet interna- tionally recognized standards. The United States should also actively support methodological work designed to improve the reliability and validity of international comparisons. If international comparisons are to be technically valid} and useful, issues of reliability and validity must be addressed outside the context of individual projects. There is a real need for more thoughtful, less constrained research on the methodology of international comparisons. The United States should collect some information through a regular cycle of specialized studies. On a regular basis, the academic achievement of U.S. students in different subject matters should be compared with that of students elsewhere. While some data on variables that might have value in helping to understand observed differences in performance should be in- cluded, for reasons of cost and efficiency the major thrust of these studies should be simply to compare academic achieve- ment of U.S. and other students. Such descriptive studies should be conducted frequently enough for policy makers to monitor changes in educational progress, but not so frequently that there is little likelihood of a detectable change. Aspects of educational systems can also be monitored through a system of comparative education indicators. Much of the data for such indicators is already collected regularly by countries, although there may be a serious question about the comparability of indicators. The United States should participate in interna- tional programs, such as one currently being developed in the Organization for Economic Cooperation and Development (OECD), to provide education indicators on a systematic basis, but it must be recognized that the project faces both technical and conceptual difficulties. Studies that explore influences on learning in some clepth- by investigating such factors as details of school management, curricular diversification, classroom interaction patterns, com- munity and parental influences, classroom material resources, or teacher quality should be done from time to time as relevant theoretical models or significant new educational practices are developed. Every effort should be made to coordinate the ad- ministrative mechanisms for these two types of studies so that
FRAMEWORK 11 duplication of effort is kept to a minimum and the opportunity costs for students and school officials is kept low. The United States should also participate in issue-centered studies on particular problems about which other countries have a common interest. These special studies may not require national representative samples and need not occur on a regular basis. Examples of questions addressed by such studies are: Does classroom competition affect ethnic groups differently? What incentive programs are most successful in attracting the best science and mathematics teachers? How many classroom preparations a day are optimal for good teaching? Are outcomes better when employers finance industry-wide mechanisms for vocational training? There win always be a need for international information on issues of this kind. The timing and depth of analysis, however, should be determined by the level of need and the specific resources required for each problem separately. Comparative studies take time away from classroom activity and may be seen as intrusive by school administrators and teachers. Good motivation to participate in the studies on the part of both students and teachers, however, is necessary if the studies are to be done well and the results are to be valid for each country. It is therefore important that every effort be made to develop studies that are useful to teachers and schools officials in improving the performance of schools, as well as useful to policy makers and researchers. Feedback to schools about the results of the studies and their implications for ec3u- cational practice is also important. Timing and Focus of Proposed Studies Data collected over time in time-series or cohort designs can be of significantly greater value than single, cross-sectional studies, especially when data are collected at regular intervals. For that reason, high priority should be given to continued U.S. involvement in studies for which failure to participate would jeopardize valuable trend lines. Conversely, because it is difficult to make substantial alterations in the content or administration procedures used in data collected over time, the United States
12 INTERNATIONAL COMPARATIVE STUDIES IN EDUCATION should strive to ensure that studies intended to initiate a series represent the state of the art in design and instrumentation. The scheduling of data collections embedded in a series is closely constrained. Flexibility in the timing of cross-national studies may also be limited by school calendars around the world and by the logistics of international cooperation. None- theless, the optimum timing of international studies should be considered in decisions to participate in them. Reasons for accelerating or delaying studies might include: · Effects on the participation of nations and of sampled units within nations if too many cross-national studies are carried out simultaneously; · The opportunity to evaluate specific, significant educational policies or investments in the United States or abroad; · The expected impact on the diagnosis of major shortcom- ings of educational systems and on the development of remecliating strategies and policies; · Desirability of timing the release of findings to maximize impact; · Documentation of educational systems or practices soon to be altered or eliminated; · Likelihood that resources available for studies may be di- verted to other purposes if there are undue delays, or con- versely, that additional resources may become available at some future date. Proposers of studies should also consider the potential over- lap of any new study with other recent or ongoing studies. The utility of overlap for calibration of measures, comparison, and cross-valiciation must be weighed against the value of new distinctive data. The distinctiveness of a proposed study might be reflected in several key features: nations represented, academic content area, types of learning outcomes examined, age or grade levels involved, and research methods used. Values to Different Constituencies The primary factor in deciding on U.S. participation in com- parative studies should be the information needs of the United States. in making the decision to participate, however, consid-
FRAME WORK 13 oration should also be given to a proposed study's value to other participants. As part of a global community, the United States cannot take an exclusively national view of any study's utility. The information needs of other nations, especially de- veloping nations, may differ from our own, and the United States may sometimes be called on to join in studies that are of greater value to other countries than to itself. Decision makers at different levels of the educational system have varying needs for information. Teachers and administrators at the school and district levels may seek information about specific instructional practices, while state and federal policy makers are more likely to be concerned with the effects of broach policies and programs. International organizations may wish to compare educational systems or evaluate development initiatives at the level of entire nations. At the national level especially, a study's importance may lie as much in drawing attention to an educational problem and catalyzing action as in providing new knowledge. Other things being equal, preference should be given to cross-national studies that address needs at more than one level. The aims and priorities of each study, however, should be clearly stated at the outset. A proposed stucly's importance to constituencies other than those of the sponsoring agency should also be considered. In the light of the enormous economic importance of a sound educational system, leaders in business and industry may wish to consult comparative educational studies in their international planning. Textbook publishers, developers of educational software, and other educational vendors may use these studies to identify needs and markets for new products. Finally, if international research is to serve the end of scientific knowledge, it must be available to and used by the educational research community. Reporting and dissemination targeted to the needs of different audiences will enhance the value of an international study.
