INTERNATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS

International Organization

Educational Testing Service

Years of Data Collection

International Assessment of Educational Progress - I: 1988

International Assessment of Educational Progress - II: 1991

Purpose

Since 1983 Educational Testing Service has administered the National Assessment of Educational Progress and related projects. NAEP is an ongoing, congressionally mandated project established to conduct national surveys of the educational attainments of students in the United States. Its primary goal is to determine and report the status of and trends over time in educational achievement. NAEP was initiated in 1969 to obtain comprehensive and dependable national educational-achievement data in a uniform, scientific manner.

International Assessment of Educational Progress - I: The purpose of IAEP-I was to collect and report data on what students know and can do, on the educational and cultural factors associated with achievement, and on students' attitudes. IAEP-I assessed mathematics and science achievement of 13-year-old students in the United States and five other countries. The United States joined with the five other countries to explore the feasibility of reducing the time and money requirements for international comparative studies by capitalizing on the National Assessment of Educational Progress materials and procedures. IAEP-I permitted interested countries to experiment with NAEP technologies to determine their appropriateness for local evaluation projects.

International Evaluation of Educational Progress - II: The purpose of IAEP-II was to collect and report data on what students know and can do, on the educational and cultural factors associated with achievement, and on students' attitudes. IAEP-II assessed mathematics and science skills of samples of 9- and 13-year-old students from the United States and 19 other countries, using technology developed for the National Assessment of Educational Progress. By utilizing existing NAEP technology and procedures, time and money required to conduct these international comparative studies was reduced and many interested countries were able to experiment with the innovative psychometric techniques.

Countries that participated are large and small, rich and poor, and have varied ethnic, religious, language, and cultural traditions. Educational goals, expectations, and the meaning of achievement vary among each of these countries. Each of the countries that participated did so for its own reasons: to compare its results with those of neighbors or competitors; to learn about the educational policies and practices of countries whose students seem to regularly achieve success in mathematics and science; to establish a baseline of data within its own country against which progress could be measured in the future.

All participants shared a common interest in identifying what is possible for 9- and 13-year-olds to know and to be able to do in mathematics and science. Knowledge of what is possible produces new enthusiasm, raises sights, establishes new challenges, and ultimately can improve personal and societal performance. The assessment information can be used by each



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 48
International Comparative Studies in Education: Descriptions of Selected Large-Scale Assessments and Case Studies INTERNATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS International Organization Educational Testing Service Years of Data Collection International Assessment of Educational Progress - I: 1988 International Assessment of Educational Progress - II: 1991 Purpose Since 1983 Educational Testing Service has administered the National Assessment of Educational Progress and related projects. NAEP is an ongoing, congressionally mandated project established to conduct national surveys of the educational attainments of students in the United States. Its primary goal is to determine and report the status of and trends over time in educational achievement. NAEP was initiated in 1969 to obtain comprehensive and dependable national educational-achievement data in a uniform, scientific manner. International Assessment of Educational Progress - I: The purpose of IAEP-I was to collect and report data on what students know and can do, on the educational and cultural factors associated with achievement, and on students' attitudes. IAEP-I assessed mathematics and science achievement of 13-year-old students in the United States and five other countries. The United States joined with the five other countries to explore the feasibility of reducing the time and money requirements for international comparative studies by capitalizing on the National Assessment of Educational Progress materials and procedures. IAEP-I permitted interested countries to experiment with NAEP technologies to determine their appropriateness for local evaluation projects. International Evaluation of Educational Progress - II: The purpose of IAEP-II was to collect and report data on what students know and can do, on the educational and cultural factors associated with achievement, and on students' attitudes. IAEP-II assessed mathematics and science skills of samples of 9- and 13-year-old students from the United States and 19 other countries, using technology developed for the National Assessment of Educational Progress. By utilizing existing NAEP technology and procedures, time and money required to conduct these international comparative studies was reduced and many interested countries were able to experiment with the innovative psychometric techniques. Countries that participated are large and small, rich and poor, and have varied ethnic, religious, language, and cultural traditions. Educational goals, expectations, and the meaning of achievement vary among each of these countries. Each of the countries that participated did so for its own reasons: to compare its results with those of neighbors or competitors; to learn about the educational policies and practices of countries whose students seem to regularly achieve success in mathematics and science; to establish a baseline of data within its own country against which progress could be measured in the future. All participants shared a common interest in identifying what is possible for 9- and 13-year-olds to know and to be able to do in mathematics and science. Knowledge of what is possible produces new enthusiasm, raises sights, establishes new challenges, and ultimately can improve personal and societal performance. The assessment information can be used by each

OCR for page 48
International Comparative Studies in Education: Descriptions of Selected Large-Scale Assessments and Case Studies country to set goals and develop curriculums and practices in harmony with its own values and culture. An optional geography component was included in this project, as a first step toward acquiring international measures of geography skills, knowledge, and education. Through this optional geography component, participants hoped to identify behaviors and practices that contribute to high levels of achievement, so that education policy makers would have information that might enable them to improve domestic performance in concert with national educational goals. Organization and Management International Assessment of Educational Progress - I: A project director, located at the Educational Testing Service, worked in collaboration with the National Foundation for Educational Research (UK); Ministry of Education and Science (Spain); New Brunswick, Ontario, and Quebec Ministries of Education (Canada); Laval University data analysts; and Educational Testing Service consultants. International Assessment of Educational Progress - II: International coordination was done by staff of the Canadian Data Analysis Group, Educational Testing Service, and Westat. In all countries except Brazil and Mozambique an independent, trained observer interviewed the country project manager about all aspects of the project and visited one or more test administration sites. In most cases, the observer was fluent in the language of the assessment. Decisions concerning the design and implementation of the project were made collaboratively by the representatives of the provinces and countries involved in the survey. The National Academy of Sciences/National Research Council's Board on International Comparative Studies in Education reviewed plans for the project at several stages of its development and made suggestions to improve the technical quality of the study. Design Participants International Assessment of Educational Progress - I: Five countries and four Canadian provinces with extensive experience in large-scale assessment participated. (Canada does not have a federal system of education.) Twelve student populations were included from: British Columbia, Ireland, Korea, New Brunswick (English), New Brunswick (French), Ontario (English), Ontario (French), Quebec (English), Quebec (French), Spain, United Kingdom (sample drawn from students in England, Scotland, and Wales), and United States. Students were 13 years old (born January 1, 1974-December 31, 1974), and were selected from public and private elementary, middle, and secondary schools. International Assessment of Educational Progress - II: Twenty countries assessed the achievement of 13-year-old students and 14 assessed 9-year-old students in mathematics and science. In some cases, participants assessed virtually all age-eligible children in their countries; in other cases they confined samples to certain geographic regions, language groups, or grade levels. In some countries significant proportions of age-eligible children were not represented because they did not attend school. Low rates of school or student participation in some countries mean results may be biased.

OCR for page 48
International Comparative Studies in Education: Descriptions of Selected Large-Scale Assessments and Case Studies Country participation is as follows: Brazil cities of São Paulo and Fortaleza, restricted grades, in-school population Canada four provinces at age 9 and nine out of 10 provinces at age 13 China 20 out of 29 provinces and independent cities, restricted grades, in-school population England all students, low participation at ages 9 and 13 France all students Hungary all students Ireland all students Israel Hebrew-speaking schools Italy province of Emilia-Romagna, low participation at age 9 Jordan all students Korea all students Mozambique cities of Maputo and Beira, in-school population, low participation (Mozambique did not assess Science.) Portugal restricted grades, in-school population at age 13 Scotland all students, low participation at age 9 Slovenia all students Soviet Union 14 out of 15 republics, Russian-speaking schools Spain all regions except Cataluña, Spanish-speaking schools Switzerland 15 out of 26 cantons Taiwan all students United States all students The geography portion of the IAEP was a special probe into the geographic knowledge and skills of 13-year-olds. Twenty countries assessed the mathematics and science achievement of 13-year-olds; of these 20 countries, nine countries also administered the optional geography component: Canada eight out of 10 provinces Hungary all students Ireland all students Korea all students Scotland all students Slovenia all students Soviet Union 14 out of 15 republics, Russian-language schools Spain all regions except Cataluña, Spanish-language schools United States all students Sample International Assessment of Educational Progress - I: A random sample of about 2,000 students from 100 different schools was selected from each population. In the United States, the sample size was about 1,000 students in 200 schools. A total of 24,000 students was assessed. International Assessment of Educational Progress - II: Typically, a random sample of 3,300 students from about 110 different schools was selected from each population at each age level; half were assessed in mathematics and half in science. A total of about 175,000 9- and 13-year-olds (born in calendar years 1981 and 1977) were tested in 13 different languages in March 1991. Some countries drew samples from virtually all children in the appropriate age group; others assessed only children in specific geographic areas, language groups, or grade

OCR for page 48
International Comparative Studies in Education: Descriptions of Selected Large-Scale Assessments and Case Studies levels. The definition of populations often followed the structure of school systems, political divisions, and cultural distinctions. Procedures and Summary of Content International Assessment of Educational Progress - I: All countries and provinces followed standardized administration procedures and administered the assessments during February 1988. International Assessment of Educational Progress - II: Participants recognized fundamental differences from country to country, but assembled tests that focus on common curriculum elements. In order to form contexts for interpreting student achievement data, they added questions about students' home background and classroom experiences and the characteristics of the schools they attend. As a survey research project, this assessment could not establish cause-and-effect relationships, but it could provide clues that might help to explain high and low performance. The IAEP-II assessment used a four-part survey: a main assessment of 13-year-olds' performance in mathematics and science; an assessment of 9-year-olds' performance in mathematics and science; an experimental, performance-based assessment of 13-year-olds' ability to use equipment and materials to solve mathematics and science problems; and a short probe of the geography skills and knowledge of 13-year-olds. All countries participated in the main assessment of 13-year-olds; participation in the other assessment components was optional. Each participating country was responsible for carrying out all aspects of the project, including sampling, survey administration, quality control, and data entry using standardized procedures that were developed for the project. Several training manuals were developed for the project. Several international training sessions were held, during which each step of the assessment process was explained. The assessment was developed through a consensus-building process that involved curriculum and measurement experts from each of the participating countries and provinces. Several existing NAEP frameworks and the IAEP-I framework were reviewed by participants and evaluated as to their appropriateness for their own countries' curriculums. Together, the participants then adapted the NAEP frameworks to reflect an international consensus of subject-specific topics and cognitive processes that they believed reasonably reflected curriculums being implemented in their own school systems. The nine countries that participated in the geography component participated in the development of the geography framework that guided the design of the instruments. Curricular experts in each country reviewed the appropriateness of all potential questions for their own students. Once the participants had agreed upon common frameworks and the relative emphases that would be placed on each topic and cognitive process category of the assessment, more than one-half submitted test items from their countries' own assessment programs that they felt were appropriate and met the requirements of the IAEP assessment. Many questions from the United States' NAEP assessments were included as well. The items with the highest ratings across all countries were placed into a pool of acceptable questions from which a subset was selected and pilot-tested in all of the participating provinces and countries.

OCR for page 48
International Comparative Studies in Education: Descriptions of Selected Large-Scale Assessments and Case Studies While acceptable to all, the content areas measured in the tests were not equally representative of each country's curriculum and the multiple-choice question testing format was not equally familiar to students from all countries. Countries differ in the age at which students start school and in policies for promotion, students at age 13 were further along in their schooling in some countries than in others. Data Collection and Analyses International Assessment of Educational Progress - I: Students were administered a 45-minute mathematics assessment (63 questions) and a 45-minute science assessment (60 questions), selected from the total pool of 281 mathematics and 188 science questions used in the 1986 Unites States' National Assessment of Educational Progress. In addition, students answered questions about their school experiences and attitudes, and teachers rated students' exposure to the concepts tested by the items. The assessment was administered in February 1988, except in the United States, where the data were collected during the January-through-middle-March NAEP assessment. Each country and province was responsible for developing a data file following a standard format, for checking ranges of responses, and resolving inconsistencies in the data. Quebec Ministry of Education staff also checked the files. ETS staff calculated weights for the United States and Canadian participants and verified weights for other participants. A Laval University (Quebec) research team conducted data analysis in consultation with ETS researchers and data analysts. The first stage of analysis involved calculation of the percentage of correct answers and standard errors for individual questions and groups of questions. The second stage of analysis involved scaling of mathematics and science results using item response theory technology. The assessment collected opportunity-to-learn information to determine to what extent students in the participating populations had been exposed to various mathematics and science content areas covered in the IAEP questions. Educational Testing Service measurement specialists worked with colleagues from five other countries to translate and adapt techniques used in the United States for the National Assessment of Educational Progress. Achievement results that permit comparisons and present valid and reliable findings were presented in A World of Differences: An International Assessment of Mathematics and Science. Each participating country had the opportunity to experiment with new measurement practices. Cost-effectiveness of sampling techniques, the power and limitations of Item Response Theory, and the usefulness of new reporting techniques were demonstrated. Experts in each participating country had hands-on experiences with problems and potential of new assessment techniques. International Assessment of Educational Progress - II: The achievement tests lasted one hour. The tests given to 9-year-olds included 62 questions in mathematics and 60 questions in science. The tests given to 13-year-olds included 76 questions in mathematics and 72 questions in science. In addition, students at each age spent about 10 minutes responding to questions about their backgrounds and home and school experiences. Finally, students in countries assessing geography spent an additional 10 minutes responding to questions on geography. School administrators completed a school questionnaire. National Center for Education Statistics commissioner Emerson Elliott emphasized factors to be kept in mind in interpreting the results of this assessment: 1) the importance of cultural

OCR for page 48
International Comparative Studies in Education: Descriptions of Selected Large-Scale Assessments and Case Studies differences and of each country's educational traditions and practices and 2) the consensus process that produced the mathematics and science frameworks used in the development of IAEP II reflects only the curricular elements common to all participating countries. Data analysis carried out between September and December 1991 involved checking dimensionality of the mathematics and science items at each age level through factor analysis techniques and item calibration on each of the dimensions established; computation of item and average percents correct; ability scoring using the plausible-values technology developed for the NAEP study; scale anchoring; and linkage of the two age groups. Assessment results were obtained for 52 populations: Population Language 9-year-olds mathematics 9-year-olds science 13-year-olds mathematics 13-year-olds science Brazil Portuguese -- -- X X Canada           Alberta English -- -- X X Br.Col. English X X X X Manitoba English -- -- X X Manitoba French -- -- X X NewBr. English X X X X New Br. French -- -- X X Newfound. English -- -- X X Nova Sc. English -- -- X X Ontario English X X X X Ontario French X X X X Quebec English X X X X Quebec French X X X X Saskatch. English -- -- X X Saskatch. French -- -- X X China Mandarin -- -- X X England English X X X X France French -- -- X X Hungary Hungarian X X X X Ireland English X X X X Israel Hebrew X X X X Italy Italian X X X X Jordan Arabic -- -- X X Korea Korean X X X X Mozambique Portuguese -- -- -- X Portugal Portuguese X X X X Scotland English X X X X Slovenia Slovene X X X X Soviet Un. Russian X X X X Spain Spanish X X X X Switzerland French/German/Italian -- -- X X Taiwan Mandarin X X X X United St. English X X X X

OCR for page 48
International Comparative Studies in Education: Descriptions of Selected Large-Scale Assessments and Case Studies Population 13-year-olds geography option 13-year-olds performance option Canada     Alberta X X British Columbia X X Manitoba X -- New Brunswick X -- Newfoundland X -- Nova Scotia -- X Ontario X X Quebec X -- Saskatchewan X X England X X Hungary X -- Ireland X -- Korea X -- Scotland X X Slovenia X X Soviet Union X X Spain X -- Taiwan -- X United States X -- Analysis of the second International Assessment of Educational Progress using item response theory technology was completed and almanacs of results were produced in 1992. A second volume of the IAEP Technical Report, which describes the steps in applying item response theory methodology, was also produced in 1992. This technical report describes the creation of a reference population that was used for all subsequent analyses. After ensuring that the reference population adequately represented each of the participating populations and met assumptions of unidimensionality, these data were used to calibrate item perimeters. Proficiency scores were then estimated, making use of plausible values technology. Each age group was scaled separately and then linked using the Stocking-Lord procedure. The resulting scale was assigned a mean of 500 and a standard deviation of 100. Average proficiency scores were then calculated for participating populations, following the same procedures used for the percent correct analyses. Secondary analysis of IAEP results. The Educational Testing Service Center for the Assessment of Educational Progress prepared two short reports that focus on implications of the IAEP results for U.S. efforts at standard setting and curriculum reform. These reports, written in collaboration with U.S. mathematics and science educators, make use of item-by-item results and comment on what students in the United States know and can do well and where U.S. students fare behind student peers in other countries. They will be published by the National Science Foundation. Timetable International Assessment of Educational Progress - I:

OCR for page 48
International Comparative Studies in Education: Descriptions of Selected Large-Scale Assessments and Case Studies 1987-1989 Project implementation was carried out through a series of meetings to select assessment items, review pilot-test results, and review and interpret final results. Decisions were made collaboratively; follow-up coordination was provided by ETS staff. International Assessment of Educational Progress - II: 1989 Pilot testing was conducted November 1989-February 1990. 1990 The pilot review meeting was held in Montreal. (May) Data was collected in the southern hemisphere. (September) 1991 Data was collected in the northern hemisphere. (March) 1992 Reports were published on results of IAEP geography, mathematics and science. An almanac of results of IAEP using item response theory published. Two volumes of the IAEP Technical Report were published. Publications A World of Differences: An International Assessment of Mathematics and Science. Report of the International Assessment of Educational Progress. Educational Testing Service. (1989) A World of Differences: Technical Report. Report of the International Assessment of Educational Progress. Educational Testing Service. (1989) Learning Mathematics. Report of The International Assessment of Educational Progress. Educational Testing Service. (1992) Learning Science. Report of The International Assessment of Educational Progress. Educational Testing Service. (1992) Learning Geography. Report of the International Assessment of Educational Progress. Educational Testing Service. (1992) Performance Assessment: An International Experiment. Report of the International Assessment of Educational Progress. Educational Testing Service. (1992) IAEP Technical Report. Report of the International Assessment of Educational Progress. Educational Testing Service. (1992) IAEP Technical Report: Volume Two. Report of the International Assessment of Educational Progress. Educational Testing Service. (1992) Two short reports on implications of IAEP results for U.S. efforts at standard setting and curriculum reform will be published by the National Science Foundation. Funding International Assessment of Educational Progress - I: Funding was provided by the U.S. Department of Education and the National Science Foundation for overall coordination,

OCR for page 48
International Comparative Studies in Education: Descriptions of Selected Large-Scale Assessments and Case Studies sampling, U.S. data analysis, and reporting. Participating countries and provinces acquired support for local data collection and coordination. International Assessment of Educational Progress - II: Funding was provided by The National Science Foundation and the U.S. Department of Education's National Center for Education Statistics provided a grant for the expenses of overall coordination, sampling, data analysis, and reporting. The Carnegie Corporation provided additional funds to cover the travel expenses of some of the participants who could not meet the financial burdens of traveling to the project 's coordination and training meetings, held in Canada, England, France, Hong Kong, and the United States. Participating countries acquired support for local data collection and coordination. Information Sources Nancy A. Mead, International Coordinator/Project Director Center for the Assessment of Educational Progress Educational Testing Service P.O. Box 6710 Princeton, New Jersey 08541-6710 telephone: 609/ 734-1526 e-mail: nmead@rosedale.org Educan Inc. 1991 IAEP Notes. 4:September. Educan Inc., Longueuil, Quebec, Canada. Educational Testing Service 1991 The 1991 IAEP Assessment: Objectives for Mathematics, Science, and Geography. Educational Testing Service, Princeton, New Jersey. 1992 IAEP Technical Report. Report of the International Assessment of Educational Progress. ETS, Princeton, New Jersey. 1992 IAEP Technical Report: Volume Two. Report of the International Assessment of Educational Progress. ETS, Princeton, New Jersey. Lapointe, Archie E. 1988 IAEP preliminary findings; comparison with IEA findings. Presentation to the Board on International Comparative Studies in Education. November 4. 1991 IAEP analysis and report plans. Presentation to the Board on International Comparative Studies in Education. January 28. 1991 IAEP report draft. Presentation to the Board on International Comparative Studies in Education. September 23. Lapointe, Archie E., Nancy A. Mead, and Eugene Johnson 1990 IAEP status. Presentation to the Board on International Comparative Studies in Education. February 2. Lapointe, Archie E., Nancy A. Mead, and Janice M. Askew 1992 Learning Mathematics. Report of the International Assessment of Educational Progress. Report No. 22-CAEP-01. February. Educational Testing Service, Princeton, New Jersey. Lapointe, Archie E., Janice M. Askew, and Nancy A. Mead 1992 Learning Science. Report of the International Assessment of Educational Progress. Report No. 22-CAEP-02. February. Educational Testing Service, Princeton, New Jersey.

OCR for page 48
International Comparative Studies in Education: Descriptions of Selected Large-Scale Assessments and Case Studies Lapointe, Archie E., Nancy A. Mead, and Gary W. Phillips 1989 A World of Differences: An International Assessment of Mathematics and Science. Report of the First International Assessment of Educational Progress. Report No. 19-CAEP-01. January. Educational Testing Service, Princeton, New Jersey. Lazer, Stephen 1992 Learning About the World. Report of The International Assessment of Educational Progress. Report No. 22-CAEP-05. June. Educational Testing Service, Princeton, New Jersey. Mead, Nancy A. 1990 Second International Assessment of Educational Progress. Status report prepared for the Board on International Comparative Studies in Education. April 25. 1992 Second International Assessment of Educational Progress. Status report prepared for the Board on International Comparative Studies in Education. September. 1993 Secondary Analysis of IAEP Results. Status report prepared for the Board on International Comparative Studies in Education. January. Semple, Brian McLean 1992 Performance Assessment: An International Experiment. Report of the International Assessment of Educational Achievement. Report No. 22-CAEP-06. July. Educational Testing Service, Princeton, New Jersey. ****** NOTE: This study summary was reviewed and edited by Nancy Mead at the Educational Testing Service on June 10, 1994.