INTERNATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS
International Organization |
Educational Testing Service |
Years of Data Collection
International Assessment of Educational Progress - I: 1988
International Assessment of Educational Progress - II: 1991
Purpose
Since 1983 Educational Testing Service has administered the National Assessment of Educational Progress and related projects. NAEP is an ongoing, congressionally mandated project established to conduct national surveys of the educational attainments of students in the United States. Its primary goal is to determine and report the status of and trends over time in educational achievement. NAEP was initiated in 1969 to obtain comprehensive and dependable national educational-achievement data in a uniform, scientific manner.
International Assessment of Educational Progress - I: The purpose of IAEP-I was to collect and report data on what students know and can do, on the educational and cultural factors associated with achievement, and on students' attitudes. IAEP-I assessed mathematics and science achievement of 13-year-old students in the United States and five other countries. The United States joined with the five other countries to explore the feasibility of reducing the time and money requirements for international comparative studies by capitalizing on the National Assessment of Educational Progress materials and procedures. IAEP-I permitted interested countries to experiment with NAEP technologies to determine their appropriateness for local evaluation projects.
International Evaluation of Educational Progress - II: The purpose of IAEP-II was to collect and report data on what students know and can do, on the educational and cultural factors associated with achievement, and on students' attitudes. IAEP-II assessed mathematics and science skills of samples of 9- and 13-year-old students from the United States and 19 other countries, using technology developed for the National Assessment of Educational Progress. By utilizing existing NAEP technology and procedures, time and money required to conduct these international comparative studies was reduced and many interested countries were able to experiment with the innovative psychometric techniques.
Countries that participated are large and small, rich and poor, and have varied ethnic, religious, language, and cultural traditions. Educational goals, expectations, and the meaning of achievement vary among each of these countries. Each of the countries that participated did so for its own reasons: to compare its results with those of neighbors or competitors; to learn about the educational policies and practices of countries whose students seem to regularly achieve success in mathematics and science; to establish a baseline of data within its own country against which progress could be measured in the future.
All participants shared a common interest in identifying what is possible for 9- and 13-year-olds to know and to be able to do in mathematics and science. Knowledge of what is possible produces new enthusiasm, raises sights, establishes new challenges, and ultimately can improve personal and societal performance. The assessment information can be used by each
country to set goals and develop curriculums and practices in harmony with its own values and culture.
An optional geography component was included in this project, as a first step toward acquiring international measures of geography skills, knowledge, and education. Through this optional geography component, participants hoped to identify behaviors and practices that contribute to high levels of achievement, so that education policy makers would have information that might enable them to improve domestic performance in concert with national educational goals.
Organization and Management
International Assessment of Educational Progress - I: A project director, located at the Educational Testing Service, worked in collaboration with the National Foundation for Educational Research (UK); Ministry of Education and Science (Spain); New Brunswick, Ontario, and Quebec Ministries of Education (Canada); Laval University data analysts; and Educational Testing Service consultants.
International Assessment of Educational Progress - II: International coordination was done by staff of the Canadian Data Analysis Group, Educational Testing Service, and Westat. In all countries except Brazil and Mozambique an independent, trained observer interviewed the country project manager about all aspects of the project and visited one or more test administration sites. In most cases, the observer was fluent in the language of the assessment.
Decisions concerning the design and implementation of the project were made collaboratively by the representatives of the provinces and countries involved in the survey. The National Academy of Sciences/National Research Council's Board on International Comparative Studies in Education reviewed plans for the project at several stages of its development and made suggestions to improve the technical quality of the study.
Design
Participants
International Assessment of Educational Progress - I: Five countries and four Canadian provinces with extensive experience in large-scale assessment participated. (Canada does not have a federal system of education.) Twelve student populations were included from: British Columbia, Ireland, Korea, New Brunswick (English), New Brunswick (French), Ontario (English), Ontario (French), Quebec (English), Quebec (French), Spain, United Kingdom (sample drawn from students in England, Scotland, and Wales), and United States. Students were 13 years old (born January 1, 1974-December 31, 1974), and were selected from public and private elementary, middle, and secondary schools.
International Assessment of Educational Progress - II: Twenty countries assessed the achievement of 13-year-old students and 14 assessed 9-year-old students in mathematics and science. In some cases, participants assessed virtually all age-eligible children in their countries; in other cases they confined samples to certain geographic regions, language groups, or grade levels. In some countries significant proportions of age-eligible children were not represented because they did not attend school. Low rates of school or student participation in some countries mean results may be biased.
Country participation is as follows:
Brazil |
cities of São Paulo and Fortaleza, restricted grades, in-school population |
Canada |
four provinces at age 9 and nine out of 10 provinces at age 13 |
China |
20 out of 29 provinces and independent cities, restricted grades, in-school population |
England |
all students, low participation at ages 9 and 13 |
France |
all students |
Hungary |
all students |
Ireland |
all students |
Israel |
Hebrew-speaking schools |
Italy |
province of Emilia-Romagna, low participation at age 9 |
Jordan |
all students |
Korea |
all students |
Mozambique |
cities of Maputo and Beira, in-school population, low participation (Mozambique did not assess Science.) |
Portugal |
restricted grades, in-school population at age 13 |
Scotland |
all students, low participation at age 9 |
Slovenia |
all students |
Soviet Union |
14 out of 15 republics, Russian-speaking schools |
Spain |
all regions except Cataluña, Spanish-speaking schools |
Switzerland |
15 out of 26 cantons |
Taiwan |
all students |
United States |
all students |
The geography portion of the IAEP was a special probe into the geographic knowledge and skills of 13-year-olds. Twenty countries assessed the mathematics and science achievement of 13-year-olds; of these 20 countries, nine countries also administered the optional geography component:
Canada |
eight out of 10 provinces |
Hungary |
all students |
Ireland |
all students |
Korea |
all students |
Scotland |
all students |
Slovenia |
all students |
Soviet Union |
14 out of 15 republics, Russian-language schools |
Spain |
all regions except Cataluña, Spanish-language schools |
United States |
all students |
Sample
International Assessment of Educational Progress - I: A random sample of about 2,000 students from 100 different schools was selected from each population. In the United States, the sample size was about 1,000 students in 200 schools. A total of 24,000 students was assessed.
International Assessment of Educational Progress - II: Typically, a random sample of 3,300 students from about 110 different schools was selected from each population at each age level; half were assessed in mathematics and half in science. A total of about 175,000 9- and 13-year-olds (born in calendar years 1981 and 1977) were tested in 13 different languages in March 1991. Some countries drew samples from virtually all children in the appropriate age group; others assessed only children in specific geographic areas, language groups, or grade
levels. The definition of populations often followed the structure of school systems, political divisions, and cultural distinctions.
Procedures and Summary of Content
International Assessment of Educational Progress - I: All countries and provinces followed standardized administration procedures and administered the assessments during February 1988.
International Assessment of Educational Progress - II: Participants recognized fundamental differences from country to country, but assembled tests that focus on common curriculum elements. In order to form contexts for interpreting student achievement data, they added questions about students' home background and classroom experiences and the characteristics of the schools they attend. As a survey research project, this assessment could not establish cause-and-effect relationships, but it could provide clues that might help to explain high and low performance.
The IAEP-II assessment used a four-part survey: a main assessment of 13-year-olds' performance in mathematics and science; an assessment of 9-year-olds' performance in mathematics and science; an experimental, performance-based assessment of 13-year-olds' ability to use equipment and materials to solve mathematics and science problems; and a short probe of the geography skills and knowledge of 13-year-olds. All countries participated in the main assessment of 13-year-olds; participation in the other assessment components was optional.
Each participating country was responsible for carrying out all aspects of the project, including sampling, survey administration, quality control, and data entry using standardized procedures that were developed for the project. Several training manuals were developed for the project. Several international training sessions were held, during which each step of the assessment process was explained.
The assessment was developed through a consensus-building process that involved curriculum and measurement experts from each of the participating countries and provinces. Several existing NAEP frameworks and the IAEP-I framework were reviewed by participants and evaluated as to their appropriateness for their own countries' curriculums. Together, the participants then adapted the NAEP frameworks to reflect an international consensus of subject-specific topics and cognitive processes that they believed reasonably reflected curriculums being implemented in their own school systems.
The nine countries that participated in the geography component participated in the development of the geography framework that guided the design of the instruments. Curricular experts in each country reviewed the appropriateness of all potential questions for their own students.
Once the participants had agreed upon common frameworks and the relative emphases that would be placed on each topic and cognitive process category of the assessment, more than one-half submitted test items from their countries' own assessment programs that they felt were appropriate and met the requirements of the IAEP assessment. Many questions from the United States' NAEP assessments were included as well. The items with the highest ratings across all countries were placed into a pool of acceptable questions from which a subset was selected and pilot-tested in all of the participating provinces and countries.
While acceptable to all, the content areas measured in the tests were not equally representative of each country's curriculum and the multiple-choice question testing format was not equally familiar to students from all countries. Countries differ in the age at which students start school and in policies for promotion, students at age 13 were further along in their schooling in some countries than in others.
Data Collection and Analyses
International Assessment of Educational Progress - I: Students were administered a 45-minute mathematics assessment (63 questions) and a 45-minute science assessment (60 questions), selected from the total pool of 281 mathematics and 188 science questions used in the 1986 Unites States' National Assessment of Educational Progress. In addition, students answered questions about their school experiences and attitudes, and teachers rated students' exposure to the concepts tested by the items. The assessment was administered in February 1988, except in the United States, where the data were collected during the January-through-middle-March NAEP assessment.
Each country and province was responsible for developing a data file following a standard format, for checking ranges of responses, and resolving inconsistencies in the data. Quebec Ministry of Education staff also checked the files. ETS staff calculated weights for the United States and Canadian participants and verified weights for other participants. A Laval University (Quebec) research team conducted data analysis in consultation with ETS researchers and data analysts.
The first stage of analysis involved calculation of the percentage of correct answers and standard errors for individual questions and groups of questions. The second stage of analysis involved scaling of mathematics and science results using item response theory technology. The assessment collected opportunity-to-learn information to determine to what extent students in the participating populations had been exposed to various mathematics and science content areas covered in the IAEP questions.
Educational Testing Service measurement specialists worked with colleagues from five other countries to translate and adapt techniques used in the United States for the National Assessment of Educational Progress. Achievement results that permit comparisons and present valid and reliable findings were presented in A World of Differences: An International Assessment of Mathematics and Science. Each participating country had the opportunity to experiment with new measurement practices. Cost-effectiveness of sampling techniques, the power and limitations of Item Response Theory, and the usefulness of new reporting techniques were demonstrated. Experts in each participating country had hands-on experiences with problems and potential of new assessment techniques.
International Assessment of Educational Progress - II: The achievement tests lasted one hour. The tests given to 9-year-olds included 62 questions in mathematics and 60 questions in science. The tests given to 13-year-olds included 76 questions in mathematics and 72 questions in science. In addition, students at each age spent about 10 minutes responding to questions about their backgrounds and home and school experiences. Finally, students in countries assessing geography spent an additional 10 minutes responding to questions on geography. School administrators completed a school questionnaire.
National Center for Education Statistics commissioner Emerson Elliott emphasized factors to be kept in mind in interpreting the results of this assessment: 1) the importance of cultural
differences and of each country's educational traditions and practices and 2) the consensus process that produced the mathematics and science frameworks used in the development of IAEP II reflects only the curricular elements common to all participating countries.
Data analysis carried out between September and December 1991 involved checking dimensionality of the mathematics and science items at each age level through factor analysis techniques and item calibration on each of the dimensions established; computation of item and average percents correct; ability scoring using the plausible-values technology developed for the NAEP study; scale anchoring; and linkage of the two age groups.
Assessment results were obtained for 52 populations:
Population |
Language |
9-year-olds mathematics |
9-year-olds science |
13-year-olds mathematics |
13-year-olds science |
Brazil |
Portuguese |
-- |
-- |
X |
X |
Canada |
|||||
Alberta |
English |
-- |
-- |
X |
X |
Br.Col. |
English |
X |
X |
X |
X |
Manitoba |
English |
-- |
-- |
X |
X |
Manitoba |
French |
-- |
-- |
X |
X |
NewBr. |
English |
X |
X |
X |
X |
New Br. |
French |
-- |
-- |
X |
X |
Newfound. |
English |
-- |
-- |
X |
X |
Nova Sc. |
English |
-- |
-- |
X |
X |
Ontario |
English |
X |
X |
X |
X |
Ontario |
French |
X |
X |
X |
X |
Quebec |
English |
X |
X |
X |
X |
Quebec |
French |
X |
X |
X |
X |
Saskatch. |
English |
-- |
-- |
X |
X |
Saskatch. |
French |
-- |
-- |
X |
X |
China |
Mandarin |
-- |
-- |
X |
X |
England |
English |
X |
X |
X |
X |
France |
French |
-- |
-- |
X |
X |
Hungary |
Hungarian |
X |
X |
X |
X |
Ireland |
English |
X |
X |
X |
X |
Israel |
Hebrew |
X |
X |
X |
X |
Italy |
Italian |
X |
X |
X |
X |
Jordan |
Arabic |
-- |
-- |
X |
X |
Korea |
Korean |
X |
X |
X |
X |
Mozambique |
Portuguese |
-- |
-- |
-- |
X |
Portugal |
Portuguese |
X |
X |
X |
X |
Scotland |
English |
X |
X |
X |
X |
Slovenia |
Slovene |
X |
X |
X |
X |
Soviet Un. |
Russian |
X |
X |
X |
X |
Spain |
Spanish |
X |
X |
X |
X |
Switzerland |
French/German/Italian |
-- |
-- |
X |
X |
Taiwan |
Mandarin |
X |
X |
X |
X |
United St. |
English |
X |
X |
X |
X |
Population |
13-year-olds geography option |
13-year-olds performance option |
Canada |
||
Alberta |
X |
X |
British Columbia |
X |
X |
Manitoba |
X |
-- |
New Brunswick |
X |
-- |
Newfoundland |
X |
-- |
Nova Scotia |
-- |
X |
Ontario |
X |
X |
Quebec |
X |
-- |
Saskatchewan |
X |
X |
England |
X |
X |
Hungary |
X |
-- |
Ireland |
X |
-- |
Korea |
X |
-- |
Scotland |
X |
X |
Slovenia |
X |
X |
Soviet Union |
X |
X |
Spain |
X |
-- |
Taiwan |
-- |
X |
United States |
X |
-- |
Analysis of the second International Assessment of Educational Progress using item response theory technology was completed and almanacs of results were produced in 1992. A second volume of the IAEP Technical Report, which describes the steps in applying item response theory methodology, was also produced in 1992. This technical report describes the creation of a reference population that was used for all subsequent analyses. After ensuring that the reference population adequately represented each of the participating populations and met assumptions of unidimensionality, these data were used to calibrate item perimeters. Proficiency scores were then estimated, making use of plausible values technology. Each age group was scaled separately and then linked using the Stocking-Lord procedure. The resulting scale was assigned a mean of 500 and a standard deviation of 100. Average proficiency scores were then calculated for participating populations, following the same procedures used for the percent correct analyses.
Secondary analysis of IAEP results. The Educational Testing Service Center for the Assessment of Educational Progress prepared two short reports that focus on implications of the IAEP results for U.S. efforts at standard setting and curriculum reform. These reports, written in collaboration with U.S. mathematics and science educators, make use of item-by-item results and comment on what students in the United States know and can do well and where U.S. students fare behind student peers in other countries. They will be published by the National Science Foundation.
Timetable
International Assessment of Educational Progress - I:
1987-1989
Project implementation was carried out through a series of meetings to select assessment items, review pilot-test results, and review and interpret final results. Decisions were made collaboratively; follow-up coordination was provided by ETS staff.
International Assessment of Educational Progress - II:
1989 |
Pilot testing was conducted November 1989-February 1990. |
1990 |
The pilot review meeting was held in Montreal. (May) Data was collected in the southern hemisphere. (September) |
1991 |
Data was collected in the northern hemisphere. (March) |
1992 |
Reports were published on results of IAEP geography, mathematics and science. An almanac of results of IAEP using item response theory published. Two volumes of the IAEP Technical Report were published. |
Publications
A World of Differences: An International Assessment of Mathematics and Science. Report of the International Assessment of Educational Progress. Educational Testing Service. (1989)
A World of Differences: Technical Report. Report of the International Assessment of Educational Progress. Educational Testing Service. (1989)
Learning Mathematics. Report of The International Assessment of Educational Progress. Educational Testing Service. (1992)
Learning Science. Report of The International Assessment of Educational Progress. Educational Testing Service. (1992)
Learning Geography. Report of the International Assessment of Educational Progress. Educational Testing Service. (1992)
Performance Assessment: An International Experiment. Report of the International Assessment of Educational Progress. Educational Testing Service. (1992)
IAEP Technical Report. Report of the International Assessment of Educational Progress. Educational Testing Service. (1992)
IAEP Technical Report: Volume Two. Report of the International Assessment of Educational Progress. Educational Testing Service. (1992)
Two short reports on implications of IAEP results for U.S. efforts at standard setting and curriculum reform will be published by the National Science Foundation.
Funding
International Assessment of Educational Progress - I: Funding was provided by the U.S. Department of Education and the National Science Foundation for overall coordination,
sampling, U.S. data analysis, and reporting. Participating countries and provinces acquired support for local data collection and coordination.
International Assessment of Educational Progress - II: Funding was provided by The National Science Foundation and the U.S. Department of Education's National Center for Education Statistics provided a grant for the expenses of overall coordination, sampling, data analysis, and reporting. The Carnegie Corporation provided additional funds to cover the travel expenses of some of the participants who could not meet the financial burdens of traveling to the project 's coordination and training meetings, held in Canada, England, France, Hong Kong, and the United States. Participating countries acquired support for local data collection and coordination.
Information Sources
Nancy A. Mead, International Coordinator/Project Director
Center for the Assessment of Educational Progress
Educational Testing Service
P.O. Box 6710
Princeton, New Jersey 08541-6710
telephone: 609/ 734-1526
e-mail: nmead@rosedale.org
Educan Inc.
1991 IAEP Notes. 4:September. Educan Inc., Longueuil, Quebec, Canada.
Educational Testing Service
1991 The 1991 IAEP Assessment: Objectives for Mathematics, Science, and Geography. Educational Testing Service, Princeton, New Jersey.
1992 IAEP Technical Report. Report of the International Assessment of Educational Progress. ETS, Princeton, New Jersey.
1992 IAEP Technical Report: Volume Two. Report of the International Assessment of Educational Progress. ETS, Princeton, New Jersey.
Lapointe, Archie E.
1988 IAEP preliminary findings; comparison with IEA findings. Presentation to the Board on International Comparative Studies in Education. November 4.
1991 IAEP analysis and report plans. Presentation to the Board on International Comparative Studies in Education. January 28.
1991 IAEP report draft. Presentation to the Board on International Comparative Studies in Education. September 23.
Lapointe, Archie E., Nancy A. Mead, and Eugene Johnson
1990 IAEP status. Presentation to the Board on International Comparative Studies in Education. February 2.
Lapointe, Archie E., Nancy A. Mead, and Janice M. Askew
1992 Learning Mathematics. Report of the International Assessment of Educational Progress. Report No. 22-CAEP-01. February. Educational Testing Service, Princeton, New Jersey.
Lapointe, Archie E., Janice M. Askew, and Nancy A. Mead
1992 Learning Science. Report of the International Assessment of Educational Progress. Report No. 22-CAEP-02. February. Educational Testing Service, Princeton, New Jersey.
Lapointe, Archie E., Nancy A. Mead, and Gary W. Phillips
1989 A World of Differences: An International Assessment of Mathematics and Science. Report of the First International Assessment of Educational Progress. Report No. 19-CAEP-01. January. Educational Testing Service, Princeton, New Jersey.
Lazer, Stephen
1992 Learning About the World. Report of The International Assessment of Educational Progress. Report No. 22-CAEP-05. June. Educational Testing Service, Princeton, New Jersey.
Mead, Nancy A.
1990 Second International Assessment of Educational Progress. Status report prepared for the Board on International Comparative Studies in Education. April 25.
1992 Second International Assessment of Educational Progress. Status report prepared for the Board on International Comparative Studies in Education. September.
1993 Secondary Analysis of IAEP Results. Status report prepared for the Board on International Comparative Studies in Education. January.
Semple, Brian McLean
1992 Performance Assessment: An International Experiment. Report of the International Assessment of Educational Achievement. Report No. 22-CAEP-06. July. Educational Testing Service, Princeton, New Jersey.
******
NOTE: This study summary was reviewed and edited by Nancy Mead at the Educational Testing Service on June 10, 1994.