Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 42
International Comparative Studies in Education: Descriptions of Selected Large-Scale Assessments and Case Studies INTERNATIONAL ADULT LITERACY STUDY International Organization Statistics Canada Years of Data Collection Pilot: 1993 Main data collection: 1994 Purpose The International Assessment of Adult Literacy is a joint project of the Educational Testing Service and Statistics Canada. Its basic objective is to concurrently profile the literacy skills of adults in a number of countries/languages. Methodologically the study represents a fusion of educational testing and survey research. In order to build policy-relevant new data sets to feed the public discourse on emerging policy issues, the IALS is designed as a fusion of educational testing (of the type that goes on in the school system) with household surveys. The objective is to concurrently administer the same measure in a number of languages and countries to profile adult literacy skills. Although this objective is simple, achieving it is complicated. IALS is based on the collective experience of Educational Testing Service and Statistics Canada with about 50,000 respondents. It conceptually close to the Survey of Literacy Skills Used in Daily Activities (LSUDA), which was fielded by Statistics Canada in 1989. The IALS will incorporate an international dimension to enable comparison of literacy profiles between countries. International objectives are to: test the notion that the decoding and decision making skills embodied in the Canadian and U.S. assessments are stable across language groups and cultures promote the conception of literacy underlying the direct assessments conducted in Canada and the United States provide comparative data on adults and workers in countries that represent a cross-section of Canada's current and emerging trading partners compare and contrast the literacy skill profiles for economically important sub-populations across countries and language groups. In addition, this study has objectives specific to Canada: provide an updated profile of adult literacy abilities for Canada for comparison to that provided by the Survey of Literacy Skills Used in Daily Activities provide sufficiently large numbers of visible minorities, seniors, recent graduates, and out of school youth to profile their skill levels shed light on the relationship between performance, educational attainment, labor market participation and employment for those individuals found to be at Level 3. IALS promotes a modified model of learning. The old model was based on the concept that learning is centered exclusively in a formal system; a single learning event early in life (focus on youth); once acquired, literacy skills are viewed as a static commodity, and the adult milieu has no effect on skill retention or enhancement. The new model is based on the concept that self-directed instruction
OCR for page 43
International Comparative Studies in Education: Descriptions of Selected Large-Scale Assessments and Case Studies complements the formal system; repeated learning events (focus includes adults); skill attrition; and opportunity to learn plays a crucial role in skill maintenance. IALS incorporates a modified model of reading and yields current estimates of the magnitude of the literacy “problem” across a continuum of skill levels. IALS allows analysis of the covariates of measured performance. This information is useful in understanding etiology of the phenomenon and identifying populations at risk. This study model offers information on behaviors, motivational factors, and self-assessed skill levels. These data are useful in identifying barriers to remediation and are key input for optimizing program design, delivery, and marketing. The IALS model also renders explicit the relationship between tested performance and traditional literacy proxies, such as educational attainment. This information is useful in projecting the evolution of skill profile and in generating profiles for small geographic areas. Some higher order benefits of the IALS are: unique collaboration between policy makers, collection experts, and measurement experts that have the subject matter knowledge; promotion of NALS measurement protocol; and determining the cultural limits of the conceptual framework. U.S. Department of Education recent interest in the current IALS design (which now contains connections with the work force) is in its relating to the importance placed on end results of the educational process and to U.S. national education goals, particularly the fifth goal on work force skills and adult literacy concerns (which in international comparison is work force competence). As Statistics Canada has assumed leadership in this project, it becomes possibly the first government-to-government international study. The U.S. Department of Education, as well as the U.S. Department of Labor and other U.S. government agencies are interested in a joint collaboration, co-funding the U.S. participation in this study. It is smaller in scale in terms of the number of countries that might be involved, more clearly targeted towards policymakers ' interest, and coordinated or carried out by government agencies that have the same concerns about the quality of the studies demanded for supporting policy decision making. The U.S. Department of Education is particularly interested in conducting a linkage study to the U.S. national adult literacy survey, to link the rich background information to an international context and look at the performance of different groups in the U.S. population relative to other countries, as well as a country-to-country comparison. An additional higher order benefit of the IALS is getting other governments interested in government-to-government initiatives that would become another model for conducting international studies and perhaps a more efficient and effective model for doing that. In the interest of stability and viability of the organization structure for international studies, an IALS government-to-government initiative would provide a valuable opportunity to begin networking with stable government organizations -- which have people who are concerned about the quality of a study and who have data management experience as well as the existence of a survey that can serve as the vehicle for a study --provides an interesting new approach for conducting international studies. Organization and Management An agreement in principle with National Center for Education Statistics has been established that the study should become a joint government undertaking (rather than an Educational Testing Service-Statistics Canada undertaking). Negotiation of terms is underway with NCES. Whatever the outcome of these negotiations, ETS will continue to play a central technical role in the study.
OCR for page 44
International Comparative Studies in Education: Descriptions of Selected Large-Scale Assessments and Case Studies Design Participants Canada, France, Germany, Ireland, the Netherlands, Poland, Switzerland, and the United States (Mexico had to withdraw). Discussions are underway with Australia, Italy, Japan, New Zealand, Portugal, and the United Kingdom concerning the costs and criteria for joining IALS at this juncture. Sample The study will yield a sample of 3,000 adult Canadians. This sample will be enriched to ensure a minimum yield of 500 cases for seniors, in-school youth, out-of-school youth, official-language visible minorities, and allophone visible minorities, bringing the total sample to approximately 4,800. IALS methodology requires a representative probability sample of adults 16-65, a minimum yield of roughly 3,000 cases, and 500-600 cases required for each domain/geography to be profiled. Procedures and Summary of Content The study will include a background questionnaire, a brief set of core items, and a large main test set of simulation tasks. The core items and simulation tasks will assess three aspects of literacy: prose literacy, document literacy, and quantitative literacy. In order to ensure broad coverage of the three literacy domains and not to overburden individual assessment takers, a Balanced Incomplete Block (BIB) assessment design will be utilized. Item response theory (IRT) will be utilized to estimate performance on each of the three literacy domains. Results will be reported in terms of proficiency levels. IALS technical challenges include development of a multilingual test with good psychometric properties; ensuring comparability of key elements of a background questionnaire (e.g., educational attainment, industry, occupation); extension of a test to more skill dimensions (e.g., problem solving, communication); interviewer administered collection in each respondent's home; and interview length of roughly 90 minutes. (In Canada each case costs $150-$200.) An interviewer administers each element serially in a respondent's home. The IALS background questionnaire includes classification variables (household arrangements, place of birth and immigration, income, education, language, and ancestral/parental information); labor market activity (labor force history, labor force status, career strategies); workplace skills (general skills and literacy skills); training/adult education (nature of, attitudes to, outcomes of); and general literacy concerns. Data Collection and Analyses Assessment is virtually complete, having passed the final series of reviews. The background questionnaire is also in the final review stage, with participants having agreed to a set of mandatory items to be carried in all countries. Canada, France, Germany, Ireland, and the Netherlands have all completed large scale pilot surveys that have been analyzed by ETS. Switzerland and Poland have conducted more modest tests of their instruments. In April 1994 participating countries met in Hamburg to discuss the results of the pilot surveys and to plan for implementation of the main study (main data collection in September 1994). Decisions made at the April 1994 meeting were: Based on an analysis of pilot assessment data by ETS and Stan Jones, Canada's technical consultant on the study, participating countries have agreed to a final item selection and test design that will afford the psychometric equivalence needed to support international comparisons.
OCR for page 45
International Comparative Studies in Education: Descriptions of Selected Large-Scale Assessments and Case Studies Countries have committed to field a comprehensive set of standardized background variables including modules on adult education participation and income. Both of these modules had been problematic to at least some of the countries prior to piloting. Countries have agreed to a proposed content and structure for the international comparative report, as well as a process for vetting successive drafts of the report as they become available. Approval in principle was also reached to have the report released as a joint Statistics Canada-OECD publication. A compromise was reached concerning access to microdata following release of the international comparative report. The Canadian tradition would have seen non-confidential microdata being released concurrently with the comparative report. Countries have agreed in principle to a controlled release for a period of a year after which the dataset would become freely available. Timetable 1993 The survey was piloted. 1994 Main data will be collected. (September) Publications International comparative report: A single international report will focus on the similarities and differences in the literacy profiles of participating countries and their implications for ational and international policy making. This report will be released as a joint Statistics Canada-OECD publication. Canadian reports will include a first report to present a comparison of the IALS data to LSUDA data; a second report that will focus on rare population subgroups for which LSUDA failed to provide sufficient sample; a third report that will reflect on the policy implications of the comparisons to other participating countries. Database Name International Adult Literacy Survey (IALS) Description A public-use microdata file and associated documentation in electronic form containing results from the International Adult Literacy Survey (IALS), a nine country direct assessment of prose literacy, document literacy, and numeracy. To whom available All researchers. Restrictions Restricted access to microdata file January-December 1996. Interested users will be required to submit a research proposal, which will be vetted by a review committee. Data from 1997 file will be freely accessible. By what means available The file will take the form of a CD-ROM, which will contain the data file, extraction software, and related documentation. Charge to user To be determined.
OCR for page 46
International Comparative Studies in Education: Descriptions of Selected Large-Scale Assessments and Case Studies When available January 1996. For further Information about access to this database, contact: Jean Pignal The Special Surveys Division, Statistics Canada Ottawa, Ontario, Canada K1A 0T6 telephone: 613/ 951-3317 facsimile: 613/ 951-0562 electronic mail: email@example.com Funding The European Community and IALS European participants have agreed to underwrite $200,000 of the estimated $400,000 cost of international overheads associated with the study. The National Center for Education Statistics has also agreed in principle to support the study and continue to search for the funds necessary to do so. The bulk of this money will be directed towards the purchase of services from the Educational Testing Service. The Educational Testing Service has provided a proposal to Statistics Canada to cover their continued involvement in the study, and Statistics Canada, as study manager, has agreed to fund most of what has been presented. In addition, Statistics Canada has been working with the U.S. Bureau of the Census to develop a collection strategy for the U.S. linking study, which the National Center for Education Statistics can fund. This study, which would see the administration of a hybrid NALS/IALS assessment to 3,150 Americans, is a critical element in providing the statistical basis for placing other countries on the U.S. scales. NCES has also agreed in principle to sign an inter-agency agreement to cover this work. Information Sources T. Scott Murray, Assistant Director Household Surveys, Statistics Canada Ottawa, Canada K1A 0T6 telephone: 613 / 951-9476 facsimile: 613/ 951-0562 Internet: firstname.lastname@example.org Statistics Canada, The Special Surveys Group 1992 An International Assessment of Adult Literacy: A Proposal. Draft. December. Murray, T. Scott 1993 Update of 1992 draft proposal. May 28 letter to Dorothy M. Gilford, director, Board on International Comparative Studies in Education. 1993 International Adult Literacy Survey. Status report to the Board on International Comparative Studies in Education. October. 1994 International Adult Literacy Survey. Status reports to the Board on International Comparative Studies in Education. January and April. Presentation to the board. May. ******
OCR for page 47
International Comparative Studies in Education: Descriptions of Selected Large-Scale Assessments and Case Studies NOTE: This study summary was reviewed and edited by T. Scott Murray at Statistics Canada in Ottawa on May 31, 1994; database information was provided by Murray on December 9, 1994.
Representative terms from entire chapter: