Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
THE NATIONAL ACADEMIES Advisers to the Nation on Scienre, Engineering, and Meditine Division of Behavioral and Social Sciences and Education Center for Education Board on Testing and Assessment Peggy Carr, Ph.D. Associate Commissioner for Assessment National Center for Education Statistics 1990 K St., NW Washington DC 20006 Dear Dr. Carr: 500 Fifth Street, NW Washington, DC 20001 Phone: 202 334 3087 Fax: 202 334 1294 E-mail: bota1@nas.edu www. nationalacademies.org/bota June 26, 2003 We are writing on behalf of the National Research Council's Committee on Performance . Levels for Adult Literacy. As you know, the committee's charge is to examine and evaluate procedures used to arrive at the performance levels used for the 1992 National Adult Literacy Survey ant! to make recommendations regarding performance levels for the 2003 National Assessment of Aclult Literacy (NAAL). During the course of this 29-month project, the committee will review the process for setting the ~ 992 performance levels and the research that has been conducted on the ~ 992 assessment, ant! will arrange for activities and analyses to be concluctec} that can be used to establish the performance levels and associated cut points for the 2003 assessment. Current plans call for the committee to issue its final report in prepublication form in December 2004, with formal publication in March 2005. Our information-gathering efforts to ciate have informed us about the various enhancements the National Center for Education Statistics (NCES) plans to implement in conjunction with the 2003 assessment, inclucling Spanish translations of several of the assessment components, the Aclult Literacy Supplemental Assessment (ADS A), which is designecl to yield additional information about inclividuals with limited literacy skills, and the Fluency Addition to the NAAL (FAN), which assesses oral reacting skills. We commend you on these truly important efforts to expand the assessment and broaden its coverage of the country's aclult population. These enhancements should provide valuable information about the literacy skills of Americans. However, the committee is concerned! about several aspects of NCES's activities to date that could degracle the integrity of the 2003 results ant! the comparability of the 1992 and 2003 results. Several of these issues are time sensitive and should be addressed as soon as possible. The purpose of this letter is to bring these concerns to your attention prior to our final report and to recommenct that NCES conduct three special studies as part of the National Assessment of Adult Literacy. NATIONAL ACADEMY OF SCIENCES · NATIONALACADEMY OF ENGINEERING · INSTITUTE OF MEDICINE · NATIONAL RESEARCH COUNCIL
Classification Accuracy of the Core Questions As mentioned above, ALSA is clesigned to obtain more accurate information about individuals with limited literacy skills than is possible with the main NAAL assessment. A set of seven "core" questions is used as a routing test to estimate whether an individual's skills are sufficient to participate in the main NAA:L assessment or whether the incliviclual should be routed to ALSA. The core consists of relatively easy tasks that are similar in structure to those on NAAL. Each core task consists of stimulus materials presented in English and associates] questions available in English anct Spanish. The questions and directions for ALSA are similarly available in English ant! Spanish; NAAL is available only in English. NCES conducted a small field test on a convenience sample of 350 individuals that proviclec! limited information on the accuracy of the core questions for routing individuals to the correct version of the assessment. In this field test, the respondents completed all three components-the core questions, ALSA, and NAAL. Using the field test data, classification rules were developed to place individuals into ALSA or NAAL. The rules were derived from an a priori assumption that approximately ~ O percent of the field! test respondents should be screener! into ALSA. Separate classification rules were developed for individuals who took the English version ant! the Spanish version of the core questions. Classification consistency was examined by comparing the routing decision that would have been made on the basis of correct responses to the core questions with the score on NAAL. Respondents who scored five or lower on NAAL were considered to be individuals who should have been routed to ALSA (using the target of routing the lowest-scoring ~ O percent on the main NAAL to the A:LSA). Of the 350 subjects, 7 (2 percent) were misclassified: 4 were incorrectly routed to NAAL, and 3 were incorrectly routed to ALSA (U.S. Department of Education, n.d.~; see the table below. Classification Result NAAL Correctly Classified 31 Incorrectly Classified 4 Total 315 ALSA ~2 Because of the policy importance of identifying individuals with the most limited levels of literacy and understanding the nature of their skills, the estimate of the proportion of the adult population who would be routed to ALSA will be a natural focal point for discussions about assessment results. It is important to understand that the design of the 2003 assessment, with this separation into two instruments, essentially establishes a breakpoint that cannot be ignored when reporting the results or when determining performance levels, which is the primary charge of the committee. Therefore, it will be critical for NCES to understand the properties of the classification rules and the nature and extent of the literacy proficiency of individuals who are routed to ALSA. Furthermore, it is important to recognize that once the 2003 assessment establishes the breakpoint between ALSA and NAAL, future literacy assessments will need to be able to 2
establish a trend in the estimates of the proportion of the aclult population at this lowest level of literacy. The need for trend information will place a premium on designing a set of core questions for future assessments that will have the same properties as those of the current assessment. However, that task will be impossible unless the properties of the current core and its classification rules have been established firmly anal consistently in the current assessment. If NAAL were designed to report individual-level results, it would be critical that the set of routing questions be highly accurate in classifying individuals into the appropriate assessment. However, given that NAAL is designed to report on group-level performance and given the statistical methoclology used (Kirsch, Yamamoto, Norris, Rock, Jungeblut, & O'Reilly, 2001), accurate classification of inclividuals is not quite as critical. Post hoc statistical modeling enables estimation of group-level results, even when misclassifications occur. Nevertheless, the modeling requires reliable estimates of classification accuracy, as well as information about the relationships between performance on NAAL and ALSA. The committee has reviewed the information provided about the field test and finds that it was not sufficient to support precise estimates of the classification accuracy of the core questions. It also floes not provide sufficient information about the relationships between performance on NAAL and ALSA. Reliable estimates of classification error are not possible from a convenience sample for which the probabilities of selection are not known (Cochran, 1963~. Also, calculation of error rates should be based on appropriate populations, distinguishing people near the cutpoint for the main assessment and ALSA from other people, and English speakers from Spanish speakers, given the different rules for classification. RECOMMENDATION: The Department of Education' should cona~uct a special study in which a stratified random sample of individuals responds to the core and to both versions of the assessment, NAAl and ALSO. The sample should be a probability sample that is designed to estimate classification errors accurately and should be stratified on relevant variables, such as location, race or ethnicity, and educational attainment or years of schooling. The sample should be large enough to provide reliable information about classification accuracy, especially near the decision point for routing individuals to NEAL or ALSO, and it should be designed to yield information about ciassifcation errors in both directions. Incarcerated Indivicluals The second issue of concern to the committee pertains to assessment of the literacy skills of incarcerated inclividuals. The ~ 992 assessment incluclec! a special study of the inmate population in the Unitecl States and reported both aggregated and (1isaggregated results for incarcerated and nonincarcerated inclivicluals. It is our unclerstanding that NCES is considering, but has not yet made definite plans for, a special study of incarcerated individuals in conjunction with the 2003 NAAL. We believe such a study is critical. Failure to include a sample of incarcerates! individuals will both degrade the integrity of comparisons of the results from 1992 to 2003 and misrepresent the literacy skills of key groups that have experiences! sharp increases in incarceration rates. Since the ~ 992 NALS, there has been a substantial increase in the proportion of incarcerated individuals in the Uniter! States. 3
According to the U.S. Bureau of Justice Statistics (Harnson & Beck, 2002: Table 6.23), the incarceration rate in fecleral and state prisons increased from 332 per 100,000 in 1992 to 470 per 100,000 in 2001, a 41 percent increase in the incarceration rate. This increase reflects several factors, ranging from the increasingly popular attitude of being "tough on crime" since the early 199Os to new laws with stricter sentencing for personal crimes, misdemeanors, and drug use ant! possession, as well as stricter enforcement of existing laws. Rates of incarceration are exceptionally high among young black men. In 2001, 3.5 percent of black non-Hispanic men were incarcerated, in comparison with 0.5 percent of white males (Harrison & Beck, 2002, Table 16~. The comparable figures for 1992 were 2.7 percent of black non-Hispanic males and 0.37 percent of whites (Gilliarc! & Beck, 1994; pg. 9; Beck ant! Gilliard, 1995, Table ~ I). For black non-Hispanic males, the increase from 2.7 percent to 3.5 percent represents an increase of 0.S percentage points. Moreover, for some age ranges of black males, the rate of incarceration ant! increases in incarceration rates over the past decacle are several times larger. For example, for black males aged 20-24, the incarceration rate increased from 5.3 percent in 1992 to 7.9 percent in 2001, an increase of 2.6 percentage points. (Harnson & Beck, 2002) incarceration rates have increased sharply for black men who have not completed high school, a group of special interest with respect to literacy skills, and these rates have risen substantially over the past clecacle. In 1999, the incarceration rate among black men aged 22 to 30 who were high school dropouts was 40 percent (Western & Pettit, 2000~. The comparable figure for 1992 was 26 percent (Western & Pettit, 20004. These incarceration rates are so high among young black male dropouts that failure to include them in the 2003 NAAL will seriously bias the sampling frame for young black male dropouts, for all black men, and for all blacks, with unknown consequences for the literacy estimates for these groups. RECOMMENDATION: The Department of Education' shouldproceed with plans to conduct a special study of the literacy skills of incarcerated individuals and incorporate the information into the reported NEAT results, as appropriate. Non respon dents The thirc! issue of concern to the committee relates to coverage of the population and decisions about who is included in the assessment and who is not. Our concerns relate primarily to procedures for evaluating the literacy skills (in English) of inclividuals who do not have a strong command of English. There has been a significant increase in such indivicluals. In 1990, 13.S percent of the U.S. population over the age of 5 spoke a language other than English in the home: for 7.5 percent of the population, Spanish was spoken in the home, and for 1.9 percent of the population, Asian and Pacific Island languages were spoken in the home. According to the most recent census, these percentages have increased over the past decacle. Just 10 years later in ~ , 2000' 17.9 percent ofthe U.S. population over age 5 spoke a language other than English in the home: for 10.7 percent of the population, Spanish is the language spoken in the home, and for 2.7 percent, Asian and Pacific Island languages are spoken in the home 4
(http://www.censusscope.org/us/chart_language.htmI). Since NAAL is intended to be an assessment of the English literacy skills of the entire aclult population, generalization of the results will be limited to the extent that certain groups, such as non-native English speakers, are excluded. Since procedures are aireacly in place and data collection is about to begin, it may not be practical to alter or modify the soon-to-be-operational assessment. However, it should be possible to conduct a follow-up study of nonrespondents to evaluate population coverage, particularly as it relates to language issues, to estimate nonresponse bias, anct to gather information about needs for the next adult literacy assessments. Studies of nonrespondents are a standard part of high quality large-scale surveys, such as the U.S. census, and we believe they shouicI be undertaken for NAAL (Groves & Couper,1998; Kalton & Kasprzyk,19864. According to the information proviclect to us (personal communication, 2/26/03), there are three decision points when the interviewer may decide to discontinue the assessment before it is complete. The way that language issues are addressed at each stage is as follows: 1. When an interviewer arrives at a sampled househoIct, a screening crevice (the "screener") is used to determine if there is an eligible person in the household to participate in the assessment. If the respondent cannot unclerstand the English or Spanish spoken by the interviewer (or vice versa), the interviewer can solicit translation assistance from another household member or a neighbor. If this is not possible, the assessment wouict cease, and the case would be coded as "language problem." 2. After the screener has been administered, the interviewer begins the background questionnaire, which is available in English and Spanish. Translation assistance is not allowed for the background} questionnaire; thus if the respondent cannot unclerstand the English or Spanish spoken by the interviewer (or vice versa), the background questionnaire would not be aciministered, and the case would be coded as "language problem," ant! the cognitive components wouIct not be aciministered.~ 3. The next stage of the assessment involves administering the cognitive components, beginning with the core and moving to either NAAL or ALSA. The guidelines at this ~ ' 1 ' ~ 11 ~ ' ~ ~1 1 ' ~ · 1 1 ~ 1 1 , 1 stage ~na~cate anal ~~ tne suspect Is unable to speak, uncterstanct, read, or write English or Spanish, these components cannot be administered. Incomplete assessments are coded accorcting to the extent of completion (partially complete or not started) and reason for noncompletion (language problem or reacting/writing banner). 1 ~ - / ~ 1 The individuals who make it through these fiItenng stages become the survey respondents and define the population to which the results generalize. Thus, it is important to understand! who is excluder! from the NAAL assessment ant! the effects of such nonparticipation on the estimated literacy of the U.S. adult population. ' The procedure for moving from the background questionnaire to the core may be implemented somewhat more flexibly than the stated rules. The 2003 background questionnaire is administered through a computer-assisted personal interview (CAPI ~ system, which allows individuals to move to the core questions even though the background questionnaire is not completed. as
In the ~992 assessment, the overall unweighted response rates (after al] nonresponse conversion attempts) were 89. ~ percent for the screener, ~ ~ .0 percent for the background Questionnaire. and SS.6 percent for the exercise Locket of the cognitive assessment (Kir.~ch et .. . . ~ ~ . ~ . al., 2001:125-26~.2 Although it is important to evaluate nonresponse bias in general, it is particularly important for an assessment of literacy to evaluate nonresponse bias that is attributed to a "language problem." In the ~ 992 assessment. the nonresnonse rates attributer! to a language 4_ ~ ~ ~ 1 =7 o 1 1 A ~ ~ 1` J 1 ~ ~ . 1 1 ~ ~ . · ' problem were (~.3 percent for the screener, (). / percent tor the background questionnaire, and 3.2 percent for the exercise booklet (composed of I.8 percent total incompletes ant! I.4 percent partial incompletes) (Kirsch et al., 20014. A sensitive nonresponse study could tease out additional information about reasons for cropping out or refusing to participate, about the nature and extent of language problems that prevent NAAL participation, and about the competency in English of those who could not participate. It is quite likely that the reasons for cropping out of the NAAL assessment are related to literacy proficiency or lack of English proficiency regardless of the stated reasons. For instance, individuals may decline to participate or (lrop out during the first two stages simply because they realize that the assessment is about literacy (in English) ant! believe that it is not one of their strong points. Focused follow-up studies could provide aclditional information about the English language ant! literacy skills of those who dropped out during the early stages by making adaptations to the stanciarct procedures, such as: orgy aciministenng the core questions (to obtain an estimate of the percentage of nonresponclents who would have been routed to NAAL or ALSA), arranging for a translator for speakers of languages other than English or Spanish, or verifying the validity of other reasons for refusing to participate (such as problems with visual acuity) or some combination of these adaptations. As part of the follow up studies, information could be gathered on the language spoken in the home and the length of time individuals have lived in the United States. An attempt should also be made to learn about indivicluals' literacy skills in the native language. We are not suggesting that NCES (levelop mechanisms for formally assessing literacy skills in a multitude of languages, only that information be gathered that could further our unclerstanding about the literacy skills of individuals who do not speak English. Information from these follow-up studies could enhance understanding of the 2003 results, assist in developing policy regarding remedial literacy and ESE programs for adults, ant} be user! in designing future literacy assessments. By offering the instructions and questions related to the core stimulus materials ant! ALSA in Spanish, NCES has macle great strides in expancling the coverage of this assessment. However, it is important to be clear about exactly who is included, who is exclucled, and reasons for nonparticipation. 2 Response rates were calculated as follows: 89.1 percent responded to the screener; of those, 81.0 percent responded to the background questionnaire; of those, 88.6 percent responded to the cognitive component itself. 6
RECOMMENDATION: The Department of Education should conduct a special follow-up study of nonrespondents. The study should involve identifying a probability sample of norlrespond~ents (perhaps stratif ed' by geographic location or other information to identify types of nonrespondents), focusing particularly on individuals who are eliminated during the f rst two stages. Data collected from this group shouic! focus on reasons for not participating and should probe further into language issues by ascertaining information about command of English and the likely [eve! of literacy skills in the native language. Analysis of data from the follow-up studies should examine systematic differences between respondents and nonrespor`dents that might bias the results and should! identify issues to consider for future administrations of HAAG. We hope that you find our recommendations useful and informative, and we urge funding for these three important studies. We look forward to working with you further on this project. Sincerely, Christopher F. Edley, Jr., Co-chair U. ~ .. ,W': Robert M. Hauser, Co-chalr 7
References Beck, A., and Gilliard, D. (19954. Bureau of Justice Statistics Bulletin: Prisoners in 1994. Washington, DC: US Department of Justice. Cochran, William G. (1963~. Sampling Techniques (second edition). New York: Wiley. Giliard, D., ant! Beck, A. (1994~. Bureau of Justice Statistics Bulletin: Prisoners in 1993. Washington, DC: US Department of Justice. Groves, Robert M., ant! Mick P. Couper (1998~. Nonresponse in Household interview Surveys. New York: Wiley. Harrison, P., and Beck, A. (20024. Bureau of Justice Statistics Bulletin: Washington, DC: US Department of Justice. Prisoners in 2001. Kalton, G., and Kasprzyk, D. (19864. The treatment of missing survey data. Survey Methodology, 112, I-16. Kirsch, 1., Yamamoto, K., Norris, N., Rock, D., Jungeblut, A., O'Reilly, P., Berlin, M., Mohandjer, L., Waksberg, J., Goksel, H., Burke, J., Rieger, S., Green, J., Klein, M., Campbell, A., Jenkins L., Kolstad, A., Mosenthal, P., and Baldi, S. (2001~. Technical Report and Data Fzle User's manualfor the 1992 National Aclult Literacy Survey (NCES 2001-457). Washington, DC: US Department of Education, National Center for Education Statistics. US Department of Education. (n.cI.~. Selecting ALSO Respondents. Washington, DC: Author. Western, B., and Pettit, B. (20004. Incarceration anti racial inequality in men's employment. Inclustrial and Labor Relations Review, 54:3-16.
COMMITTEE ON PERFORMANCE LEVELS FOR ADULT LITERACY CHRISTOPHER F. EDLEY, Jr. (Co-Chair), Harvard Law School ROBERT M. HAWSER (Co-Chair), Center for Demography of Health and Aging, University of Wisconsin JUDITH A. ALAMPRESE, Abt Associates Inc., Bethesda, MD MICHAEL X. DELLl CARPINI, The Pew Charitable Trusts, Philaclelphia, PA VIVIAN L. GADSDEN, Graduate School of Education, University of Pennsylvania ANDREW J. HARTMAN, Bell Policy Center, Denver, CO GLYNDA A. HULL, Graduate School of Education, University of California, Berkeley REBECCA A. MAYNARD, Graduate School of Education, University of Pennsylvania LORRAINE MCDONNELL, Department of Political Science, University of California, Santa Barbara LARRY MIKULECKY, Language Eclucation Department, Tncliana University ROBERT J. MISLEVY, Department of Measurement, Statistics, and Evaluation, University of Maryland NORMAN G. PETERSON, Satisfaction Performance Research (SPR) Center, Inc., Minneapolis, MN JOHN P. POGGIO, Department of Psychology and Research in Eclucation, University of Kansas RIMA RUDD, Department of Health ant} Social Behavior, Harvard School of Public Health MARY JANE SCHMITT, TERC, Cambridge, MA DAVID M. THISSEN, Department of Psychology, University of North Carolina, Chapel Hill HElDE SPRUCK WRIGLEY, Aguirre International, HQ, San Mateo, CA JUDITH A. KOENIG, Study Director CONNIE CITRO, Senior Program Officer STUART W. ELLIOTT, Senior Program Officer ANDREW E. TOMPKINS, Research Assistant MICHAEL DECARMINE, Project Assistant
