|
Items in cart [0] |
Questions? Call 888-624-8373 |
|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
| ||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 1
THE NATIONAL ACADEMIES
Advisers to the Nation on Scienre, Engineering, and Meditine
Division of Behavioral and Social Sciences and Education
Center for Education
Board on Testing and Assessment
Peggy Carr, Ph.D.
Associate Commissioner for Assessment
National Center for Education Statistics
1990 K St., NW
Washington DC 20006
Dear Dr. Carr:
500 Fifth Street, NW
Washington, DC 20001
Phone: 202 334 3087
Fax: 202 334 1294
E-mail: bota1@nas.edu
www. nationalacademies.org/bota
June 26, 2003
We are writing on behalf of the National Research Council's Committee on Performance
.
Levels for Adult Literacy. As you know, the committee's charge is to examine and evaluate
procedures used to arrive at the performance levels used for the 1992 National Adult Literacy
Survey ant! to make recommendations regarding performance levels for the 2003 National
Assessment of Aclult Literacy (NAAL).
During the course of this 29-month project, the committee will review the process for
setting the ~ 992 performance levels and the research that has been conducted on the ~ 992
assessment, ant! will arrange for activities and analyses to be concluctec} that can be used to
establish the performance levels and associated cut points for the 2003 assessment. Current
plans call for the committee to issue its final report in prepublication form in December 2004,
with formal publication in March 2005.
Our information-gathering efforts to ciate have informed us about the various
enhancements the National Center for Education Statistics (NCES) plans to implement in
conjunction with the 2003 assessment, inclucling Spanish translations of several of the
assessment components, the Aclult Literacy Supplemental Assessment (ADS A), which is
designecl to yield additional information about inclividuals with limited literacy skills, and the
Fluency Addition to the NAAL (FAN), which assesses oral reacting skills. We commend you on
these truly important efforts to expand the assessment and broaden its coverage of the country's
aclult population. These enhancements should provide valuable information about the literacy
skills of Americans. However, the committee is concerned! about several aspects of NCES's
activities to date that could degracle the integrity of the 2003 results ant! the comparability of the
1992 and 2003 results. Several of these issues are time sensitive and should be addressed as
soon as possible. The purpose of this letter is to bring these concerns to your attention prior to
our final report and to recommenct that NCES conduct three special studies as part of the
National Assessment of Adult Literacy.
NATIONAL ACADEMY OF SCIENCES · NATIONALACADEMY OF ENGINEERING · INSTITUTE OF MEDICINE · NATIONAL RESEARCH COUNCIL
OCR for page 2
Classification Accuracy of the Core Questions
As mentioned above, ALSA is clesigned to obtain more accurate information about
individuals with limited literacy skills than is possible with the main NAAL assessment. A set of
seven "core" questions is used as a routing test to estimate whether an individual's skills are
sufficient to participate in the main NAA:L assessment or whether the incliviclual should be routed
to ALSA. The core consists of relatively easy tasks that are similar in structure to those on
NAAL. Each core task consists of stimulus materials presented in English and associates]
questions available in English anct Spanish. The questions and directions for ALSA are similarly
available in English ant! Spanish; NAAL is available only in English.
NCES conducted a small field test on a convenience sample of 350 individuals that
proviclec! limited information on the accuracy of the core questions for routing individuals to the
correct version of the assessment. In this field test, the respondents completed all three
components-the core questions, ALSA, and NAAL. Using the field test data, classification rules
were developed to place individuals into ALSA or NAAL. The rules were derived from an a
priori assumption that approximately ~ O percent of the field! test respondents should be screener!
into ALSA. Separate classification rules were developed for individuals who took the English
version ant! the Spanish version of the core questions.
Classification consistency was examined by comparing the routing decision that would
have been made on the basis of correct responses to the core questions with the score on NAAL.
Respondents who scored five or lower on NAAL were considered to be individuals who should
have been routed to ALSA (using the target of routing the lowest-scoring ~ O percent on the main
NAAL to the A:LSA). Of the 350 subjects, 7 (2 percent) were misclassified: 4 were incorrectly
routed to NAAL, and 3 were incorrectly routed to ALSA (U.S. Department of Education, n.d.~;
see the table below.
Classification Result NAAL
Correctly Classified 31
Incorrectly Classified 4
Total 315
ALSA
~2
Because of the policy importance of identifying individuals with the most limited levels
of literacy and understanding the nature of their skills, the estimate of the proportion of the adult
population who would be routed to ALSA will be a natural focal point for discussions about
assessment results. It is important to understand that the design of the 2003 assessment, with this
separation into two instruments, essentially establishes a breakpoint that cannot be ignored when
reporting the results or when determining performance levels, which is the primary charge of the
committee. Therefore, it will be critical for NCES to understand the properties of the
classification rules and the nature and extent of the literacy proficiency of individuals who are
routed to ALSA.
Furthermore, it is important to recognize that once the 2003 assessment establishes the
breakpoint between ALSA and NAAL, future literacy assessments will need to be able to
2
OCR for page 3
establish a trend in the estimates of the proportion of the aclult population at this lowest level of
literacy. The need for trend information will place a premium on designing a set of core
questions for future assessments that will have the same properties as those of the current
assessment. However, that task will be impossible unless the properties of the current core and
its classification rules have been established firmly anal consistently in the current assessment.
If NAAL were designed to report individual-level results, it would be critical that the set
of routing questions be highly accurate in classifying individuals into the appropriate assessment.
However, given that NAAL is designed to report on group-level performance and given the
statistical methoclology used (Kirsch, Yamamoto, Norris, Rock, Jungeblut, & O'Reilly, 2001),
accurate classification of inclividuals is not quite as critical. Post hoc statistical modeling enables
estimation of group-level results, even when misclassifications occur. Nevertheless, the
modeling requires reliable estimates of classification accuracy, as well as information about the
relationships between performance on NAAL and ALSA.
The committee has reviewed the information provided about the field test and finds that it
was not sufficient to support precise estimates of the classification accuracy of the core
questions. It also floes not provide sufficient information about the relationships between
performance on NAAL and ALSA. Reliable estimates of classification error are not possible
from a convenience sample for which the probabilities of selection are not known (Cochran,
1963~. Also, calculation of error rates should be based on appropriate populations,
distinguishing people near the cutpoint for the main assessment and ALSA from other people,
and English speakers from Spanish speakers, given the different rules for classification.
RECOMMENDATION: The Department of Education' should cona~uct a special study
in which a stratified random sample of individuals responds to the core and to both
versions of the assessment, NAAl and ALSO. The sample should be a probability
sample that is designed to estimate classification errors accurately and should be
stratified on relevant variables, such as location, race or ethnicity, and educational
attainment or years of schooling. The sample should be large enough to provide
reliable information about classification accuracy, especially near the decision point
for routing individuals to NEAL or ALSO, and it should be designed to yield
information about ciassifcation errors in both directions.
Incarcerated Indivicluals
The second issue of concern to the committee pertains to assessment of the literacy skills
of incarcerated inclividuals. The ~ 992 assessment incluclec! a special study of the inmate
population in the Unitecl States and reported both aggregated and (1isaggregated results for
incarcerated and nonincarcerated inclivicluals. It is our unclerstanding that NCES is considering,
but has not yet made definite plans for, a special study of incarcerated individuals in conjunction
with the 2003 NAAL. We believe such a study is critical. Failure to include a sample of
incarcerates! individuals will both degrade the integrity of comparisons of the results from 1992
to 2003 and misrepresent the literacy skills of key groups that have experiences! sharp increases
in incarceration rates. Since the ~ 992 NALS, there has been a substantial increase in the
proportion of incarcerated individuals in the Uniter! States.
3
OCR for page 4
According to the U.S. Bureau of Justice Statistics (Harnson & Beck, 2002: Table 6.23),
the incarceration rate in fecleral and state prisons increased from 332 per 100,000 in 1992 to 470
per 100,000 in 2001, a 41 percent increase in the incarceration rate. This increase reflects
several factors, ranging from the increasingly popular attitude of being "tough on crime" since
the early 199Os to new laws with stricter sentencing for personal crimes, misdemeanors, and drug
use ant! possession, as well as stricter enforcement of existing laws.
Rates of incarceration are exceptionally high among young black men. In 2001, 3.5
percent of black non-Hispanic men were incarcerated, in comparison with 0.5 percent of white
males (Harrison & Beck, 2002, Table 16~. The comparable figures for 1992 were 2.7 percent of
black non-Hispanic males and 0.37 percent of whites (Gilliarc! & Beck, 1994; pg. 9; Beck ant!
Gilliard, 1995, Table ~ I). For black non-Hispanic males, the increase from 2.7 percent to 3.5
percent represents an increase of 0.S percentage points. Moreover, for some age ranges of black
males, the rate of incarceration ant! increases in incarceration rates over the past decacle are
several times larger. For example, for black males aged 20-24, the incarceration rate increased
from 5.3 percent in 1992 to 7.9 percent in 2001, an increase of 2.6 percentage points. (Harnson
& Beck, 2002)
incarceration rates have increased sharply for black men who have not completed high
school, a group of special interest with respect to literacy skills, and these rates have risen
substantially over the past clecacle. In 1999, the incarceration rate among black men aged 22 to
30 who were high school dropouts was 40 percent (Western & Pettit, 2000~. The comparable
figure for 1992 was 26 percent (Western & Pettit, 20004. These incarceration rates are so high
among young black male dropouts that failure to include them in the 2003 NAAL will seriously
bias the sampling frame for young black male dropouts, for all black men, and for all blacks,
with unknown consequences for the literacy estimates for these groups.
RECOMMENDATION: The Department of Education' shouldproceed with plans to
conduct a special study of the literacy skills of incarcerated individuals and incorporate
the information into the reported NEAT results, as appropriate.
Non respon dents
The thirc! issue of concern to the committee relates to coverage of the population and
decisions about who is included in the assessment and who is not. Our concerns relate primarily
to procedures for evaluating the literacy skills (in English) of inclividuals who do not have a
strong command of English. There has been a significant increase in such indivicluals. In 1990,
13.S percent of the U.S. population over the age of 5 spoke a language other than English in the
home: for 7.5 percent of the population, Spanish was spoken in the home, and for 1.9 percent of
the population, Asian and Pacific Island languages were spoken in the home. According to the
most recent census, these percentages have increased over the past decacle. Just 10 years later in
~ ,
2000' 17.9 percent ofthe U.S. population over age 5 spoke a language other than English in the
home: for 10.7 percent of the population, Spanish is the language spoken in the home, and for
2.7 percent, Asian and Pacific Island languages are spoken in the home
4
OCR for page 5
(http://www.censusscope.org/us/chart_language.htmI). Since NAAL is intended to be an
assessment of the English literacy skills of the entire aclult population, generalization of the
results will be limited to the extent that certain groups, such as non-native English speakers, are
excluded.
Since procedures are aireacly in place and data collection is about to begin, it may not be
practical to alter or modify the soon-to-be-operational assessment. However, it should be
possible to conduct a follow-up study of nonrespondents to evaluate population coverage,
particularly as it relates to language issues, to estimate nonresponse bias, anct to gather
information about needs for the next adult literacy assessments. Studies of nonrespondents are a
standard part of high quality large-scale surveys, such as the U.S. census, and we believe they
shouicI be undertaken for NAAL (Groves & Couper,1998; Kalton & Kasprzyk,19864.
According to the information proviclect to us (personal communication, 2/26/03), there are
three decision points when the interviewer may decide to discontinue the assessment before it is
complete. The way that language issues are addressed at each stage is as follows:
1.
When an interviewer arrives at a sampled househoIct, a screening crevice (the "screener")
is used to determine if there is an eligible person in the household to participate in the
assessment. If the respondent cannot unclerstand the English or Spanish spoken by the
interviewer (or vice versa), the interviewer can solicit translation assistance from another
household member or a neighbor. If this is not possible, the assessment wouict cease, and
the case would be coded as "language problem."
2. After the screener has been administered, the interviewer begins the background
questionnaire, which is available in English and Spanish. Translation assistance is not
allowed for the background} questionnaire; thus if the respondent cannot unclerstand the
English or Spanish spoken by the interviewer (or vice versa), the background
questionnaire would not be aciministered, and the case would be coded as "language
problem," ant! the cognitive components wouIct not be aciministered.~
3. The next stage of the assessment involves administering the cognitive components,
beginning with the core and moving to either NAAL or ALSA. The guidelines at this
~ ' 1 ' ~ 11 ~ ' ~ ~1 1 ' ~ · 1 1 ~ 1 1 , 1
stage ~na~cate anal ~~ tne suspect Is unable to speak, uncterstanct, read, or write English or
Spanish, these components cannot be administered. Incomplete assessments are coded
accorcting to the extent of completion (partially complete or not started) and reason for
noncompletion (language problem or reacting/writing banner).
1 ~ - / ~ 1
The individuals who make it through these fiItenng stages become the survey
respondents and define the population to which the results generalize. Thus, it is important to
understand! who is excluder! from the NAAL assessment ant! the effects of such nonparticipation
on the estimated literacy of the U.S. adult population.
' The procedure for moving from the background questionnaire to the core may be implemented somewhat more
flexibly than the stated rules. The 2003 background questionnaire is administered through a computer-assisted
personal interview (CAPI ~ system, which allows individuals to move to the core questions even though the
background questionnaire is not completed.
as
OCR for page 6
In the ~992 assessment, the overall unweighted response rates (after al] nonresponse
conversion attempts) were 89. ~ percent for the screener, ~ ~ .0 percent for the background
Questionnaire. and SS.6 percent for the exercise Locket of the cognitive assessment (Kir.~ch et
.. . . ~ ~ . ~ .
al., 2001:125-26~.2 Although it is important to evaluate nonresponse bias in general, it is
particularly important for an assessment of literacy to evaluate nonresponse bias that is attributed
to a "language problem." In the ~ 992 assessment. the nonresnonse rates attributer! to a language
4_ ~ ~ ~ 1 =7 o
1 1 A ~ ~ 1` J 1 ~ ~ . 1 1 ~ ~ . · '
problem were (~.3 percent for the screener, (). / percent tor the background questionnaire, and 3.2
percent for the exercise booklet (composed of I.8 percent total incompletes ant! I.4 percent
partial incompletes) (Kirsch et al., 20014.
A sensitive nonresponse study could tease out additional information about reasons for
cropping out or refusing to participate, about the nature and extent of language problems that
prevent NAAL participation, and about the competency in English of those who could not
participate. It is quite likely that the reasons for cropping out of the NAAL assessment are
related to literacy proficiency or lack of English proficiency regardless of the stated reasons. For
instance, individuals may decline to participate or (lrop out during the first two stages simply
because they realize that the assessment is about literacy (in English) ant! believe that it is not
one of their strong points.
Focused follow-up studies could provide aclditional information about the English
language ant! literacy skills of those who dropped out during the early stages by making
adaptations to the stanciarct procedures, such as: orgy aciministenng the core questions (to obtain
an estimate of the percentage of nonresponclents who would have been routed to NAAL or
ALSA), arranging for a translator for speakers of languages other than English or Spanish, or
verifying the validity of other reasons for refusing to participate (such as problems with visual
acuity) or some combination of these adaptations. As part of the follow up studies, information
could be gathered on the language spoken in the home and the length of time individuals have
lived in the United States. An attempt should also be made to learn about indivicluals' literacy
skills in the native language.
We are not suggesting that NCES (levelop mechanisms for formally assessing literacy
skills in a multitude of languages, only that information be gathered that could further our
unclerstanding about the literacy skills of individuals who do not speak English. Information
from these follow-up studies could enhance understanding of the 2003 results, assist in
developing policy regarding remedial literacy and ESE programs for adults, ant} be user! in
designing future literacy assessments.
By offering the instructions and questions related to the core stimulus materials ant!
ALSA in Spanish, NCES has macle great strides in expancling the coverage of this assessment.
However, it is important to be clear about exactly who is included, who is exclucled, and reasons
for nonparticipation.
2 Response rates were calculated as follows: 89.1 percent responded to the screener; of those, 81.0 percent
responded to the background questionnaire; of those, 88.6 percent responded to the cognitive component itself.
6
OCR for page 7
RECOMMENDATION: The Department of Education should conduct a special
follow-up study of nonrespondents. The study should involve identifying a probability
sample of norlrespond~ents (perhaps stratif ed' by geographic location or other
information to identify types of nonrespondents), focusing particularly on individuals
who are eliminated during the f rst two stages. Data collected from this group shouic!
focus on reasons for not participating and should probe further into language issues by
ascertaining information about command of English and the likely [eve! of literacy
skills in the native language. Analysis of data from the follow-up studies should
examine systematic differences between respondents and nonrespor`dents that might
bias the results and should! identify issues to consider for future administrations of
HAAG.
We hope that you find our recommendations useful and informative, and we urge funding
for these three important studies. We look forward to working with you further on this project.
Sincerely,
Christopher F. Edley, Jr., Co-chair
U. ~ ..
,W':
Robert M. Hauser, Co-chalr
7
OCR for page 8
References
Beck, A., and Gilliard, D. (19954. Bureau of Justice Statistics Bulletin: Prisoners in 1994.
Washington, DC: US Department of Justice.
Cochran, William G. (1963~. Sampling Techniques (second edition). New York: Wiley.
Giliard, D., ant! Beck, A. (1994~. Bureau of Justice Statistics Bulletin: Prisoners in 1993.
Washington, DC: US Department of Justice.
Groves, Robert M., ant! Mick P. Couper (1998~. Nonresponse in Household interview Surveys. New
York: Wiley.
Harrison, P., and Beck, A. (20024. Bureau of Justice Statistics Bulletin:
Washington, DC: US Department of Justice.
Prisoners in 2001.
Kalton, G., and Kasprzyk, D. (19864. The treatment of missing survey data. Survey Methodology, 112,
I-16.
Kirsch, 1., Yamamoto, K., Norris, N., Rock, D., Jungeblut, A., O'Reilly, P., Berlin, M., Mohandjer,
L., Waksberg, J., Goksel, H., Burke, J., Rieger, S., Green, J., Klein, M., Campbell, A., Jenkins
L., Kolstad, A., Mosenthal, P., and Baldi, S. (2001~. Technical Report and Data Fzle User's
manualfor the 1992 National Aclult Literacy Survey (NCES 2001-457). Washington, DC: US
Department of Education, National Center for Education Statistics.
US Department of Education. (n.cI.~. Selecting ALSO Respondents. Washington, DC: Author.
Western, B., and Pettit, B. (20004. Incarceration anti racial inequality in men's employment.
Inclustrial and Labor Relations Review, 54:3-16.
OCR for page 9
COMMITTEE ON PERFORMANCE LEVELS FOR ADULT LITERACY
CHRISTOPHER F. EDLEY, Jr. (Co-Chair), Harvard Law School
ROBERT M. HAWSER (Co-Chair), Center for Demography of Health and Aging, University of
Wisconsin
JUDITH A. ALAMPRESE, Abt Associates Inc., Bethesda, MD
MICHAEL X. DELLl CARPINI, The Pew Charitable Trusts, Philaclelphia, PA
VIVIAN L. GADSDEN, Graduate School of Education, University of Pennsylvania
ANDREW J. HARTMAN, Bell Policy Center, Denver, CO
GLYNDA A. HULL, Graduate School of Education, University of California, Berkeley
REBECCA A. MAYNARD, Graduate School of Education, University of Pennsylvania
LORRAINE MCDONNELL, Department of Political Science, University of California, Santa
Barbara
LARRY MIKULECKY, Language Eclucation Department, Tncliana University
ROBERT J. MISLEVY, Department of Measurement, Statistics, and Evaluation, University of
Maryland
NORMAN G. PETERSON, Satisfaction Performance Research (SPR) Center, Inc., Minneapolis,
MN
JOHN P. POGGIO, Department of Psychology and Research in Eclucation, University of Kansas
RIMA RUDD, Department of Health ant} Social Behavior, Harvard School of Public Health
MARY JANE SCHMITT, TERC, Cambridge, MA
DAVID M. THISSEN, Department of Psychology, University of North Carolina, Chapel Hill
HElDE SPRUCK WRIGLEY, Aguirre International, HQ, San Mateo, CA
JUDITH A. KOENIG, Study Director
CONNIE CITRO, Senior Program Officer
STUART W. ELLIOTT, Senior Program Officer
ANDREW E. TOMPKINS, Research Assistant
MICHAEL DECARMINE, Project Assistant
Representative terms from entire chapter:
core questions
