Appendix B
Examination of the Dimensionality of NALS

CONFIRMATORY FACTOR ANALYSIS OF NALS

The National Adult Literacy Survey (NALS) used a balanced incomplete block (BIB) spiraling design for assigning items to test booklets and booklets to test takers, much like what is done for NAEP. There are 26 NALS test booklets. Each booklet contains 3 blocks of items, and test takers are given about 15 minutes per block. Each block of items appears in three different test booklets; each block appears with every other block at least once.

In 1992, all test takers were given a set of six “core” questions to familiarize them with the examination and testing procedures. The core questions were relatively easy and consisted of two questions per literacy area (prose, document, and quantitative).

NALS included some questions/tasks that had been developed for the 1985 Young Adult Literacy Survey (n = 85) and some newly developed questions/tasks (n = 81). Table B-1 shows the distribution of the tasks across the three literacy areas.

For this analysis, six booklets were selected, and the responses of the household survey participants (n = 24,944) were studied. This analysis replicated some of the procedures that had been used for prior dimensionality analyses (e.g., Rock and Yamamoto, 2001) but selected different blocks of items. Table B-2 shows the booklets and blocks of items included in the analysis along with the distribution of the tasks across the three literacy areas and the number of test takers who received each of the tasks.

Using LISREL, six confirmatory factor analyses were run in which a



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 214
Measuring Literacy: Performance Levels for Adults Appendix B Examination of the Dimensionality of NALS CONFIRMATORY FACTOR ANALYSIS OF NALS The National Adult Literacy Survey (NALS) used a balanced incomplete block (BIB) spiraling design for assigning items to test booklets and booklets to test takers, much like what is done for NAEP. There are 26 NALS test booklets. Each booklet contains 3 blocks of items, and test takers are given about 15 minutes per block. Each block of items appears in three different test booklets; each block appears with every other block at least once. In 1992, all test takers were given a set of six “core” questions to familiarize them with the examination and testing procedures. The core questions were relatively easy and consisted of two questions per literacy area (prose, document, and quantitative). NALS included some questions/tasks that had been developed for the 1985 Young Adult Literacy Survey (n = 85) and some newly developed questions/tasks (n = 81). Table B-1 shows the distribution of the tasks across the three literacy areas. For this analysis, six booklets were selected, and the responses of the household survey participants (n = 24,944) were studied. This analysis replicated some of the procedures that had been used for prior dimensionality analyses (e.g., Rock and Yamamoto, 2001) but selected different blocks of items. Table B-2 shows the booklets and blocks of items included in the analysis along with the distribution of the tasks across the three literacy areas and the number of test takers who received each of the tasks. Using LISREL, six confirmatory factor analyses were run in which a

OCR for page 214
Measuring Literacy: Performance Levels for Adults TABLE B-1 Distribution of NALS Tasks Across the Three Literacy Areas Scale Prose Doc. Quant. Total Task Blocks 1992 New Tasks 27 26 28 81 7 1985 Old Tasks 14 56 15 85 6 Total 1992 Tasks 41 82 43 166 13 TABLE B-2 Booklets and Blocks of Items Included in the Exploratory Factor Analysis Booklet (Blocks) N Prose Tasks Doc. Tasks Quant. Tasks # of Tasks 1 (1, 2, 13) 957 11 22 10 43 6 (9, 7, 10) 895 12 16 12 40 8 (8, 6, 12) 920 11 25 10 46 12 (12, 5, 3) 855 11 23 8 42 15 (2, 4, 6) 925 8 32 11 51 17 (4, 9, 11) 929 11 11 17 39 three-factor model was specified. Analyses were run separately for each test booklet. Because the six core items evidenced limited variability, the analyses were repeated with these six items removed. Tables B-3 and B-4 present the results with the core items included and excluded, respectively. Correlations between the literacy scales were quite high. When the core items were included in the analyses, correlations between the prose and document scales ranged from .89 to .94 for the six booklets, from .77 to .97 for the document and quantitative scales, and from .80 to .97 for the prose and quantitative scales. When the core items were removed from the analyses, correlations between the prose and document scales ranged from .86 to .94 for the five booklets (data matrix for Booklet 1 was not positive definite), from .75 to .95 for the document and quantitative scales, and from .79 to .97 for the prose and quantitative scales. Model fit was evaluated using the root mean square error of approximation (RMSEA). Fit tended to decrease slightly when the core items were

OCR for page 214
Measuring Literacy: Performance Levels for Adults TABLE B-3 LISREL Results for a Three-Factor Model When the Six Core Tasks Were Included in the Analyses Booklet RMSEA* Intercorrelations Prose/Doc. Doc./Quant. Prose/Quant. 1 .06 .92 .85 .88 6 .07 .92 .98 .91 8 .08 .89 .85 .97 12 .05 .91 .97 .95 15 .08 .94 .77 .80 17 .08 .94 .89 .88 *The RMSEA provides an estimate of the fit of the model to the data. TABLE B-4 LISREL Results for a Three-Factor Model When the Six Core Tasks Were Excluded from the Analyses Booklet RMSEA* Intercorrelations Prose/Doc. Doc./Quant. Prose/Quant. 1 Did not converge 6 .08 .91 .95 .90 8 .10 .87 .81 .97 12 .05 .86 .94 .97 15 .10 .94 .75 .79 17 .08 .92 .89 .87 *The RMSEA provides an estimate of the fit of the model to the data. removed. These results suggest that a three-factor model provided acceptable fit to the data. EXAMINATION OF THE RELATIONSHIPS BETWEEN LITERACY SCORES AND SOCIAL AND ECONOMIC CHARACTERISTICS Another set of statistical analyses addressed questions about the relationships between the prose, document, and quantitative scores and an array of literacy outcomes. First, were the dimensions of literacy associated differentially with social and economic characteristics? For example, was prose more highly associated with outcome x than with outcome y, while

OCR for page 214
Measuring Literacy: Performance Levels for Adults quantitative was more highly associated with outcome y than with outcome x? If so, there would be empirical support for use of each separate dimension to guide adult education policy and the activities of adult educators. If not, either the assessments do not measure the dimensions independently, or there is little practical significance to the distinctions among them. Second, were some dimensions of literacy more highly related to the social and economic characteristics than others? For example, is prose the most important type of literacy, or are document and quantitative equally important? The answer to the second question is both simpler and more important if the answer to the first is that the dimensions of literacy are not associated differentially with their correlates. That is, if one weighted combination of the prose, document, and quantitative scores adequately describes the relationship of measured literacy to the several possible correlates, the weights of prose, document, and quantitative become more instructive. These analyses were based upon the national and state household samples from the 1992 NALS.1 The total sample size was 25,987. The possible literacy correlates used in the analysis were similar to those used in the panel’s search for break points in the distributions of literacy scores (as described in Chapter 4): years of school completed, immigration within the last five years, reporting at least one health impairment, reporting a health problem that limits work, reporting not reading well, voting within the last five years, being in the labor force, weekly earnings (log), reporting never reading newspaper, reporting reading no books, working in an occupation with high formal training requirements (professional, technical, managerial, nonretail sales), working in an occupation with low formal training requirements (skilled worker, semi-skilled worker, labor, service work, farm work), using Food Stamps within the past year, having interest income in the past year, reporting use of reading on the job, reporting help needed with written material, and reporting use of math on the job. 1   Thus, the sample of incarcerated persons was not included.

OCR for page 214
Measuring Literacy: Performance Levels for Adults The prose, document, and quantitative scores used were the first set of plausible values in the public release of the data (http://www.nces.ed.gov/naal/analysis/resources.asp). The analyses were based on a multiple-indicator, multiple-cause (MIMIC) model (Hauser and Goldberger 1971; Joreskog and Goldberger 1975), estimated by maximum likelihood. Rather than estimating separate regressions, one for each of the correlates of prose, document, and quantitative, the model posits that there is one linear composite of prose, document, and quantitative—much like the predicted values in a single regression equation—and that the statistical effects of prose, document, and quantitative on the correlates are completely described by the relationships of the correlates with the composite variable. An equivalent way of describing the model is that the statistical effects of prose, document, and quantitative on each of the correlates are in the same proportion. This model was estimated in the total household sample of NALS and in groups defined by race-ethnicity (black, Hispanic), gender, and age (16-29, 20-44, 45-59, and 60 and older). The constrained model of the effects of prose, document, and quantitative on literacy correlates never fits statistically. This is to be expected because the sample is so large that any deviation from the model, no matter how trivial in substance, is statistically reliable. However, the actual deviations of the data from the constraints of the model are neither large nor numerous.2 Typical deviations from the model are that (a) using mathematics on the job is more highly correlated with quantitative literacy, (b) voting within the past five years is less highly correlated with document literacy, and (c) earnings are more highly correlated with quantitative literacy. Nevertheless the model provides a useful framework for assessing the relative importance of prose, document, and quantitative. As shown in summary in Table B-5, estimates from the constrained model are roughly similar across all of the groups. For fully constrained models, the left hand panel shows the effects of each of the dimensions of literacy. Prose, document, and quantitative are in the score metric, and the coefficients show effects on grouped levels of educational attainment.3 The right hand panel shows corresponding standardized coefficients. That is, 2   For example, in the total sample, the model yields a likelihood ratio fit statistic of 2093.5 with 32 degrees of freedom, but the adjusted goodness of fit index (AGFI) is 0.949, a value that is commonly regarded as acceptable. 3   The choice of educational attainment as the outcome variable is completely arbitrary. Any of the correlates could have been used because the effects of P, D, and Q on each outcome are in the same proportion in the constrained model.

OCR for page 214
Measuring Literacy: Performance Levels for Adults TABLE B-5 Constrained Associations of Literacy Dimensions with Life Outcomes Population Test Score Coefficients Standardized Coefficients Prose Doc. Quant. Prose Doc. Quant. Total 6.907 (0.170) 1.378 (0.179) 4.766 (0.151) 0.532 (0.013) 0.107 (0.014) 0.394 (0.012) Black 6.932 (0.167) 0.751 (0.175) 4.943 (0.139) 0.548 (0.013) 0.060 (0.014) 0.432 (0.012) Hispanic 4.601 (0.117) 2.185 (0.135) 4.349 (0.116) 0.417 (0.011) 0.200 (0.012) 0.414 (0.011) Female 7.404 (0.173) 1.386 (0.176) 4.131 (0.151) 0.574 (0.013) 0.110 (0.014) 0.347 (0.013) Male 6.595 (0.171) 1.129 (0.183) 5.436 (0.153) 0.506 (0.013) 0.085 (0.014) 0.441 (0.012) Ages 16-29 6.038 (0.148) 1.089 (0.155) 2.748 (0.132) 0.625 (0.015) 0.113 (0.016) 0.295 (0.014) Ages 30-44 7.999 (0.172) 1.095 (0.185) 4.227 (0.156) 0.610 (0.013) 0.083 (0.014) 0.335 (0.012) Ages 45-59 8.611 (0.187) 0.880 (0.211) 5.329 (0.173) 0.587 (0.013) 0.059 (0.014) 0.381 (0.012) Ages 60-99 5.899 (0.151) 3.310 (0.167) 4.353 (0.128) 0.436 (0.011) 0.236 (0.012) 0.382 (0.011) NOTE: Standard errors are in parentheses.

OCR for page 214
Measuring Literacy: Performance Levels for Adults the variables are all expressed in standard-deviation units. All three types of literacy have statistically significant associations with the correlates. The effect of document literacy is much less than that of prose or quantitative literacy and the effect of prose literacy is slightly larger than that of quantitative literacy. Model fit deteriorates markedly, however, if the effect of either document or quantitative literacy is ignored. Thus, while the panel notes the apparently prime importance of prose literacy, the other dimensions should not be ignored, and for some purposes it may be useful to construct a composite of the three literacy scores. It is not clear how to interpret the separate effects of the three literacy dimensions because they are so highly confounded by design in NALS and NAAL. That is, as long as the same task yields items scored on multiple dimensions, prose, document, and quantitative scores are intrinsically confounded. REFERENCES Hauser, R.M., and Goldberger, A.S, (1971). The treatment of unobservable variables in path analysis. In H.L. Costner (Ed.), Sociological methodology 1971 (pp. 81-117). San Francisco: Jossey-Bass. Joreskog, KG., and Goldberger, A.S. (1975). Estimation of a model with multiple indicators and multiple causes of a single latent variable. Journal of the American Statistical Association, 70, 631-639.