National Academies Press: OpenBook

Measuring Literacy: Performance Levels for Adults (2005)

Chapter: Appendix B Examination of the Dimensionality of NALS

« Previous: Appendix A The Committee's Public Forums on Performance Levels for NAAL
Suggested Citation:"Appendix B Examination of the Dimensionality of NALS." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.
×
Page 214
Suggested Citation:"Appendix B Examination of the Dimensionality of NALS." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.
×
Page 215
Suggested Citation:"Appendix B Examination of the Dimensionality of NALS." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.
×
Page 216
Suggested Citation:"Appendix B Examination of the Dimensionality of NALS." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.
×
Page 217
Suggested Citation:"Appendix B Examination of the Dimensionality of NALS." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.
×
Page 218
Suggested Citation:"Appendix B Examination of the Dimensionality of NALS." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.
×
Page 219
Suggested Citation:"Appendix B Examination of the Dimensionality of NALS." National Research Council. 2005. Measuring Literacy: Performance Levels for Adults. Washington, DC: The National Academies Press. doi: 10.17226/11267.
×
Page 220

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Appendix B Examination of the Dimensionality of NALS CONFIRMATORY FACTOR ANALYSIS OF NALS The National Adult Literacy Survey (NALS) used a balanced incom- plete block (BIB) spiraling design for assigning items to test booklets and booklets to test takers, much like what is done for NAEP. There are 26 NALS test booklets. Each booklet contains 3 blocks of items, and test takers are given about 15 minutes per block. Each block of items appears in three different test booklets; each block appears with every other block at least once. In 1992, all test takers were given a set of six “core” questions to familiarize them with the examination and testing procedures. The core questions were relatively easy and consisted of two questions per literacy area (prose, document, and quantitative). NALS included some questions/tasks that had been developed for the 1985 Young Adult Literacy Survey (n = 85) and some newly developed questions/tasks (n = 81). Table B-1 shows the distribution of the tasks across the three literacy areas. For this analysis, six booklets were selected, and the responses of the household survey participants (n = 24,944) were studied. This analysis replicated some of the procedures that had been used for prior dimension- ality analyses (e.g., Rock and Yamamoto, 2001) but selected different blocks of items. Table B-2 shows the booklets and blocks of items included in the analysis along with the distribution of the tasks across the three literacy areas and the number of test takers who received each of the tasks. Using LISREL, six confirmatory factor analyses were run in which a 214

EXAMINATION OF THE DIMENSIONALITY OF NALS 215 TABLE B-1 Distribution of NALS Tasks Across the Three Literacy Areas Task Scale Prose Doc. Quant. Total Blocks 1992 New Tasks 27 26 28 81 7 1985 Old Tasks 14 56 15 85 6 Total 1992 Tasks 41 82 43 166 13 TABLE B-2 Booklets and Blocks of Items Included in the Exploratory Factor Analysis Booklet Prose Doc. Quant. # of (Blocks) N Tasks Tasks Tasks Tasks 1 (1, 2, 13) 957 11 22 10 43 6 (9, 7, 10) 895 12 16 12 40 8 (8, 6, 12) 920 11 25 10 46 12 (12, 5, 3) 855 11 23 8 42 15 (2, 4, 6) 925 8 32 11 51 17 (4, 9, 11) 929 11 11 17 39 three-factor model was specified. Analyses were run separately for each test booklet. Because the six core items evidenced limited variability, the analy- ses were repeated with these six items removed. Tables B-3 and B-4 present the results with the core items included and excluded, respectively. Correlations between the literacy scales were quite high. When the core items were included in the analyses, correlations between the prose and document scales ranged from .89 to .94 for the six booklets, from .77 to .97 for the document and quantitative scales, and from .80 to .97 for the prose and quantitative scales. When the core items were removed from the analy- ses, correlations between the prose and document scales ranged from .86 to .94 for the five booklets (data matrix for Booklet 1 was not positive defi- nite), from .75 to .95 for the document and quantitative scales, and from .79 to .97 for the prose and quantitative scales. Model fit was evaluated using the root mean square error of approxi- mation (RMSEA). Fit tended to decrease slightly when the core items were

216 APPENDIX B TABLE B-3 LISREL Results for a Three-Factor Model When the Six Core Tasks Were Included in the Analyses Intercorrelations Booklet RMSEA* Prose/Doc. Doc./Quant. Prose/Quant. 1 .06 .92 .85 .88 6 .07 .92 .98 .91 8 .08 .89 .85 .97 12 .05 .91 .97 .95 15 .08 .94 .77 .80 17 .08 .94 .89 .88 *The RMSEA provides an estimate of the fit of the model to the data. TABLE B-4 LISREL Results for a Three-Factor Model When the Six Core Tasks Were Excluded from the Analyses Intercorrelations Booklet RMSEA* Prose/Doc. Doc./Quant. Prose/Quant. 1 Did not converge 6 .08 .91 .95 .90 8 .10 .87 .81 .97 12 .05 .86 .94 .97 15 .10 .94 .75 .79 17 .08 .92 .89 .87 *The RMSEA provides an estimate of the fit of the model to the data. removed. These results suggest that a three-factor model provided accept- able fit to the data. EXAMINATION OF THE RELATIONSHIPS BETWEEN LITERACY SCORES AND SOCIAL AND ECONOMIC CHARACTERISTICS Another set of statistical analyses addressed questions about the rela- tionships between the prose, document, and quantitative scores and an array of literacy outcomes. First, were the dimensions of literacy associated differentially with social and economic characteristics? For example, was prose more highly associated with outcome x than with outcome y, while

EXAMINATION OF THE DIMENSIONALITY OF NALS 217 quantitative was more highly associated with outcome y than with outcome x? If so, there would be empirical support for use of each separate dimen- sion to guide adult education policy and the activities of adult educators. If not, either the assessments do not measure the dimensions independently, or there is little practical significance to the distinctions among them. Second, were some dimensions of literacy more highly related to the social and economic characteristics than others? For example, is prose the most important type of literacy, or are document and quantitative equally important? The answer to the second question is both simpler and more important if the answer to the first is that the dimensions of literacy are not associated differentially with their correlates. That is, if one weighted com- bination of the prose, document, and quantitative scores adequately de- scribes the relationship of measured literacy to the several possible corre- lates, the weights of prose, document, and quantitative become more instructive. These analyses were based upon the national and state household samples from the 1992 NALS.1 The total sample size was 25,987. The possible literacy correlates used in the analysis were similar to those used in the panel’s search for break points in the distributions of literacy scores (as described in Chapter 4): • years of school completed, • immigration within the last five years, • reporting at least one health impairment, • reporting a health problem that limits work, • reporting not reading well, • voting within the last five years, • being in the labor force, • weekly earnings (log), • reporting never reading newspaper, • reporting reading no books, • working in an occupation with high formal training requirements (professional, technical, managerial, nonretail sales), • working in an occupation with low formal training requirements (skilled worker, semi-skilled worker, labor, service work, farm work), • using Food Stamps within the past year, • having interest income in the past year, • reporting use of reading on the job, • reporting help needed with written material, and • reporting use of math on the job. 1Thus, the sample of incarcerated persons was not included.

218 APPENDIX B The prose, document, and quantitative scores used were the first set of plausible values in the public release of the data (http://www.nces.ed.gov/ naal/analysis/resources.asp). The analyses were based on a multiple-indicator, multiple-cause (MIMIC) model (Hauser and Goldberger 1971; Joreskog and Goldberger 1975), estimated by maximum likelihood. Rather than estimating separate regressions, one for each of the correlates of prose, document, and quanti- tative, the model posits that there is one linear composite of prose, docu- ment, and quantitative—much like the predicted values in a single regres- sion equation—and that the statistical effects of prose, document, and quantitative on the correlates are completely described by the relationships of the correlates with the composite variable. An equivalent way of describ- ing the model is that the statistical effects of prose, document, and quanti- tative on each of the correlates are in the same proportion. This model was estimated in the total household sample of NALS and in groups defined by race-ethnicity (black, Hispanic), gender, and age (16- 29, 20-44, 45-59, and 60 and older). The constrained model of the effects of prose, document, and quantitative on literacy correlates never fits statis- tically. This is to be expected because the sample is so large that any deviation from the model, no matter how trivial in substance, is statistically reliable. However, the actual deviations of the data from the constraints of the model are neither large nor numerous.2 Typical deviations from the model are that (a) using mathematics on the job is more highly correlated with quantitative literacy, (b) voting within the past five years is less highly correlated with document literacy, and (c) earnings are more highly corre- lated with quantitative literacy. Nevertheless the model provides a useful framework for assessing the relative importance of prose, document, and quantitative. As shown in summary in Table B-5, estimates from the constrained model are roughly similar across all of the groups. For fully constrained models, the left hand panel shows the effects of each of the dimensions of literacy. Prose, document, and quantitative are in the score metric, and the coefficients show effects on grouped levels of educational attainment.3 The right hand panel shows corresponding standardized coefficients. That is, 2For example, in the total sample, the model yields a likelihood ratio fit statistic of 2093.5 with 32 degrees of freedom, but the adjusted goodness of fit index (AGFI) is 0.949, a value that is commonly regarded as acceptable. 3The choice of educational attainment as the outcome variable is completely arbitrary. Any of the correlates could have been used because the effects of P, D, and Q on each outcome are in the same proportion in the constrained model.

TABLE B-5 Constrained Associations of Literacy Dimensions with Life Outcomes Test Score Coefficients Standardized Coefficients Population Prose Doc. Quant. Prose Doc. Quant. Total 6.907 1.378 4.766 0.532 0.107 0.394 (0.170) (0.179) (0.151) (0.013) (0.014) (0.012) Black 6.932 0.751 4.943 0.548 0.060 0.432 (0.167) (0.175) (0.139) (0.013) (0.014) (0.012) Hispanic 4.601 2.185 4.349 0.417 0.200 0.414 (0.117) (0.135) (0.116) (0.011) (0.012) (0.011) Female 7.404 1.386 4.131 0.574 0.110 0.347 (0.173) (0.176) (0.151) (0.013) (0.014) (0.013) Male 6.595 1.129 5.436 0.506 0.085 0.441 (0.171) (0.183) (0.153) (0.013) (0.014) (0.012) Ages 16-29 6.038 1.089 2.748 0.625 0.113 0.295 (0.148) (0.155) (0.132) (0.015) (0.016) (0.014) Ages 30-44 7.999 1.095 4.227 0.610 0.083 0.335 (0.172) (0.185) (0.156) (0.013) (0.014) (0.012) Ages 45-59 8.611 0.880 5.329 0.587 0.059 0.381 (0.187) (0.211) (0.173) (0.013) (0.014) (0.012) Ages 60-99 5.899 3.310 4.353 0.436 0.236 0.382 (0.151) (0.167) (0.128) (0.011) (0.012) (0.011) NOTE: Standard errors are in parentheses. 219

220 APPENDIX B the variables are all expressed in standard-deviation units. All three types of literacy have statistically significant associations with the correlates. The effect of document literacy is much less than that of prose or quantitative literacy and the effect of prose literacy is slightly larger than that of quanti- tative literacy. Model fit deteriorates markedly, however, if the effect of either document or quantitative literacy is ignored. Thus, while the panel notes the apparently prime importance of prose literacy, the other dimensions should not be ignored, and for some purposes it may be useful to construct a composite of the three literacy scores. It is not clear how to interpret the separate effects of the three literacy dimen- sions because they are so highly confounded by design in NALS and NAAL. That is, as long as the same task yields items scored on multiple dimensions, prose, document, and quantitative scores are intrinsically confounded. REFERENCES Hauser, R.M., and Goldberger, A.S, (1971). The treatment of unobservable variables in path analysis. In H.L. Costner (Ed.), Sociological methodology 1971 (pp. 81-117). San Fran- cisco: Jossey-Bass. Joreskog, KG., and Goldberger, A.S. (1975). Estimation of a model with multiple indicators and multiple causes of a single latent variable. Journal of the American Statistical Asso- ciation, 70, 631-639.

Next: Appendix C July 2004 Bookmark Standard-Setting Session with the 1992 NALS Data »
Measuring Literacy: Performance Levels for Adults Get This Book
×
Buy Paperback | $55.00 Buy Ebook | $43.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The National Assessment of Adult Literacy (NAAL) is a household survey conducted periodically by the Department of Education that evaluates the literacy skills of a sample of adults in the United Stages ages 16 and older. NAAL results are used to characterize adults' literacy skills and to inform policy and programmatic decisions. The Committee on Performance Levels for Adult Literacy was convened at the Department's request for assistance in determining a means for booking assessment results that would be useful and understandable for NAAL'S many varied audiences. Through a process detailed in the book, the committee determined that five performance level categories should be used to characterize adults' literacy skills: nonliterate in English, below basic literacy, basic literacy, intermediate literacy, and advanced literacy. This book documents the process the committee used to determine these performance categories, estimates the percentages of adults whose literacy skills fall into each category, recommends ways to communicate about adults' literacy skills based on NAAL, and makes suggestions for ways to improve future assessments of adult literacy.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!