MEASURING LITERACY

PERFORMANCE LEVELS FOR ADULTS

Committee on Performance Levels for Adult Literacy

Robert M. Hauser, Christopher F. Edley, Jr., Judith Anderson Koenig, and Stuart W. Elliott, editors

Board on Testing and Assessment

Center for Education

Division of Behavioral and Social Sciences and Education

NATIONAL RESEARCH COUNCIL OF THE NATIONAL ACADEMIES

THE NATIONAL ACADEMIES PRESS
Washington, D.C.
www.nap.edu



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
Measuring Literacy: Performance Levels for Adults MEASURING LITERACY PERFORMANCE LEVELS FOR ADULTS Committee on Performance Levels for Adult Literacy Robert M. Hauser, Christopher F. Edley, Jr., Judith Anderson Koenig, and Stuart W. Elliott, editors Board on Testing and Assessment Center for Education Division of Behavioral and Social Sciences and Education NATIONAL RESEARCH COUNCIL OF THE NATIONAL ACADEMIES THE NATIONAL ACADEMIES PRESS Washington, D.C. www.nap.edu

OCR for page R1
Measuring Literacy: Performance Levels for Adults THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This study/publication was supported by Cooperative Agreement No. R215U990016 between the National Academy of Sciences and the U.S. Department of Education. Additional funding was provided by an award from the Presidents’ Committee of The National Academies. Any opinions, findings, conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the organizations or agencies that provided support for the project. Library of Congress Cataloging-in-Publication Data Measuring literacy : performance levels for adults / Committee on Performance Levels for Adult Literacy ; Robert M. Hauser … [et al.], editors. p. cm. Includes bibliographical references. ISBN 0-309-09652-9 (pbk.)—ISBN 0-309-55015-7 (pdf) 1. Functional literacy—United States—Evaluation. I. Hauser, Robert Mason. II. National Research Council (U.S.). Committee on Performance Levels for Adult Literacy. LC149.7.M4 2005 302.2′244—dc22 2005021251 Additional copies of this report are available from the National Academies Press, 500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu. Printed in the United States of America Copyright 2005 by the National Academy of Sciences. All rights reserved. Suggested citation: National Research Council. (2005). Measuring literacy: Performance levels for adults. Committee on Performance Levels for Adult Literacy, R.M. Hauser, C.F. Edley, Jr., J.A Koenig, and S.W. Elliott, editors. Board on Testing and Assessment, Center for Education. Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

OCR for page R1
Measuring Literacy: Performance Levels for Adults THE NATIONAL ACADEMIES Advisers to the Nation on Science, Engineering, and Medicine The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National Research Council. www.national-academies.org

OCR for page R1
Measuring Literacy: Performance Levels for Adults This page intentionally left blank.

OCR for page R1
Measuring Literacy: Performance Levels for Adults COMMITTEE ON PERFORMANCE LEVELS FOR ADULT LITERACY CHRISTOPHER F. EDLEY, Jr. (Co-Chair), School of Law, University of California, Berkeley ROBERT M. HAUSER (Co-Chair), Center for Demography of Health and Aging, University of Wisconsin–Madison JUDITH A. ALAMPRESE, Abt Associates Inc., Bethesda, MD MICHAEL X. DELLI CARPINI, Annenberg School for Communication, University of Pennsylvania VIVIAN L. GADSDEN, National Center on Fathers and Families, University of Pennsylvania ANDREW J. HARTMAN, Independent Consultant, Denver, CO GLYNDA A. HULL, Graduate School of Education, University of California, Berkeley REBECCA A. MAYNARD, University Trustee Professor of Education and Social Policy, University of Pennsylvania LORRAINE M. McDONNELL, Department of Political Science, University of California, Santa Barbara LARRY MIKULECKY, Language Education Department, Indiana University ROBERT J. MISLEVY, Department of Measurement, Statistics, and Evaluation, University of Maryland NORMAN G. PETERSON, Satisfaction Performance Research (SPR) Center, Inc., Minneapolis, MN JOHN P. POGGIO, Department of Psychology and Research in Education, University of Kansas RIMA E. RUDD, Department of Health and Social Behavior, Harvard School of Public Health MARY JANE SCHMITT, TERC, Cambridge, MA DAVID M. THISSEN, Department of Psychology, University of North Carolina, Chapel Hill HEIDE SPRUCK WRIGLEY, LiteracyWork Associates, San Mateo, CA JUDITH A. KOENIG, Study Director STUART W. ELLIOTT, Senior Program Officer CONSTANCE F. CITRO, Senior Program Officer LORI HOUGHTON WRIGHT, Program Officer ANDREW E. TOMPKINS, Research Assistant LYNNE STEUERLE SCHOFIELD, Research Assistant MICHAEL DECARMINE, Senior Project Assistant

OCR for page R1
Measuring Literacy: Performance Levels for Adults BOARD ON TESTING AND ASSESSMENT 2003-2004 EVA L. BAKER (Chair), The Center for the Study of Evaluation, University of California, Los Angeles LORRAINE MCDONNELL (Vice-Chair), Department of Political Sciences, University of California, Santa Barbara, CA LAURESS L. WISE (Vice-Chair), Human Resources Research Organization, Alexandria, VA CHRISTOPHER F. EDLEY, JR., Harvard Law School EMERSON J. ELLIOTT, Consultant, Arlington, VA MILTON D. HAKEL, Department of Psychology, Bowling Green State University ROBERT M. HAUSER, NAS, Institute for Research on Poverty, Center for Demography, University of Wisconsin, Madison PAUL W. HOLLAND, Educational Testing Service, Princeton, NJ DANIEL M. KORETZ, Graduate School of Education, Harvard University EDWARD P. LAZEAR, Stanford University, Stanford, CA RICHARD J. LIGHT, Graduate School of Education and John F. Kennedy School of Government, Harvard University ROBERT J. MISLEVY, Department of Measurement and Statistics, University of Maryland JAMES W. PELLEGRINO, University of Illinois, Chicago, IL LORETTA A. SHEPARD, School of Education, University of Colorado, Boulder KENNETH I. WOLPIN, Department of Economics, University of Pennsylvania STUART W. ELLIOTT, Director LISA D. ALSTON, Administrative Coordinator

OCR for page R1
Measuring Literacy: Performance Levels for Adults Foreword In the summer of 2002, the National Center for Education Statistics approached the Board on Testing and Assessment (BOTA) of the National Academies with a request for assistance in setting performance standards for their upcoming assessment of adults’ literacy skills. This was a unique request for BOTA. Over the years, BOTA had explored and provided advice on a variety of issues related to setting performance standards on educational achievement tests—from discussions of the standards-based reform movement in education and its effects on various groups of student to recommendations for best practice in setting performance standards for the National Assessment of Educational Progress. Undertaking the process of actually setting performance standards, however, was a new endeavor for BOTA. Setting performance standards is an inherently judgmental task. The process involves determining the number and nature of the performance levels used for reporting the test results (such as “proficient” or “below basic”), the descriptions of the levels, and the test scores used to demark the range of scores associated with each performance level. A variety of standard-setting procedures are documented in the measurement literature, procedures that lay out the methodologies and best practices, but all ultimately rely on the judgments of testing experts, policy makers, and other stakeholders and users of the test results. The answers to questions such as “How much literacy is enough?” or “What constitutes a literacy problem?”—either for an individual or for society as a whole—are not exclusively technical. Bringing scientific principles to the process of collecting and summarizing judgmental information was a daunting challenge.

OCR for page R1
Measuring Literacy: Performance Levels for Adults Consistent with its mission, BOTA accepted this request. Formed to provide scientific advice to policy makers and the public about critical issues related to testing and assessment, BOTA draws on the interdisciplinary expertise of its members to bring a wide variety of perspectives to bear on such complex problems. Members of BOTA welcomed this opportunity to demonstrate a scientific approach to the problem and offer advice about two critical and timely issues: how to help describe and understand the literacy skill levels of adults in this country and how to set performance standards in a meaningful and technically valid way. Under the auspices of BOTA, the Committee on Performance Levels for Adult Literacy was formed as an interdisciplinary panel of 17 members with expertise in the areas of adult education and adult literacy, economics, educational measurement and standard setting, law, political science, and sociology. BOTA remained actively involved with this work, with four members of BOTA serving on the committee, two of whom served as committee co-chairs. Members of BOTA provided ongoing oversight for the work in formulating the committee’s charge and overall approach to its tasks, identifying individuals to serve on the committee, offering feedback to the committee, and reviewing the draft report and recommendations. The committee was convened in December 2002 and held six meetings. During the course of its work, the committee solicited feedback from stakeholders using a variety of mechanisms, including a public forum held in February 2004. The committee also convened two standard-setting sessions, in July and September 2004, which involved experts in adult literacy, adult education, teaching, and other relevant fields. This report presents the findings and recommendations that resulted from these activities and the committee’s deliberations. It is BOTA’s hope that this report will be of use to a variety of audiences: the U.S. Department of Education in its final decision making about performance standards for its adult literacy assessments and plans for future assessments; policy makers and practitioners in the adult literacy field as they make programmatic decisions; and the psychological measurement community as they grapple with the complex technical and judgmental issues involved in the task of setting valid performance standards in similar situations. BOTA extends its sincere appreciation to the committee for its hard work on this challenging project, and particularly to Christopher Edley, Jr., and Robert Hauser, who served as co-chairs. Lauress L. Wise, Chair Board on Testing and Assessment

OCR for page R1
Measuring Literacy: Performance Levels for Adults Acknowledgments The work of the Committee on Performance Levels for Adult Literacy benefited tremendously from the contributions of many people, and the committee is grateful for their assistance and support. First, we wish to acknowledge the National Center for Education Statistics (NCES), which sponsored this project. We think that the leadership of the NCES was wise, both to pursue improvements in standard setting and—as a statistical agency—to choose an impartial external body to establish its reporting standards. The committee thanks Gary Phillips for his willingness to initiate the study and extends its heartfelt thanks to Peggy Carr for her interest in this important topic and her constant support throughout the project. During the course of this project other NCES staff members, including Sheida White and Andrew Kolstad, gave generously of their time. We thank each of them for the wealth of information they provided and their prompt answers to all of the committee’s questions. The American Institutes for Research (AIR) served as contractor to NCES for work on National Assessment of Adult Literacy (NAAL), and many of its staff were generous with both advice and assistance. Mark Kutner was an invaluable resource to the committee, and we are grateful for his responsiveness to all of the committee’s requests. We also thank AIR staff members Stephan Baldi, Elizabeth Greenburg, and Eugene Johnson for their ongoing assistance and readiness to answer the committee’s questions. Special thanks are also due to Irwin Kirsch, who led the work on the 1992 National Adult Literacy Survey (NALS) at the Educational Testing Service. Irwin was a tremendous resource to the committee as they worked to reconstruct and understand procedures that had been used to determine

OCR for page R1
Measuring Literacy: Performance Levels for Adults the performance levels for the earlier literacy assessment. His fact-checking of report text that documented these processes was a great help to the committee. Early in its tenure, the committee commissioned a literature review of studies conducted on NALS. We thank M. Cecil Smith of Northern Illinois University for his thorough review of the literature, which provided an important part of the foundation for the committee’s work. The committee held an information-gathering meeting to learn about international assessments of adult literacy, and we are grateful to presenters at this meeting, including Mariann Lemke of the U.S. Department of Education; Scott Murray then of Statistics Canada; and Irwin Kirsch of Educational Testing Service. At the fourth meeting, the committee convened a public forum. The insights provided by the participants were very useful in helping the committee determine the performance levels. For this, we are grateful to Cynthia Baur, U.S. Department of Health and Human Services; Beth Beuhlmann, U.S. Chamber of Commerce, Workforce Preparation; Richard Colvin, Hechinger Institute; Leslie Farr, Ohio State University; Milton Goldberg, Education Commission of the States; Anne Lewis, freelance journalist; Richard Long, International Reading Association; Christopher Mazzeo, National Governors Association; Gemma Santos, Miami Dade Public Schools; Tony Sarmiento, Senior Service America, Inc.; Linda Taylor, Comprehensive Adult Student Assessment System; and Robert Wedgeworth, Proliteracy Worldwide. Representatives from five state departments of adult education also provided feedback about performance levels and about how NAAL results would be used in their states. We thank Bob Bickerton, Donna Cornelius, and Ann Serino, Massachusetts Department of Education; Steve Coffman, Missouri Department of Education; Cheryl King and Reecie Stagnolia, Kentucky Department of Education; Tom Orvino, New York Department of Education; and Linda Young, Oklahoma Department of Education. The committee is indebted to the individuals who assisted with the bookmark standard-setting sessions, held in July and September 2004. We particularly thank Richard J. Patz, of R.J. Patz, Inc., who led the standard-setting procedures. His expertise and guidance were key to the success of the standard settings. We are also grateful to Jeff Hauger and April Zenisky, University of Massachusetts, Amherst, and Andrew Poggio, University of Iowa, who assisted with managing the standard-setting sessions. Their assistance was key in making the sessions run smoothly. Special thanks are due to the many individuals who served as panelists for the bookmark standard-setting sessions. The committee truly appreciates their hard work and keen insights. The panelists included Eunice Askov,

OCR for page R1
Measuring Literacy: Performance Levels for Adults Pennsylvania State University; Marjorie Ball, Mississippi State Penitentiary; Roxanne Bauer, Indianapolis Public Schools; Michelle Blantz, South Georgia Technical College; Rhodella Brown, Daytona Beach Community College, Florida; Miriam Burt, Center for Applied Linguistics; Laura Chenven, AFL-CIO Working for America Institute; Suzanne Cimochowski, EASTCONN; Marie Cora, Hotspur Partners, LLC; Christopher Coro, Northampton Community College, Pennsylvania; Susan Cowles, Oregon State Department of Community Colleges and Workforce Development; Shari Crockett, Regional Office of Education, Illinois; Lansing Davis, New Jersey State Employment and Training Commission; Kim Donehower, University of North Dakota; Suzanne Elston, Bradley County Adult Education, Tennessee; Leslie Farr, Ohio State University; Sharon Floyd, Saginaw Public Schools; Janet Geary, North Kansas City School District; Karen Gianninoto, Salisbury State University; Kimberly Gibson, Sierra College; Suzanne Grant, Arlington Public Schools, Virginia; Anne Greenwell, Jefferson County Public Schools, Kentucky; Christina Gutierrez, T.C. Williams High School, Virginia; Nancy Hampson, San Diego Community College District; James Harris, Caliber Associates; Roberta Hawkins, Shorewood High School, Washington; Fran Holthaus, Upper Valley Joint Vocational School, Ohio; Sally House, Central Mississippi Correctional Facility; Brenda Jeans, Beauregard Parish School Board, Louisiana; Paul Jurmo, New York University; Judy Kihslinger, Waukesha County Technical College, Wisconsin; Terry Kinzel, Big Bend Community College, Washington; Jaqueline Korengel, Commonwealth of Kentucky; Nathan Kuncel, University of Illinois at Urbana-Champaign; Diane Lindahl, Western Wisconsin Technical College; Ardith Loustalet, St. Vrain Valley School District, Colorado; Alfredo Lujan, Monte del Sol Charter School, New Mexico; Sanford Marks, Community College of Southern Nevada; Peggy McGuire, University of Tennessee; Maureen Meehan, University of Illinois at Chicago; Doug Molitor, 3M; Donald Mott, Wilson Mott & Associates, North Carolina; Vivian Mott, East Carolina University; Bill Muth, U.S. Federal Bureau of Prisons; Connie Nelson, Massachusetts Worker Education Roundtable; Donna Nola-Ganey, Louisiana Department of Education; Peg Perri, Western Wisconsin Technical College; Rebecca Rogers, Washington University, St. Louis; Teresa Russell, independent consultant; Sally Sandy, Parkway School District, Missouri; Kathleen Santopietro Weddel, Colorado Department of Education; Diane Schroeder, St. Charles Community College, Missouri; Don Seaman, Texas Center for the Advancement of Literacy and Learning; Jane Siveria, Florida Department of Education; Cristine Smith, World Education, Inc.; Maggie Sokolik, University of California, Berkeley; Linda Stacy, Owens Community College, Ohio; Linda Taylor, Comprehensive Adult Student Assessment System; Ray Thompson,

OCR for page R1
Measuring Literacy: Performance Levels for Adults Middle Georgia Technical College; Patricia Thorpe, University of Phoenix; Fran Tracy-Mumford, Delaware Department of Education; Karen Valbrun, Georgia State Department of Technical and Adult Education; Denise Weiner, Delaware Department of Education; Lynne Weintraub, Jones Library; Ira Yankwitt, Literacy Assistance Center; and Linda Young, Oklahoma State Department of Education. Senior staff members of the National Research Council (NRC) helped the committee move this project forward. Michael Feuer enthusiastically backed the project and lent his wisdom and advice at key stages. Patricia Morison provided sage advice throughout this project and made valuable comments on several versions of the report. Eugenia Grohman’s knowledge and experience with NRC’s procedures and the committee process were of great assistance. We thank Christine McShane for her expert editing assistance and Yvonne Wise for her work in moving this report through the publication process. The committee is indebted to Kirsten Sampson Snyder for ably guiding the report through the NRC review process. Special thanks are due to Michael DeCarmine for his masterful handling of the logistical aspects of this project. In addition to handling the responsibilities associated with organizing committee meetings, Michael very capably managed the logistics of holding the July standard-setting session with 45 participants. He was also of great assistance to the committee by attending and reporting on his observations of the training sessions for NAAL survey administrators. We also thank Lisa Alston, who provided support throughout the project. We are grateful to Dorothy Majewski, who assumed responsibility for organizing the second standard-setting session. The committee also appreciates the assistance of Teresia Wilmore and Dionna Williams, who ably stepped in to assist at various stages of the project. Many other NRC staff contributed to the success of this project. We thank Connie Citro for her wealth of experience with standard setting in other contexts. Connie provided advice throughout the project. The committee sincerely appreciates the analytical assistance provided by Lynne Steuerle Schofield. Lynne’s statistical expertise and careful attention to detail contributed greatly to the substance and quality of this report. Lori Houghton Wright played a major role in organizing and managing the standard settings and contributed greatly to their overall success. We also appreciate Lori’s assistance in helping to produce this report. We thank Andrew Tompkins for his work in observing and reporting on the training procedures for NAAL interviewers and his shrewd research assistance. It has been most rewarding for us to work with our coeditors, Judith Koenig and Stuart Elliott. They kept everyone on track, drew wisely on various areas of expertise to create an effective division of labor within the committee, initiated and coordinated contacts with all of the parties to the

OCR for page R1
Measuring Literacy: Performance Levels for Adults project, and shouldered the major share of report preparation. We cannot imagine more professional and enjoyable colleagues. Above all, we thank the committee members for their dedication and outstanding contributions to this study. They drafted text, prepared background materials, reviewed numerous versions of this report, and gave generously of their time throughout the course of this three-year project. Both of us are novices in comparison to any of them in matters relating to adult education, adult literacy, and the measurement of literacy. Their varied expertise—and their patience with us—were essential to the report and to our growing appreciation of the importance and complexity of these matters. This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: Terry C. Davis, Department of Medicine and Pediatrics, Louisiana State University Health Sciences Center-Shreveport; Reynaldo F. Macias, Department of Chicana and Chicano Studies and the César E. Chávez Center for Interdisciplinary Instruction, University of California, Los Angeles; Mark D. Reckase, Departments of Counseling, Educational Psychology, and Special Education, Michigan State University; Stephen Reder, Department of Applied Linguistics, Portland State University; Loretta A. Shepard, School of Education, University of Colorado at Boulder; Sondra G. Stein, Policy Oversight, Equipped for the Future (EFF) Work Readiness Credential, Washington, DC; Sandy Strunk, Community Education, Lancaster Lebanon Intermediate Unit 13, East Petersburg, PA; Andrew Sum, Center for Labor Market Studies, Northeastern University; Daniel Wagner, National Center on Adult Literacy/International Literacy Institute University of Pennsylvania; Lauress (Laurie) Wise, President’s Office, Human Resources Research Organization (HumRRO), Alexandria, VA. Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations nor did they see the final draft of the report before its release. The review of this report was overseen by P. David Pearson, Graduate School of Education, University of California, Berkeley, and Stephen E. Fienberg, Department of Statistics, Carnegie Mellon University. Appointed

OCR for page R1
Measuring Literacy: Performance Levels for Adults by the NRC, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution. Christopher F. Edley, Jr., and Robert M. Hauser, Co-Chairs Committee on Performance Levels for Adult Literacy

OCR for page R1
Measuring Literacy: Performance Levels for Adults Contents     EXECUTIVE SUMMARY   1 1   INTRODUCTION   15      Problem Statement,   15      Overarching Goals,   18      Limitations on Inferences About Literacy Skills,   19      Committee’s Approach to the Charge,   21      Overview of the Report,   22 2   ADULT LITERACY ASSESSMENTS AND ADULT EDUCATION   23      Literacy Demands and the Need for Assessments,   23      Literacy Assessments,   25      Uses of Results of Adult Literacy Assessments,   31 3   DEVELOPING PERFORMANCE LEVELS FOR THE NATIONAL ADULT LITERACY SURVEY   50      Background on Developing Performance Levels,   50      Development of NALS Tasks,   52      Development of Performance-Level Descriptions and Cut Scores,   54      Choice of Response Probability Values,   69      Mapping Items to Performance Levels,   71      Conclusion,   72      Technical Note,   73

OCR for page R1
Measuring Literacy: Performance Levels for Adults 4   DETERMINING PERFORMANCE LEVELS FOR THE NATIONAL ASSESSMENT OF ADULT LITERACY   87      Stakeholder Views,   88      Relationships Between Literacy Scores and Background Characteristics,   92      Developing Policy-Relevant Performance Levels,   105 5   DEVELOPING PERFORMANCE-LEVEL DESCRIPTIONS AND SETTING CUT SCORES   108      The Bookmark Standard-Setting Method,   111      Bookmark Standard Setting with 1992 Data,   117      Bookmark Standard Setting with 2003 Data,   121      Results from the Standard-Setting Sessions,   130      Contrasting Groups Standard-Setting Method,   146      Adjusting the Bookmark Cut Scores,   159      Difficulties with the Upper and Lower Ends of the Score Scale,   161 6   COMMUNICATING AND USING THE RESULTS OF LITERACY ASSESSMENTS   167      Communicating Results,   167      Communication Strategies,   175      Examples of Ways NAAL Results May Be Used,   177 7   RECOMMENDATIONS FOR FUTURE LITERACY ASSESSMENTS   182      Demand-Side Analysis of Critical Skills,   183      Improving the Assessment of Quantitative Skills,   188      Improving the Information Collected About Adult Non-English Speakers,   191      Rethinking and Broadening the Definition of Literacy,   193      Conclusion,   195     REFERENCES   196     APPENDIXES     A   The Committee’s Public Forums on Performance Levels for NAAL   203 B   Examination of the Dimensionality of NALS   214 C   July 2004 Bookmark Standard-Setting Session with the 1992 NALS Data   221 D   September 2004 Bookmark Standard-Setting Session with the 2003 NAAL Data   285 E   Biographical Sketches of Committee Members and Staff   327