PERFORMANCE LEVELS FOR ADULTS
Robert M. Hauser, Christopher F. Edley, Jr., Judith Anderson Koenig, and Stuart W. Elliott, editors
THE NATIONAL ACADEMIES PRESS
THE NATIONAL ACADEMIES PRESS
500 Fifth Street, N.W. Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This study/publication was supported by Cooperative Agreement No. R215U990016 between the National Academy of Sciences and the U.S. Department of Education. Additional funding was provided by an award from the Presidents’ Committee of The National Academies. Any opinions, findings, conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the organizations or agencies that provided support for the project.
Library of Congress Cataloging-in-Publication Data
Measuring literacy : performance levels for adults / Committee on Performance Levels for Adult Literacy ; Robert M. Hauser … [et al.], editors.
Includes bibliographical references.
ISBN 0-309-09652-9 (pbk.)—ISBN 0-309-55015-7 (pdf) 1. Functional literacy—United States—Evaluation. I. Hauser, Robert Mason. II. National Research Council (U.S.). Committee on Performance Levels for Adult Literacy.
Additional copies of this report are available from the
National Academies Press,
500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu.
Printed in the United States of America
Copyright 2005 by the National Academy of Sciences. All rights reserved.
Suggested citation: National Research Council. (2005). Measuring literacy: Performance levels for adults. Committee on Performance Levels for Adult Literacy, R.M. Hauser, C.F. Edley, Jr., J.A Koenig, and S.W. Elliott, editors. Board on Testing and Assessment, Center for Education. Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National Research Council.
COMMITTEE ON PERFORMANCE LEVELS FOR ADULT LITERACY
CHRISTOPHER F. EDLEY, Jr. (Co-Chair),
School of Law, University of California, Berkeley
ROBERT M. HAUSER (Co-Chair),
Center for Demography of Health and Aging, University of Wisconsin–Madison
JUDITH A. ALAMPRESE,
Abt Associates Inc., Bethesda, MD
MICHAEL X. DELLI CARPINI,
Annenberg School for Communication, University of Pennsylvania
VIVIAN L. GADSDEN,
National Center on Fathers and Families, University of Pennsylvania
ANDREW J. HARTMAN,
Independent Consultant, Denver, CO
GLYNDA A. HULL,
Graduate School of Education, University of California, Berkeley
REBECCA A. MAYNARD, University Trustee Professor of Education and Social Policy,
University of Pennsylvania
LORRAINE M. McDONNELL,
Department of Political Science, University of California, Santa Barbara
Language Education Department, Indiana University
ROBERT J. MISLEVY,
Department of Measurement, Statistics, and Evaluation, University of Maryland
NORMAN G. PETERSON,
Satisfaction Performance Research (SPR) Center, Inc., Minneapolis, MN
JOHN P. POGGIO,
Department of Psychology and Research in Education, University of Kansas
RIMA E. RUDD,
Department of Health and Social Behavior, Harvard School of Public Health
MARY JANE SCHMITT,
TERC, Cambridge, MA
DAVID M. THISSEN,
Department of Psychology, University of North Carolina, Chapel Hill
HEIDE SPRUCK WRIGLEY,
LiteracyWork Associates, San Mateo, CA
JUDITH A. KOENIG, Study Director
STUART W. ELLIOTT, Senior Program Officer
CONSTANCE F. CITRO, Senior Program Officer
LORI HOUGHTON WRIGHT, Program Officer
ANDREW E. TOMPKINS, Research Assistant
LYNNE STEUERLE SCHOFIELD, Research Assistant
MICHAEL DECARMINE, Senior Project Assistant
BOARD ON TESTING AND ASSESSMENT 2003-2004
EVA L. BAKER (Chair),
The Center for the Study of Evaluation, University of California, Los Angeles
LORRAINE MCDONNELL (Vice-Chair),
Department of Political Sciences, University of California, Santa Barbara, CA
LAURESS L. WISE (Vice-Chair),
Human Resources Research Organization, Alexandria, VA
CHRISTOPHER F. EDLEY, JR.,
Harvard Law School
EMERSON J. ELLIOTT, Consultant,
MILTON D. HAKEL,
Department of Psychology, Bowling Green State University
ROBERT M. HAUSER,
NAS, Institute for Research on Poverty, Center for Demography, University of Wisconsin, Madison
PAUL W. HOLLAND,
Educational Testing Service, Princeton, NJ
DANIEL M. KORETZ,
Graduate School of Education, Harvard University
EDWARD P. LAZEAR,
Stanford University, Stanford, CA
RICHARD J. LIGHT,
Graduate School of Education and John F. Kennedy School of Government, Harvard University
ROBERT J. MISLEVY,
Department of Measurement and Statistics, University of Maryland
JAMES W. PELLEGRINO,
University of Illinois, Chicago, IL
LORETTA A. SHEPARD,
School of Education, University of Colorado, Boulder
KENNETH I. WOLPIN,
Department of Economics, University of Pennsylvania
STUART W. ELLIOTT, Director
LISA D. ALSTON, Administrative Coordinator
In the summer of 2002, the National Center for Education Statistics approached the Board on Testing and Assessment (BOTA) of the National Academies with a request for assistance in setting performance standards for their upcoming assessment of adults’ literacy skills. This was a unique request for BOTA. Over the years, BOTA had explored and provided advice on a variety of issues related to setting performance standards on educational achievement tests—from discussions of the standards-based reform movement in education and its effects on various groups of student to recommendations for best practice in setting performance standards for the National Assessment of Educational Progress. Undertaking the process of actually setting performance standards, however, was a new endeavor for BOTA.
Setting performance standards is an inherently judgmental task. The process involves determining the number and nature of the performance levels used for reporting the test results (such as “proficient” or “below basic”), the descriptions of the levels, and the test scores used to demark the range of scores associated with each performance level. A variety of standard-setting procedures are documented in the measurement literature, procedures that lay out the methodologies and best practices, but all ultimately rely on the judgments of testing experts, policy makers, and other stakeholders and users of the test results. The answers to questions such as “How much literacy is enough?” or “What constitutes a literacy problem?”—either for an individual or for society as a whole—are not exclusively technical. Bringing scientific principles to the process of collecting and summarizing judgmental information was a daunting challenge.
Consistent with its mission, BOTA accepted this request. Formed to provide scientific advice to policy makers and the public about critical issues related to testing and assessment, BOTA draws on the interdisciplinary expertise of its members to bring a wide variety of perspectives to bear on such complex problems. Members of BOTA welcomed this opportunity to demonstrate a scientific approach to the problem and offer advice about two critical and timely issues: how to help describe and understand the literacy skill levels of adults in this country and how to set performance standards in a meaningful and technically valid way.
Under the auspices of BOTA, the Committee on Performance Levels for Adult Literacy was formed as an interdisciplinary panel of 17 members with expertise in the areas of adult education and adult literacy, economics, educational measurement and standard setting, law, political science, and sociology. BOTA remained actively involved with this work, with four members of BOTA serving on the committee, two of whom served as committee co-chairs. Members of BOTA provided ongoing oversight for the work in formulating the committee’s charge and overall approach to its tasks, identifying individuals to serve on the committee, offering feedback to the committee, and reviewing the draft report and recommendations.
The committee was convened in December 2002 and held six meetings. During the course of its work, the committee solicited feedback from stakeholders using a variety of mechanisms, including a public forum held in February 2004. The committee also convened two standard-setting sessions, in July and September 2004, which involved experts in adult literacy, adult education, teaching, and other relevant fields. This report presents the findings and recommendations that resulted from these activities and the committee’s deliberations. It is BOTA’s hope that this report will be of use to a variety of audiences: the U.S. Department of Education in its final decision making about performance standards for its adult literacy assessments and plans for future assessments; policy makers and practitioners in the adult literacy field as they make programmatic decisions; and the psychological measurement community as they grapple with the complex technical and judgmental issues involved in the task of setting valid performance standards in similar situations.
BOTA extends its sincere appreciation to the committee for its hard work on this challenging project, and particularly to Christopher Edley, Jr., and Robert Hauser, who served as co-chairs.
Lauress L. Wise, Chair
Board on Testing and Assessment
The work of the Committee on Performance Levels for Adult Literacy benefited tremendously from the contributions of many people, and the committee is grateful for their assistance and support.
First, we wish to acknowledge the National Center for Education Statistics (NCES), which sponsored this project. We think that the leadership of the NCES was wise, both to pursue improvements in standard setting and—as a statistical agency—to choose an impartial external body to establish its reporting standards. The committee thanks Gary Phillips for his willingness to initiate the study and extends its heartfelt thanks to Peggy Carr for her interest in this important topic and her constant support throughout the project. During the course of this project other NCES staff members, including Sheida White and Andrew Kolstad, gave generously of their time. We thank each of them for the wealth of information they provided and their prompt answers to all of the committee’s questions.
The American Institutes for Research (AIR) served as contractor to NCES for work on National Assessment of Adult Literacy (NAAL), and many of its staff were generous with both advice and assistance. Mark Kutner was an invaluable resource to the committee, and we are grateful for his responsiveness to all of the committee’s requests. We also thank AIR staff members Stephan Baldi, Elizabeth Greenburg, and Eugene Johnson for their ongoing assistance and readiness to answer the committee’s questions.
Special thanks are also due to Irwin Kirsch, who led the work on the 1992 National Adult Literacy Survey (NALS) at the Educational Testing Service. Irwin was a tremendous resource to the committee as they worked to reconstruct and understand procedures that had been used to determine
the performance levels for the earlier literacy assessment. His fact-checking of report text that documented these processes was a great help to the committee.
Early in its tenure, the committee commissioned a literature review of studies conducted on NALS. We thank M. Cecil Smith of Northern Illinois University for his thorough review of the literature, which provided an important part of the foundation for the committee’s work.
The committee held an information-gathering meeting to learn about international assessments of adult literacy, and we are grateful to presenters at this meeting, including Mariann Lemke of the U.S. Department of Education; Scott Murray then of Statistics Canada; and Irwin Kirsch of Educational Testing Service.
At the fourth meeting, the committee convened a public forum. The insights provided by the participants were very useful in helping the committee determine the performance levels. For this, we are grateful to Cynthia Baur, U.S. Department of Health and Human Services; Beth Beuhlmann, U.S. Chamber of Commerce, Workforce Preparation; Richard Colvin, Hechinger Institute; Leslie Farr, Ohio State University; Milton Goldberg, Education Commission of the States; Anne Lewis, freelance journalist; Richard Long, International Reading Association; Christopher Mazzeo, National Governors Association; Gemma Santos, Miami Dade Public Schools; Tony Sarmiento, Senior Service America, Inc.; Linda Taylor, Comprehensive Adult Student Assessment System; and Robert Wedgeworth, Proliteracy Worldwide.
Representatives from five state departments of adult education also provided feedback about performance levels and about how NAAL results would be used in their states. We thank Bob Bickerton, Donna Cornelius, and Ann Serino, Massachusetts Department of Education; Steve Coffman, Missouri Department of Education; Cheryl King and Reecie Stagnolia, Kentucky Department of Education; Tom Orvino, New York Department of Education; and Linda Young, Oklahoma Department of Education.
The committee is indebted to the individuals who assisted with the bookmark standard-setting sessions, held in July and September 2004. We particularly thank Richard J. Patz, of R.J. Patz, Inc., who led the standard-setting procedures. His expertise and guidance were key to the success of the standard settings. We are also grateful to Jeff Hauger and April Zenisky, University of Massachusetts, Amherst, and Andrew Poggio, University of Iowa, who assisted with managing the standard-setting sessions. Their assistance was key in making the sessions run smoothly.
Special thanks are due to the many individuals who served as panelists for the bookmark standard-setting sessions. The committee truly appreciates their hard work and keen insights. The panelists included Eunice Askov,
Pennsylvania State University; Marjorie Ball, Mississippi State Penitentiary; Roxanne Bauer, Indianapolis Public Schools; Michelle Blantz, South Georgia Technical College; Rhodella Brown, Daytona Beach Community College, Florida; Miriam Burt, Center for Applied Linguistics; Laura Chenven, AFL-CIO Working for America Institute; Suzanne Cimochowski, EASTCONN; Marie Cora, Hotspur Partners, LLC; Christopher Coro, Northampton Community College, Pennsylvania; Susan Cowles, Oregon State Department of Community Colleges and Workforce Development; Shari Crockett, Regional Office of Education, Illinois; Lansing Davis, New Jersey State Employment and Training Commission; Kim Donehower, University of North Dakota; Suzanne Elston, Bradley County Adult Education, Tennessee; Leslie Farr, Ohio State University; Sharon Floyd, Saginaw Public Schools; Janet Geary, North Kansas City School District; Karen Gianninoto, Salisbury State University; Kimberly Gibson, Sierra College; Suzanne Grant, Arlington Public Schools, Virginia; Anne Greenwell, Jefferson County Public Schools, Kentucky; Christina Gutierrez, T.C. Williams High School, Virginia; Nancy Hampson, San Diego Community College District; James Harris, Caliber Associates; Roberta Hawkins, Shorewood High School, Washington; Fran Holthaus, Upper Valley Joint Vocational School, Ohio; Sally House, Central Mississippi Correctional Facility; Brenda Jeans, Beauregard Parish School Board, Louisiana; Paul Jurmo, New York University; Judy Kihslinger, Waukesha County Technical College, Wisconsin; Terry Kinzel, Big Bend Community College, Washington; Jaqueline Korengel, Commonwealth of Kentucky; Nathan Kuncel, University of Illinois at Urbana-Champaign; Diane Lindahl, Western Wisconsin Technical College; Ardith Loustalet, St. Vrain Valley School District, Colorado; Alfredo Lujan, Monte del Sol Charter School, New Mexico; Sanford Marks, Community College of Southern Nevada; Peggy McGuire, University of Tennessee; Maureen Meehan, University of Illinois at Chicago; Doug Molitor, 3M; Donald Mott, Wilson Mott & Associates, North Carolina; Vivian Mott, East Carolina University; Bill Muth, U.S. Federal Bureau of Prisons; Connie Nelson, Massachusetts Worker Education Roundtable; Donna Nola-Ganey, Louisiana Department of Education; Peg Perri, Western Wisconsin Technical College; Rebecca Rogers, Washington University, St. Louis; Teresa Russell, independent consultant; Sally Sandy, Parkway School District, Missouri; Kathleen Santopietro Weddel, Colorado Department of Education; Diane Schroeder, St. Charles Community College, Missouri; Don Seaman, Texas Center for the Advancement of Literacy and Learning; Jane Siveria, Florida Department of Education; Cristine Smith, World Education, Inc.; Maggie Sokolik, University of California, Berkeley; Linda Stacy, Owens Community College, Ohio; Linda Taylor, Comprehensive Adult Student Assessment System; Ray Thompson,
Middle Georgia Technical College; Patricia Thorpe, University of Phoenix; Fran Tracy-Mumford, Delaware Department of Education; Karen Valbrun, Georgia State Department of Technical and Adult Education; Denise Weiner, Delaware Department of Education; Lynne Weintraub, Jones Library; Ira Yankwitt, Literacy Assistance Center; and Linda Young, Oklahoma State Department of Education.
Senior staff members of the National Research Council (NRC) helped the committee move this project forward. Michael Feuer enthusiastically backed the project and lent his wisdom and advice at key stages. Patricia Morison provided sage advice throughout this project and made valuable comments on several versions of the report. Eugenia Grohman’s knowledge and experience with NRC’s procedures and the committee process were of great assistance. We thank Christine McShane for her expert editing assistance and Yvonne Wise for her work in moving this report through the publication process. The committee is indebted to Kirsten Sampson Snyder for ably guiding the report through the NRC review process.
Special thanks are due to Michael DeCarmine for his masterful handling of the logistical aspects of this project. In addition to handling the responsibilities associated with organizing committee meetings, Michael very capably managed the logistics of holding the July standard-setting session with 45 participants. He was also of great assistance to the committee by attending and reporting on his observations of the training sessions for NAAL survey administrators.
We also thank Lisa Alston, who provided support throughout the project. We are grateful to Dorothy Majewski, who assumed responsibility for organizing the second standard-setting session. The committee also appreciates the assistance of Teresia Wilmore and Dionna Williams, who ably stepped in to assist at various stages of the project.
Many other NRC staff contributed to the success of this project. We thank Connie Citro for her wealth of experience with standard setting in other contexts. Connie provided advice throughout the project. The committee sincerely appreciates the analytical assistance provided by Lynne Steuerle Schofield. Lynne’s statistical expertise and careful attention to detail contributed greatly to the substance and quality of this report. Lori Houghton Wright played a major role in organizing and managing the standard settings and contributed greatly to their overall success. We also appreciate Lori’s assistance in helping to produce this report. We thank Andrew Tompkins for his work in observing and reporting on the training procedures for NAAL interviewers and his shrewd research assistance.
It has been most rewarding for us to work with our coeditors, Judith Koenig and Stuart Elliott. They kept everyone on track, drew wisely on various areas of expertise to create an effective division of labor within the committee, initiated and coordinated contacts with all of the parties to the
project, and shouldered the major share of report preparation. We cannot imagine more professional and enjoyable colleagues.
Above all, we thank the committee members for their dedication and outstanding contributions to this study. They drafted text, prepared background materials, reviewed numerous versions of this report, and gave generously of their time throughout the course of this three-year project. Both of us are novices in comparison to any of them in matters relating to adult education, adult literacy, and the measurement of literacy. Their varied expertise—and their patience with us—were essential to the report and to our growing appreciation of the importance and complexity of these matters.
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.
We wish to thank the following individuals for their review of this report: Terry C. Davis, Department of Medicine and Pediatrics, Louisiana State University Health Sciences Center-Shreveport; Reynaldo F. Macias, Department of Chicana and Chicano Studies and the César E. Chávez Center for Interdisciplinary Instruction, University of California, Los Angeles; Mark D. Reckase, Departments of Counseling, Educational Psychology, and Special Education, Michigan State University; Stephen Reder, Department of Applied Linguistics, Portland State University; Loretta A. Shepard, School of Education, University of Colorado at Boulder; Sondra G. Stein, Policy Oversight, Equipped for the Future (EFF) Work Readiness Credential, Washington, DC; Sandy Strunk, Community Education, Lancaster Lebanon Intermediate Unit 13, East Petersburg, PA; Andrew Sum, Center for Labor Market Studies, Northeastern University; Daniel Wagner, National Center on Adult Literacy/International Literacy Institute University of Pennsylvania; Lauress (Laurie) Wise, President’s Office, Human Resources Research Organization (HumRRO), Alexandria, VA.
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations nor did they see the final draft of the report before its release. The review of this report was overseen by P. David Pearson, Graduate School of Education, University of California, Berkeley, and Stephen E. Fienberg, Department of Statistics, Carnegie Mellon University. Appointed
by the NRC, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.
Christopher F. Edley, Jr., and Robert M. Hauser, Co-Chairs
Committee on Performance Levels for Adult Literacy