Keeping Score for All
The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments
Judith Anderson Koenig and Lyle F. Bachman, Editors
THE NATIONAL ACADEMIES PRESS
Washington, D.C. www.nap.edu
THE NATIONAL ACADEMIES PRESS
500 Fifth Street, N.W. Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This study was supported by Contract/Grant No. R215U990016 between the National Academy of Sciences and the U.S. Department of Education. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.
Library of Congress Cataloging-in-Publication Data
National Research Council (U.S.). Committee on Participation of English Language Learners and Students with Disabilities in NAEP and Other Large-Scale Assessments.
Keeping score for all : the effects of inclusion and accommodation policies on large-scale educational assessments / Committee on Participation of English Language Learners and Students with Disabilities in NAEP and Other Large-Scale Assessments ; Judith Anderson Koenig and Lyle F. Bachman, editors.
Includes bibliographical references.
ISBN 0-309-09253-1 (pbk.)—ISBN 0-309-53317-1 (pdf)
1. Educational tests and measurements—United States—Evaluation. 2. Inclusive education—United States—Evaluation. 3. Students with disabilities—Education—United States. 4. Limited English-proficient students—Education—United States. 5. National Assessment of Educational Progress (Project) I. Koenig, Judith A. II. Bachman, Lyle F., 1944- III. Title.
Additional copies of this report are available from the
National Academies Press,
500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu.
Printed in the United States of America
Copyright 2004 by the National Academy of Sciences. All rights reserved.
Suggested citation: National Research Council. (2004). Keeping Score for All: The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments. Committee on Participation of English Language Learners and Students with Disabilities in NAEP and Other Large-Scale Assessments. Judith A. Koenig and Lyle F. Bachman, Editors. Board on Testing and Assessment, Center for Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National Research Council.
COMMITTEE ON PARTICIPATION OF ENGLISH LANGUAGE LEARNERS AND STUDENTS WITH DISABILITIES IN NAEP AND OTHER LARGE-SCALE ASSESSMENTS
Lyle F. Bachman (Chair*),
Department of Applied Linguistics, University of California, Los Angeles
Jonathan G. Dings,
Boulder Valley School District, Boulder, CO
Judy L. Elliott,
Long Beach Unified School District, San Pedro, CA
Margaret J. McLaughlin,
Institute for the Study of Exceptional Children and Youth, University of Maryland, College Park
Mark D. Reckase (Chair**),
Measurement and Quantitative Methods, Michigan State University
Lourdes C. Rovira,
Miami-Dade County Public Schools, Miami, FL
María Medina Seidner,
Texas Education Agency, Austin, TX
Gevirtz Graduate School of Education, University of California, Santa Barbara
Judith A. Koenig, Study Director
Alexandra Beatty, Senior Program Officer
Michael DeCarmine, Senior Program Assistant
BOARD ON TESTING AND ASSESSMENT
Eva L. Baker (Chair),
Center for the Study of Evaluation, University of California, Los Angeles
Lorraine McDonnell (Vice Chair),
Department of Political Science, University of California, Santa Barbara
Lauress L. Wise (Vice Chair),
Human Resources Research Organization, Alexandria, VA
Christopher F. Edley, Jr.,
Harvard Law School
Emerson J. Elliott, Independent Consultant,
Milton D. Hakel,
Department of Psychology, Bowling Green State University, OH
Robert M. Hauser,
Institute for Research on Poverty, Center for Demography, University of Wisconsin–Madison
Paul W. Holland,
Educational Testing Service, Princeton, NJ
Daniel M. Koretz,
Graduate School of Education, Harvard University
Edward P. Lazear,
Graduate School of Business, Stanford University
Graduate School of Education, John F. Kennedy School of Government, Harvard University
Robert J. Mislevy,
Department of Measurement, Statistics, and Evaluation, University of Maryland, College Park
James W. Pellegrino,
Department of Psychology, University of Illinois, Chicago
Department of Education, University of Colorado, Boulder
Stuart W. Elliott, Director
Lisa D. Alston, Administrative Associate
The Committee on the Participation of Students with Disabilities and English Language Learners in NAEP and Other Large-Scale Assessments was formed under the auspices of the National Research Council (NRC) with the support of the National Center for Education Statistics (NCES). Its report addresses critical issues in the assessment of students with disabilities and English language learners, an issue that has come to the forefront of conversations about the effects of the No Child Left Behind Act of 2001. The NRC’s Board on Testing and Assessment (BOTA) has focused considerable attention during its ten-year history on the challenges and questions presented by the need to include these students in assessment and accountability programs. At present these students make up approximately 20 percent of the nation’s 46 million public school students, and while the responsibility for monitoring their progress is not new either to the National Assessment of Educational Progress (NAEP) or to states and districts, the provisions of the No Child Left Behind Act of 2001 have made that responsibility more public, more complex, and more urgent.
A November 2001 BOTA workshop, also sponsored by NCES, focused on the reporting and interpretation of test results for students with disabilities and English language learners who receive test accommodations. Discussions at the workshop made clear that several key issues merited more in-depth examination. The first set of concerns pertained to the way decisions regarding both the inclusion of students with special needs in assessments and the identification of appropriate accommodations for them are made. It was clear from the workshop not only that there is considerable variability across states and districts in the way these decisions are made but also that this variability can affect NAEP results in
significant ways. The second set of concerns pertained to the research that has been conducted on the effects of accommodations on test performance. The 2001 workshop stimulated considerable discussion about the conclusions that could be drawn from the existing literature base and about appropriate research approaches for evaluating the effects of accommodations on test performance.
Therefore, at the request of NCES, the Committee on the Participation of Students with Disabilities and English Language Learners in NAEP and Other Large-Scale Assessments was formed and given a three-part charge, to (1) synthesize research findings about the effects of accommodations on test performance, (2) review the procedures used for making inclusion and accommodation decisions for large-scale assessment programs, and (3) determine the implications of these findings for NAEP inclusions and accommodation policies. The work of this committee was intended to build on the discussions from the 2001 workshop (and other earlier NRC projects).
Thus, the report from the 2001 workshop served as the starting point for the committee’s work. In addition, researcher Stephen Sireci was commissioned to conduct a review and critique of the research literature on the effects of accommodations on test performance. This literature review focused on empirical studies conducted between 1990 and 2003 and was commissioned and completed in time to be ready for the committee’s first meeting in March 2003. The workshop report and this literature review served as background information for the committee as it began its in-depth examination of the relevant policies and practices in effect around the country and the state of the research in this vital area.
The resulting report is designed to set the committee’s findings and recommendations in the context of current policy and practice with regard to the inclusion and accommodation of students with disabilities and English language learners. In this report, the committee discusses the meaning of scores from accommodated assessments and the kinds of evidence that are needed to support inferences made from these scores. It is BOTA’s hope that this report will be of use to both the officials who oversee NAEP and those who oversee state and local assessments as they work to make their assessments as inclusive as possible, and to make them yield results that accurately reflect the knowledge and skills of all students.
The committee is very grateful to the many individuals who have helped with this project from its inception. It takes particular note of the contribution of Mark Reckase of Michigan State University, who served as its chair through most of its work, until his appointment to the National Assessment Governing Board required him to step down. The committee also sincerely appreciates Lyle Bachman’s willingness to assume the responsibility of chair for the completion of the project.
The committee extends its heartfelt thanks to Peggy Carr of NCES for her interest in this important topic and her willingness to fund the project. The committee also thanks Arnold Goldstein of NCES for his constant and prompt support
and quick answers to all of the committee’s questions. The committee also appreciates the assistance of Debra Hollinger Martinez, formerly of NCES, who provided valuable materials and information. Nancy Caldwell of Westat provided information about NAEP’s sampling procedures and responded to committee questions about NAEP’s administrative procedures for accommodating students with disabilities and English language learners. Jim Carlson of NAGB provided the committee with information on NAEP policies on accommodation as well as on research on the effects of accommodations on NAEP results. Mary Crovo of NAGB spoke with the committee about the constructs assessed by NAEP, and the committee is grateful to all of these busy officials for their time and assistance.
Many others assisted the committee as well. Researchers Martha Thurlow of the National Center on Education Outcomes and Charlene Rivera of the Center for Equity and Excellence provided the committee with background about states’ policies; John Olsen of the Council of Chief State School Officers (CCSSO) made a presentation about the results of the CCSSO’s annual survey of states’ practices. Presentations by Wayne Camara of the College Board and Robert Ziomek of ACT, Inc., on their research on the effects of accommodations on admissions test performance were particularly informative. The committee is grateful to Ed Haertel of Stanford University for providing an advance copy of his paper on evidentiary arguments and the comparability of scores from standard and nonstandard test administrations. Eric Hansen of Educational Testing Service briefed the committee on his work on the evidence-centered design approach and prepared a commissioned paper that was very helpful to the committee. The committee is also indebted to Stephen Sireci of the University of Massachusetts, Amherst and his colleagues Stanley Scarpati and Shuhong Li for their work on the commissioned review and critique of the literature.
The committee also wishes to thank the NRC staff who have supported the project. Study director Judith Koenig has offered leadership and support in countless ways and has guided the committee through some quite complicated territory. The committee also thanks Pasquale DeVito for his initial work on the project, and Stuart Elliott and Patricia Morison, who read and made valuable comments on several versions of the report. Michael DeCarmine’s calm and able administrative assistance throughout the project is much appreciated as well. The committee is also grateful for Alexandra Beatty’s expert writing ability. She provided invaluable assistance in drafting portions of the report and editing it so that it read with one voice. NRC editor Chris McShane provided valuable editing and smoothing in the final stages as well. Finally, the committee is indebted to Kirsten Sampson Snyder for ably guiding the report through the NRC review process.
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC’s RRC. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its
published report as sound as possible and ensuring that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. I thank the following individuals for their review of this report: Jamal Abedi, Technical Projects, Center for the Study of Evaluation, University of California, Los Angeles; Susan A. Agruso, Office of Assessment, Charlotte-Mecklenburg School District, Charlotte, NC; Richard Duran, Gevirtz Graduate School of Education, University of California, Santa Barbara; Shelley Loving-Ryder, Assistant Superintendent for Assessment and Reporting, Virginia Department of Education; Diana Pullin, Education Law and Public Policy, Boston College; and Martha Thurlow, National Center on Educational Outcomes, University of Minnesota, Minneapolis.
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by Lyle V. Jones, L.L. Thurstone Psychometric Laboratory, University of North Carolina. Appointed by the National Research Council, he was responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.
Eva L. Baker
Chair, Board on Testing and Assessment