Evaluation of the Achievement Levels
for Mathematics and Reading on the
National Assessment of Educational Progress
Committee on the Evaluation of NAEP Achievement Levels for
Mathematics and Reading
Christopher Edley, Jr., and Judith A. Koenig, Editors
Board on Testing and Assessment
Committee on National Statistics
Division of Behavioral and Social Sciences and Education
THE NATIONAL ACADEMIES PRESS
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
This activity was supported by Contract No. ED-IES-14-C-0124 from the U.S. Department of Education. Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.
International Standard Book Number-13: 978-0-309-43817-9
International Standard Book Number-10: 0-309-43817-9
Digital Object Identifier: 10.17226/23409
Additional copies of this publication are available for sale from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2017 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Academies of Sciences, Engineering, and Medicine. (2017). Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/23409.
The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president.
The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. C. D. Mote, Jr., is president.
The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.
The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.
Learn more about the National Academies of Sciences, Engineering, and Medicine at www.national-academies.org.
Reports document the evidence-based consensus of an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and committee deliberations. Reports are peer reviewed and are approved by the National Academies of Sciences, Engineering, and Medicine.
Proceedings chronicle the presentations and discussions at a workshop, symposium, or other convening event. The statements and opinions contained in proceedings are those of the participants and have not been endorsed by other participants, the planning committee, or the National Academies of Sciences, Engineering, and Medicine.
For information about other products and activities of the National Academies, please visit nationalacademies.org/whatwedo.
COMMITTEE ON THE EVALUATION OF NAEP ACHIEVEMENT LEVELS
CHRISTOPHER EDLEY, JR. (Chair), School of Law, University of California, Berkeley School of Law
PETER AFFLERBACH, Department of Teaching and Learning, Policy and Leadership, University of Maryland
SYBILLA BECKMANN, Department of Mathematics, University of Georgia
H. RUSSELL BERNARD, Institute for Social Science Research, Arizona State University, and Department of Anthropology, University of Florida
KARLA EGAN, National Center for the Improvement of Educational Assessment, Dover, NH
DAVID J. FRANCIS, Department of Psychology, University of Houston
MARGARET E. GOERTZ, Graduate School of Education, University of Pennsylvania (emerita)
LAURA HAMILTON, Education Program, RAND Corporation, Pittsburgh, PA
BRIAN JUNKER, Department of Statistics Carnegie Mellon University
SUZANNE LANE, School of Education, University of Pittsburgh
SHARON J. LEWIS, Council of the Great City Schools, Washington, DC (retired)
BERNARD L. MADISON, Department of Mathematics, University of Arkansas
SCOTT NORTON, Standards, Assessment, and Accountability, Council of Chief State School Officers, Washington, DC
SHARON VAUGHN, College of Education, University of Texas at Austin
LAURESS WISE, Human Resources Research Organization, Monterey, CA
JUDITH KOENIG, Study Director
JORDYN WHITE, Program Officer
NATALIE NIELSEN, Acting Board Director (until June 2015)
PATTY MORISON, Acting Board Director (from June 2015)
KELLY ARRINGTON, Senior Program Assistant
BOARD ON TESTING AND ASSESSMENT
DAVID J. FRANCIS (Chair), Texas Institute for Measurement, Evaluation, and Statistics, University of Houston
MARK DYNARSKI, Pemberton Research, LLC, East Windsor, NJ
JOAN HERMAN, National Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles
SHARON LEWIS, Council of Great City Schools, Washington, DC
BRIAN STECHER, Education Program, RAND Corporation, Santa Monica, CA
JOHN ROBERT WARREN, Department of Sociology, University of Minnesota
NATALIE NIELSEN, Acting Director (until June 2015)
PATTY MORISON, Acting Director (from June 2015)
COMMITTEE ON NATIONAL STATISTICS
LAWRENCE D. BROWN (Chair), Department of Statistics, Wharton School, University of Pennsylvania
JOHN M. ABOWD, School of Industrial and Labor Relations, Cornell University
FRANCINE BLAU, Department of Economics, Cornell University
MARY ELLEN BOCK, Department of Statistics, Purdue University
MICHAEL E. CHERNEW, Department of Health Care Policy, Harvard Medical School
DON A. DILLMAN, Department of Sociology, Washington State University
CONSTANTINE GATSONIS, Center for Statistical Sciences, Brown University
JAMES S. HOUSE, Survey Research Center, Institute for Social Research, University of Michigan
MICHAEL HOUT, Survey Research Center, University of California, Berkeley
THOMAS L. MESENBOURG, U.S. Census Bureau (retired)
SUSAN A. MURPHY, Department of Statistics, University of Michigan
SARAH M. NUSSER, Department of Statistics, Center for Survey Statistics and Methodology, Iowa State University
COLM A. O’MUIRCHEARTAIGH, Harris Graduate School of Public Policy Studies, University of Chicago
RUTH D. PETERSON, Criminal Justice Research Center, Ohio State University
ROBERTO RIGOBON, Sloan School of Management, Massachusetts Institute of Technology
EDWARD H. SHORTLIFFE, Biomedical Informatics, Columbia University and Arizona State University
CONSTANCE F. CITRO, Director
BRIAN HARRIS-KOJETIN, Deputy Director
Since 1969 the National Assessment of Educational Progress (NAEP) has been providing policy makers, educators, and the public with reports on the academic performance and progress of the nation’s students. Over the years, NAEP has been updated in ways that kept it current with the ever-changing educational and policy context, but maintained the trend lines it was designed to track. By the late 1980s, there were concerns about the extent to which U.S. students were learning what they need to be globally competitive in the 21st century. There were calls to set high, world-class achievement goals for U.S. students. In response, NAEP adopted standards-based reporting for the subjects and grades it assessed.
Today, some 24 years later, the focus on achievement standards is even more intense. At its best, standards-based reporting can provide a quick way to summarize students’ achievement and track their progress. It can clearly demark disparities between what we expect our students to know and be able to do as articulated in the standards—and what they actually know and can do. It can stimulate policy conversations about educational achievement across the country, identifying areas and groups with high performance as well as those with troubling disparities. It can inform policy interventions and reform measures to improve student learning.
There are potential downsides, however. Standards-based reporting can lead to erroneous interpretations. It can overstate or understate progress, particularly when the goal is to monitor the performance of subgroups. In its attempt to be easily understood by all audiences, it can lead to over-simplifications, such as when users do not do the necessary
work to ensure their understandings are correct. At a time when policy makers are vitally interested in ensuring that all of the country’s students achieve high standards, it is critical that test results are reported in a way that leads to accurate and valid interpretations, and that the standards used in reporting deserve the confidence of policy makers, practitioners, and the public.
The U.S. Department of Education turned to the National Academies of Sciences, Engineering, and Medicine to answer these questions, and the National Academies established the Committee on the Evaluation of NAEP Achievement Levels for Mathematics and Reading. The purpose of the project was to evaluate to what extent NAEP’s standards (or achievement levels) are reliable and valid. Are they reasonable? Are they informative to the public? Do they lead to appropriate interpretations? The committee of 15 brought a broad range of education and assessment experience. Our consensus findings, conclusions, and recommendations are documented in this report.
The committee benefited from the work of many others, and we wish to thank the many individuals who assisted us. We first thank the sponsor who supported this work: the U.S. Department of Education and the staff with the department’s Institute for Education Sciences who oversaw our work, Jonathon Jacobson and Audrey Pendleton.
During the course of its work, the committee met three times in person and four times by video conference. The committee’s first meeting included a public session designed to learn more about the sponsor’s goals for the project and the history of achievement-level setting for NAEP. We heard from officials from the National Assessment Governing Board (NAGB) and the National Center for Education Statistics (NCES). We especially thank Cornelia Orr, former executive director, and Sharyn Rosenberg, assistant director for psychometrics with NAGB, and Peggy Carr, acting commissioner of NCES, for the vast amount of historical information they provided.
The committee’s second meeting included a public forum designed to provide an opportunity for committee members to hear first-hand accounts of the variety of ways NAEP users interpret and use the achievement levels. These discussions were enormously enlightening to the committee, and we thank all who participated, including Patte Barth, Sonja Brookins Santelises, Sarah Butrymowicz, Michael Casserly, Enis Dogan, Wendy Geiger, Catherine Gewertz, Renee Jackson, Scott Jenkins, Mike Kane, Jacqueline King, Lyndsey Layton, Nathan Olson, Emily Richmond, Bob Rothman, Lorrie Shepard, and Dara Zeehandlaar. The agenda in Appendix A provides their titles and affiliations.
The committee gratefully acknowledges the dedicated effort provided by the staff of the Board on Testing and Assessment (BOTA) and the
Committee on National Statistics (CNSTAT) who worked directly on this project. We first thank Natalie Nielsen, former acting director of BOTA, for her extensive efforts to make this project a reality and her dedication to ensuring its success. We thank Patty Morison, current acting director of BOTA, for her support and guidance at key stages in this project. We are grateful to Connie Citro, director of CNSTAT, and Robert Hauser, executive director of the Division of Behavioral and Social Sciences and Education (DBASSE), for their sage advice throughout the course of this project. We thank Kelly Arrington, senior program assistant, for her exceptional organizational skills and her close attention to detail. Kelly handled all of the administrative details associated with the in-person and virtual meetings and the public forum, and she provided critical support in preparing the manuscript. We are grateful to Jordyn White, program officer with CNSTAT, for her adept research skills and her expertise in assimilating data and designing understandable presentations. We especially thank Judy Koenig for her intellectual and organizational skills as the study director and for her work in assembling the critical information needed to prepare this report. Judy worked tirelessly to keep us on task and to ensure that we met the multiple challenges presented by our charge.
We also thank members of the Office of Reports and Communication of DBASSE for their dedicated work on this report. We are indebted to Eugenia Grohman for her sage editorial advice on numerous versions of this manuscript. We thank Kirsten Sampson-Snyder for her work in coordinating a very intense review process and Yvonne Wise for shepherding the manuscript through production.
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the Report Review Committee of the National Academies. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.
We thank the following individuals for their review of this report: Lisa M. Abrams, Department of Foundations of Education, School of Education, Virginia Commonwealth University; Brian E. Clauser, Educational Assessment, National Board of Medical Examiners; Sarah W. Freedman, Graduate School of Education, University of California, Berkeley; Roger Howe, Department of Mathematics, Yale University; Nonie K. Lesaux, Graduate School of Education, Harvard University; Joseph A. Martineau, Center for Assessment, Dover, NH; William Penuel, School of Education, University of Colorado Boulder; Barbara Plake, Boros Center for Testing
(director emeritus), University of Nebraska–Lincoln; Mark D. Reckase, Department of Counseling, Educational Psychology and Special Education, Michigan State University; Lorrie A. Shepard, School of Education, University of Colorado Boulder; and Dale Whittington, Research and Accountability, Shaker Heights City School District, Shaker Heights, OH.
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations nor did they see the final draft of the report before its release. The review of this report was overseen by Edward H. Haertel, School of Education, Stanford University; and Ronald S. Brookmeyer, Department of Biostatistics, University of California, Los Angeles. Appointed by the National Academies, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.
As chair of the committee, I thank the extraordinary efforts of my fellow panel members, who had a broad range of expertise related to assessment, education policy, equity in education, mathematics and reading, program evaluation, social science, and statistics. All of this expertise was critical to the multifaceted issues that had to be addressed in this evaluation. The panel members freely contributed their time to accomplishing the myriad of tasks associated with assembling information and preparing this report. They actively assisted in all stages of this project, including planning the meetings and the public forum, as well as writing and rewriting multiple versions of this report. They gave generously of their time to ensure that the final product accurately represents our consensus findings, conclusions, and recommendations. Their contributions during the period in which the report was in final preparation and after the external review, when sections of the report had to be turned around on a very truncated schedule, are especially appreciated. These efforts manifested the panel members’ deep dedication to improving student learning across the country.
Christopher Edley, Jr., Chair
Committee on the Evaluation of NAEP
Achievement Levels for Mathematics and Reading
During the course of this project, the measurement community lost a towering figure. Robert Linn, distinguished professor emeritus of education with the University of Colorado Boulder, worked at the intersection of education measurement and education policy. He was known for his enormous contributions to test equating, fairness, validity, performance assessment, and educational accountability, and he was widely respected for his ability to translate theory into practical advice. His contributions to the National Assessment of Educational Progress (NAEP) spanned several decades, and he continued to serve on the NAEP Validity Studies Panel until his final months. His contributions to the National Academies of Sciences, Engineering, and Medicine were numerous, serving as a board member and chair of the Board on Testing and Assessment, and as a readily sought-after committee member and reviewer for many projects. A prodigious scholar and remarkable intellectual leader, Bob was also unceasingly patient and supportive as teacher, colleague, and friend. We are indebted to his contributions and will miss him. We dedicate this report to his memory.