GRADING THE NATION'S REPORT CARD

Evaluating NAEP and Transforming the Assessment of Educational Progress

James W. Pellegrino, Lee R. Jones, Karen J. Mitchell, editors

Committee on the Evaluation of National and State Assessments of Educational Progress

Board on Testing and Assessment

Commission on Behavioral and Social Sciences and Education

National Research Council

NATIONAL ACADEMY PRESS
Washington, D.C.
1999



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress GRADING THE NATION'S REPORT CARD Evaluating NAEP and Transforming the Assessment of Educational Progress James W. Pellegrino, Lee R. Jones, Karen J. Mitchell, editors Committee on the Evaluation of National and State Assessments of Educational Progress Board on Testing and Assessment Commission on Behavioral and Social Sciences and Education National Research Council NATIONAL ACADEMY PRESS Washington, D.C. 1999

OCR for page R1
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress NATIONAL ACADEMY PRESS 2101 Constitution Avenue, N.W. Washington, D.C. 20418 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. The study was supported by Award No. EA95083001 between the National Academy of Sciences and the U.S. Department of Education. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the organizations or agencies that provided support for this project. Library of Congress Cataloging-in-Publication Data Grading the nation's report card : evaluating NAEP and transforming the assessment of educational progress / James W. Pellegrino, Lee R. Jones, and Karen J. Mitchell, editors ; Committee on the Evaluation of National and State Assessments of Educational Progress, Board on Testing and Assessment, Commission on Behavioral and Social Sciences and Education, National Research Council. p. cm. Includes bibliographical references and index. ISBN 0-309-06285-3 (cloth) 1. National Assessment of Education Progress (Project) 2. Education —United States—Evaluation. 3. Educational tests and measurements—United States. I. Pellegrino, James W. II. Jones, Lee R. III. Mitchell, Karen Janice. IV. National Research Council (U.S.). Committee on Evaluation of National and State Assessments of Education Progress. LB3051 .G66686 1998 370′.973—dc21 98-40150 Additional copies of this report are available from: National Academy Press 2101 Constitution Avenue N.W. Washington, D.C. 20418 Call 800-624-6242 or 202-334-3313 (in the Washington Metropolitan Area). This report is also available on line at http://www.nap.edu Printed in the United States of America Copyright 1999 by the National Academy of Sciences. All rights reserved.

OCR for page R1
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress COMMITTEE ON THE EVALUATION OF NATIONAL AND STATE ASSESSMENTS OF EDUCATIONAL PROGRESS JAMES W. PELLEGRINO (Chair), Peabody College of Education and Human Development, Vanderbilt University GAIL P. BAXTER, College of Education, University of Michigan NORMAN M. BRADBURN, National Opinion Research Center, University of Chicago THOMAS P. CARPENTER, Wisconsin Center for Educational Research, University of Wisconsin-Madison ALLAN COLLINS, Bolt Beranek and Newman, Inc., Cambridge, MA PASQUALE J. DEVITO, Rhode Island Department of Education, Providence STEPHEN B. DUNBAR, College of Education, University of Iowa THOMAS H. FISHER,* Department of Education, State of Florida, Tallahassee LARRY V. HEDGES, Department of Education, University of Chicago ROBERT J. KANSKY,** Department of Teacher Education, Miami University, Oxford, OH SHARON LEWIS, Council of the Great City Schools, Washington, DC RODERICK J.A. LITTLE, Department of Biostatistics, University of Michigan ELSIE G.J. MOORE, College of Education, Arizona State University NAMBURY S. RAJU, Institute of Psychology, Illinois Institute of Technology MARLENE SCARDAMALIA, CACS/Ontario Institute for Studies in Education, University of Toronto GUADALUPE VALDÉS, School of Education, Stanford University SHEILA W. VALENCIA, College of Education, University of Washington LAURESS L. WISE, Human Resources Research Organization, Alexandria, VA LEE R. JONES, Study Director KAREN J. MITCHELL, Senior Program Officer HOLLY WELLS, Senior Project Assistant * Member until October 1996. ** Member until June 1996.

OCR for page R1
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress BOARD ON TESTING AND ASSESSMENT ROBERT L. LINN (Chair), School of Education, University of Colorado CARL F. KAESTLE (Vice Chair), Department of Education, Brown University RICHARD C. ATKINSON, President, University of California IRALINE G. BARNES, Washington, DC PAUL J. BLACK, School of Education, King's College, London RICHARD P. DURÁN, Graduate School of Education, University of California, Santa Barbara CHRISTOPHER F. EDLEY, JR., Harvard Law School PAUL W. HOLLAND, Graduate School of Education, University of California, Berkeley MICHAEL W. KIRST, School of Education, Stanford University ALAN M. LESGOLD, Learning Research and Development Center, University of Pittsburgh LORRAINE M. McDONNELL, Department of Political Sciences, University of California, Santa Barbara KENNETH PEARLMAN, Lucent Technologies, Inc., Warren, NJ PAUL R. SACKETT, Industrial Relations Center, University of Minnesota RICHARD J. SHAVELSON, School of Education, Stanford University CATHERINE E. SNOW, Graduate School of Education, Harvard University WILLIAM L. TAYLOR, Attorney at Law, Washington, DC WILLIAM T. TRENT, Office of the Chancellor, University of Illinois JACK WHALEN, Xerox Palo Alto Research Center, Palo Alto, CA KENNETH I. WOLPIN, Department of Economics, University of Pennsylvania MICHAEL J. FEUER, Director VIOLA HOREK, Administrative Associate

OCR for page R1
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress Acknowledgments The work of the Committee on the Evaluation of National and State Assessments of Educational Progress benefited tremendously from the contributions and good will of many people. Staff from the National Assessment Governing Board (NAGB), under the leadership of Roy Truby, executive director, and their subcontractor, American College Testing, Inc. (ACT), and staff from the National Center for Education Statistics (NCES), under the leadership of Pascal Forgione, commissioner of education statistics, and their subcontractors, Educational Testing Service (ETS) and Westat, Inc. were a valuable source of information and data on the National Assessment of Educational Progress (NAEP) throughout the project. Susan Loomis of ACT, Nancy Caldwell of Westat, and James Carlson, Stephen Lazer, John Mazzeo, and Christine O'Sullivan of ETS provided the committee with important information on occasions that are too numerous to mention. The committee especially extends thanks to Peggy Carr and Patricia Dabbs of NCES and Mary Lyn Bourque and Raymond Fields of the NAGB staff. In their roles as NCES and NAGB liaisons to the committee, they provided important information and perspectives throughout the course of its work. Committee members and project staff benefited tremendously by attending and learning from discussions at meetings of the National Assessment Governing Board and its committees, the Technical Advisory Committee on Standard-Setting, the NAEP Design and Analysis Committee, the NAEP Subject Area Standing Committees, and the Advisory Council on Education Statistics. We thank all

OCR for page R1
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress of the committee members and staff for opening their meetings to us and for sharing their knowledge and perspectives. The Office of Planning and Evaluation Services, U.S. Department of Education, administered the contract for this evaluation. Director Allen Ginsburg provided assistance in planning the evaluation, and Elois Scott, Collette Roney, and Audrey Pendleton each served as the contracting office's technical representative during various phases of the evaluation. The committee thanks them for their advice and assistance as they monitored its work. Between March 1996 and April 1998, the committee met nine times. At its December 1996 meeting, the committee held a workshop on standard-setting models and their applications to the NAEP achievement-level-setting process. Each of the following individuals made a helpful and insightful presentation at the workshop and prepared a written paper, the collection of which was published in a recent issue of Applied Measurement in Education (volume 11, number 1, 1998): Jeanne Goldberg (Tufts University), Lawrence Hanser (RAND), Sheila Jasanoff (Cornell University), Robert Linn (University of Colorado), Robert Mislevy (ETS), Barbara Plake (University of Nebraska), and Mark Reckase (ACT, Inc., now of Michigan State University). At its May 1997 meeting, the committee held a workshop to examine NAEP's mission, measurement objectives, and possible reconceptualization. The following individuals made presentations and provided valuable insights to inform the committee's deliberations on these complex issues: George Bohrnstedt (American Institutes for Research), Robert Boruch (University of Pennsylvania), Christopher Cross (Council for Basic Education), John Dossey (Illinois State University), Emerson Elliott (National Council for Accreditation of Teacher Education), Raymond Fields (NAGB), Robert Glaser (University of Pittsburgh), Herbert Ginsburg (Columbia University), James Greeno (Stanford University), Eugene Johnson (ETS), Daniel Koretz (RAND, now of Boston College), Alan Lesgold (University of Pittsburgh), James McBride (Human Resources Research Organization, Inc.), Robert Meyer (University of Chicago), Lawrence Mikulecky (Indiana University), Robert Mislevy (ETS), William Morrill (Mathtech, Inc.), Lois Peak (U.S. Department of Education), Andrew Porter (University of Wisconsin), Lauren Resnick (University of Pittsburgh), Lawrence Rudner (Educational Resources Information Center, University of Maryland), Richard Snow (Stanford University), David Thissen (University of North Carolina), Margaret Vickers (TERC), and Wendy Yen (CTB/McGraw Hill). Early in the committee's work, Stephen Sireci (University of Massachusetts) wrote a paper for the committee synthesizing issues regarding the dimensionality of the NAEP assessments, and NRC consultant Joanne Capner provided analyses of NAEP's current assessment development and reporting strategies. The committee also commissioned new research and syntheses on several key topics to assist them in their evaluation. The following individuals contributed their time, energy, enthusiasm, and intellect to these efforts, and their work is published in

OCR for page R1
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress the volume of research papers that accompanies this report: Sheila Barron (RAND); Robert Boruch (University of Pennsylvania) and George Terhanian (Harris Black International); Patricia Kenney (University of Pittsburgh); Michael Kolen (University of Iowa); James Minstrell (Assessment, Curriculum, and Teaching Systems for Education); Stephen Sireci, Kevin Meara, Frederic Robin, and Hariharan Swaminathan (University of Massachusetts) and H. Jane Rogers (Columbia University); James Stigler (UCLA) and Michelle Perry (University of Illinois); and Jennifer Zieleskiewicz (Illinois Institute of Technology). In November 1997, the committee convened a group of experts to discuss and explore the applications of contemporary cognitive and curricular research and theory and instructional practice to assessment development in NAEP. Consultants who shared their perspectives at this meeting included: David Pearson (Michigan State University), James Minstrell, Paul Nichols (University of Wisconsin), Leona Schauble (University of Wisconsin), Alan Schoenfeld (University of California, Berkeley), Patricia Kenney (University of Pittsburgh), Brenda Sugrue (University of Iowa), and Karen Wixson (University of Michigan). The committee's consideration of this topic was greatly enriched by the stimulating intellectual exchange at this meeting and in subsequent interactions with the attendees. The Board on Testing and Assessment (BOTA) provided especially valuable guidance and feedback at critical stages of the committee's deliberations. The chair of BOTA, Robert Linn, assisted the committee by participating in discussions at several committee meetings and by reviewing and commenting on a draft of the final report. BOTA's intellectual contributions to the committee's work are much appreciated. Many individuals at the National Research Council (NRC) provided guidance and assistance at many stages of the evaluation and during the preparation of the report. Executive director of the Commission on Behavioral and Social Sciences and Education (CBASSE) Barbara Torrey provided overall administration of the evaluation and shared her contagious enthusiasm for the work of the NRC with the committee. Alexandra Wigdor, director of the Division of Education, Labor, and Human Performance, provided continuing guidance and was especially central to the preparation of the committee's interim letter report on NAGB's proposed redesign of NAEP. Michael Feuer, director of the Board on Testing and Assessment, provided oversight of the committee's work and made frequent important contributions to committee discussions, providing unique and insightful perspectives and helping the committee maintain focus and achieve consensus. We also thank Eugenia Grohman, associate director for reports (CBASSE), for her advice on structuring the content of the report and for guiding the report through the NRC review process, and Christine McShane for her expert editing of the report manuscript and advice on the exposition of the report's main messages. The committee especially expresses gratitude to the NRC project staff for

OCR for page R1
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress their intellectual and organizational skills throughout this evaluation. Jacques Normand and Susan McCutchen served as the study director and project assistant during the early phases of the committee's work. They were succeeded by senior program officer Karen Mitchell, study director Lee Jones, and project assistant Holly Wells. Karen and Lee tirelessly assisted the committee in many ways—serving as valuable sources of information about NAEP, organizing and synthesizing the committee's work, keeping the committee moving forward through its deliberations and the report drafting process, and providing energy, enthusiasm, and exceptional good humor along the way. Holly Wells capably and admirably managed the operational aspects of the evaluation—arranging meeting and workshop logistics, producing multiple iterations of drafts of committee writings and report text, and being available at all times to assist with committee requests, however large or small. The committee is deeply indebted to Holly for her commitment to the committee's work, her dedication to meeting the committee's many needs for information and service, her problem solving skills, and her affability in all circumstances. This report has been reviewed by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC's Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the authors and the NRC in making the published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The content of the review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals, who are neither officials nor employees of the NRC, for their participation in the review of this report: Lizanne DeStefano, School of Education, University of Illinois, Champaign-Urbana; Emerson J. Elliott, National Council for the Accreditation for Teacher Education, Washington, D.C.; Susan Fuhrman, Graduate School of Education, University of Pennsylvania; Eric Hanushek, Wallis Institute of Political Economy, University of Rochester; Lyle V. Jones, L.L. Thurstone Psychometric Laboratory, University of North Carolina, Chapel Hill; Carl K. Kaestle, Department of Education, Brown University; P. David Pearson, Department of Education, Michigan State University; Gloria M. Rogers, Office of Institutional Research and Assessment, Rose-Hulman Institute of Technology; Bruce D. Spencer, Department of Statistics, Northwestern University; David M. Thissen, Department of Psychology, University of North Carolina, Chapel Hill; and Linda F. Wightman, Educational Research Methodology, University of North Carolina, Greensboro. Although the individuals listed above have provided many constructive comments and suggestions, responsibility for the final content of this report rests solely with the authoring committee and the NRC. Finally, as chair, I would like to sincerely thank all of my fellow committee members, who generously contributed their time and intellect to this evaluation.

OCR for page R1
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress Our work covered an exceedingly broad array of complex topics and issues, and committee members exhibited a remarkable commitment to learning from each other's expertise, examining NAEP from new and varied perspectives, continuing the dialogue on some very tough issues, and producing a final report that clearly reflected a consensus among all members. All of this occurred in an atmosphere of substantial and ongoing collegiality and cordiality. It has been a professionally stimulating and personally gratifying experience to work with the members of this committee and the NRC project staff. It is my hope that their high standards and expectations have been fulfilled on this evaluation project and by our final report. James W. Pellegrino, Chair Committee on the Evaluation of National and State Assessments of Educational Progress

OCR for page R1
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. William A. Wulf is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. William A. Wulf are chairman and vice chairman, respectively, of the National Research Council.

OCR for page R1
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress Contents     EXECUTIVE SUMMARY   1     INTRODUCTION   9      Changing Sociopolitical Context of NAEP   9      Committee Charge   11      History and Current Status of NAEP   12      Current Context and Demands   19      Overview of the Report   20  1   CREATING A COORDINATED SYSTEM OF EDUCATION INDICATORS   22      Introduction   22      NAEP's Current Mission   24      Views of NAEP's Purpose and Use from Prior Evaluators   25      Findings from Our Evaluation: Uses of NAEP   27      Purpose and Use of Indicator Systems   35      Potential Value of a Coordinated System of Indicators   42      Designing and Supporting a Coordinated System   45      Planning and Managing the System   51      Summary   54

OCR for page R1
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress  2   STREAMLINING THE DESIGN OF NAEP   56      Introduction   56      Overview of NAEP's Current Sampling, Data Collection, Analysis, and Reporting Procedures   57      Selected Findings from Previous NAEP Evaluations   65      The Committee's Evaluation   68      Toward a More Unified Design for NAEP   73      Summary of Proposed Design Features   84      Major Conclusions and Recommendations   84  3   ENHANCING THE PARTICIPATION AND MEANINGFUL ASSESSMENT OF ALL STUDENTS IN NAEP   87      Introduction   87      English-Language Learners and Students with Disabilities   89      Efforts to Enhance Participation in NAEP and Other Large-Scale Assessments   91      Review of Progress Through 1996   92      Problem of Consistent and Accurate Identification   102      Goals for Enhancing Participation and Accommodation   106      A Research Agenda   109      Major Conclusions and Recommendations   112  4   FRAMEWORKS AND THE ASSESSMENT DEVELOPMENT PROCESS: PROVIDING MORE INFORMATIVE PORTRAYALS OF STUDENT PERFORMANCE   114      Introduction   115      Overview of NAEP's Current Assessment Development Process   116      Selected Findings from Previous NAEP Evaluations   122      The Committee's Evaluation   124      A Vision for Assessment Development in NAEP   157      Major Conclusions and Recommendations   159  5   SETTING REASONABLE AND USEFUL PERFORMANCE STANDARDS   162      Introduction   162      NAEP Performance Standards and the Achievement-Level-Setting Process   163      Selected Findings from Past NAEP Evaluations and Research   166      1996 Science Achievement-Level Setting   168      The Committee's Evaluation   171      Achievement-Level Setting in Future NAEP Assessments   181      Major Conclusions and Recommendations   182

OCR for page R1
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress  6   STRATEGIES FOR IMPLEMENTING THE COMMITTEE'S RECOMMENDATIONS FOR TRANSFORMING NAEP   185      Recapitulation of the Primary Recommendations   186      Recent Work That Provides a Foundation for New Paradigm NAEP   189      Operationalizing Changes to the NAEP Program   191      Conclusion   196     REFERENCES   198     APPENDIXES       A Enhancing the Assessment of Reading   219     B Research About Student Learning as a Basis for Developing Assessment Materials: An Example from Science   231     C A Sample Family of Items Based on Number Patterns at Grade 4   237     D Exploring New Models for Achievement-Level Setting   256     E Biographical Sketches   262     INDEX   267

OCR for page R1
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress This page in the original is blank.

OCR for page R1
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress GRADING THENATION'SREPORTCARD

OCR for page R1
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress This page in the original is blank.