Click for next page ( R2


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
Prepublication Copy Uncorrected Proofs Developing Assessments for the Next Generation Science Standards Committee on Developing Assessments of Science Proficiency in K-12 James W. Pellegrino, Mark R. Wilson, Judith A. Koenig, and Alexandra S. Beatty, Editors Board on Testing and Assessment Board on Science Education Division of Behavioral and Social Sciences and Education

OCR for page R1
PREPUBLICATION COPY- Uncorrected proofs THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This study was supported by the S.D. Bechtel, Jr. Foundation, the Carnegie Corporation of New York under Contract No. B8834, and the William and Flora Hewlett Foundation under Contract No. 2012-7436. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the organizations or agencies that provided support for the project. Library of Congress Cataloging-in-Publication Data Additional copies of this report are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu. Copyright 2014 by the National Academy of Sciences. All rights reserved. Printed in the United States of America Suggested citation: National Research Council. (2014). Developing Assessments for the Next Generation Science Standards. Committee on Developing Assessments of Science Proficiency in K-12. Board on Testing and Assessment and Board on Science Education, James W. Pellegrino, Mark R. Wilson, Judith A. Koenig, and Alexandra S. Beatty, Editors. Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press. FM-ii

OCR for page R1
PREPUBLICATION COPY- Uncorrected proofs The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. C. D. Mote, Jr., is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. C. D. Mote Jr., are chair and vice chair, respectively, of the National Research Council. www.national-academies.org . FM-iii

OCR for page R1
PREPUBLICATION COPY- Uncorrected proofs FM-iv

OCR for page R1
PREPUBLICATION COPY- Uncorrected proofs COMMITTEE ON DEVELOPING ASSESSMENTS OF SCIENCE PROFICIENCY IN K-12 JAMES W. PELLEGRINO (Cochair), Learning Sciences Research Institute, University of Illinois at Chicago MARK R. WILSON (Cochair), Graduate School of Education, University of California, Berkeley RICHARD M. AMASINO, Department of Biochemistry, University of Wisconsin, Madison EDWARD H. HAERTEL, School of Education, Stanford University JOAN HERMAN, National Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles RICHARD LEHRER, Department of Teaching and Learning, Peabody College of Vanderbilt University SCOTT F. MARION, National Center for the Improvement of Education Assessment PETER McLAREN, Rhode Island Department of Elementary and Secondary Education KNUT NEUMANN, Leibniz Institute for Science and Mathematics Education, University of Kiel WILLIAM PENUEL, School of Education, University of Colorado at Boulder HELEN R. QUINN, SLAC National Accelerator Laboratory, Stanford University BRIAN J. REISER, School of Education and Social Policy, Northwestern University KATHLEEN SCALISE, Educational Methodology, Policy and Leadership, University of Oregon JEROME M. SHAW, Education Department, University of California, Santa Cruz NANCY BUTLER SONGER, School of Education, University of Michigan ROBERTA TANNER, Retired Physics Teacher, Thompson School District, Loveland, Colorado CATHERINE J. WELCH, Department of Psychological and Quantitative Foundations, Educational Measurement, and Statistics, University of Iowa JUDITH A. KOENIG, Study Director, Senior Program Officer ALEXANDRA S. BEATTY, Senior Program Officer STUART W. ELLIOTT, Director of Board on Testing and Assessment HEIDI SCHWEINGRUBER, Senior Program Officer MARTIN STORKSDIECK, Director of Board on Science Education KELLY N. ARRINGTON, Senior Program Assistant FM-v

OCR for page R1
PREPUBLICATION COPY- Uncorrected proofs BOARD ON TESTING AND ASSESSMENT EDWARD HAERTEL (Chair), School of Education, Stanford University GARY CHAMBERLAIN, Department of Economics, Harvard University MARK DYNARSKI, Pemberton Research, LLC DAVID J. FRANCIS, Texas Institute for Measurement, Evaluation, and Statistics, University of Houston JOAN HERMAN, National Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles MICHAEL KANE, Test Validity, Educational Testing Service, Princeton, New Jersey SHARON LEWIS, Council of Great City Schools ROBERT MARE, Department of Sociology, University of California, Los Angeles DIANA C. PULLIN, School of Education, Boston College ANN MARIE RYAN, Department of Psychology, Michigan State University BRIAN STECHER, Education Program, The RAND Corporation, Santa Monica, California JOHN ROBERT WARREN, Department of Sociology, University of Minnesota MARK WILSON, Graduate School of Education, University of California, Berkeley REBECCA ZWICK, Research and Development, Educational Testing Service, Santa Barbara, California STUART W. ELLIOTT, Director FM-vi

OCR for page R1
PREPUBLICATION COPY- Uncorrected proofs BOARD ON SCIENCE EDUCATION HELEN R. QUINN (Chair), Stanford Linear Accelerator Center, Stanford University GEORGE BOGGS, Palomar College, San Marcos, California MELANIE COOPER, Department of Chemistry, Michigan State University RODOLFO DIRZO, Department of Biology, Stanford University JACQUELYNNE ECCLES, Department of Psychology, University of Michigan JOSEPH FRANCISCO, Department of Chemistry, Purdue University MARGARET A. HONEY, New York Hall of Science, New York, New York SUSAN KIEFFER, Department of Geology, University of Illinois, Urbana MATTHEW KREHBIEL, Kansas State Department of Education MICHAEL LACH, Urban Education Institute, University of Chicago LYNN LIBEN, Department of Psychology, The Pennsylvania State University BRIAN REISER¸ School of Education and Social Policy, Northwestern University MARSHALL “MIKE” SMITH, Carnegie Foundation for the Advancement of Teaching ROBERTA TANNER, Retired Physics Teacher, Thompson School District, Loveland, Colorado SUZANNE WILSON, Department of Teacher Education, Michigan State University YU XIE, Department of Sociology, University of Michigan MARTIN STORKSDIECK, Director HEIDI SCHWEINGRUBER, Deputy Director FM-vii

OCR for page R1
PREPUBLICATION COPY- Uncorrected proofs FM-viii

OCR for page R1
PREPUBLICATION COPY- Uncorrected proofs Preface The U.S. public education system has seen many reform efforts come and go, and the claim that few leave lasting benefits after they are out of fashion can be made with some credibility. The 2012 Framework for K-12 Science Education offers the promise of something quite different. The framework proposed a dramatic rethinking of science education grounded in a thoughtful analysis of the reasons science education has fallen short. With its insistence that science education integrate the practices, core disciplinary ideas, and crosscutting concepts of science and engineering in a coherent fashion across the K-12 years, the framework established goals that cannot be achieved through tinkering. Implementing its vision will require a thorough rethinking of each element of science education, including science assessment. Assessments, understood as tools for tracking what and how well students have learned, play a critical role in the education system--from classrooms to statehouses. Frequent misapplication of these tools and misuse of their results have tarnished their reputation. But the new K-12 framework makes clear that such tools, reflecting new modes of assessment designed to measure the integrated learning it envisions, will be essential. Our committee was asked to develop an approach to science assessment that would support and enable attainment of this vision as it has been elaborated in the Next Generation Science Standards (NGSS), which were developed in response to the framework. Both documents are new, and the changes they call for are barely under way, but new assessments will be needed as soon as states and districts begin the process of implementing the NGSS and changing their approach to science education. This meant that our committee had to work quickly to assemble and evaluate a wide range of information related to research and practice and to assimilate thinking and perspectives from across many disciplines. With funding from the S.D. Bechtel, Jr. Foundation, the Carnegie Corporation of New York, and the William and Flora Hewlett Foundation, the National Research Council (NRC) established the Committee on Developing Assessment of Science Proficiency in K-12 to carry out a consensus study under the aegis of the Board on Testing and Assessment and the Board on Science Education. The committee was asked to recommend strategies for developing assessments that validly measure student proficiency in science as laid out in the new K-12 science education framework. The committee benefited from the work of many others, and we wish to thank the many individuals who assisted us. We first thank the sponsors who supported this work: the S.D. Bechtel, Jr. Foundation, the Carnegie Corporation of New York, and the William and Flora Hewlett Foundation. We particularly thank the representatives from the sponsoring groups for their ongoing assistance and insights about the project: Andres Henriquez with Carnegie; Dennis Udall, with Hewlett; and Soo Venkateson with Bechtel. During the course of its work, the committee met four times, including two public sessions. The first public session was held in Palo Alto at the offices of the Moore Foundation. We thank the staff at the Moore Foundation, particularly Janet Coffey, for their gracious hospitality in hosting this meeting. At this meeting, we heard from representatives of the two Race to the Top assessment consortia with regard to their plans for using computer-based assessments, performance tasks, and other innovative approaches to assessing English language arts and mathematics that might be applied to assessment of science. We thank Jeff Nelhaus and Enis Dogan for their presentations on the work of the Partnership for Assessment of Readiness FM-ix

OCR for page R1
PREPUBLICATION COPY- Uncorrected proofs for College and Careers, and we thank Joe Wilhoft and Stan Rabinowitz for their presentations on the work of the Smarter Balanced Assessment Consortia. The second meeting included a public workshop designed for the committee to learn more about innovative approaches to science assessment. We thank Alan Friedman, former member of the National Assessment Governing Board, and Peggy Carr, with the National Center for Education Statistics, for their presentation about the Computer Interactive and Hands-On Science Assessment of the National Assessment of Educational Progress; Rosemary Reshetar, with the College Board, for her presentation about the newly revised Advance Placement assessment in biology; Edys Quellmalz, with WestEd, for her presentation about the SimScientist program; and Joseph Krajcik, with Michigan State University, for his presentation about the Investigating and Questioning our World through Science and Technology program. The workshop also provided time for the committee to learn more about science assessments that are currently used in some states, as well as time for state science instruction and assessment specialists to discuss the assessment challenges associated with the NGSS. To organize this part of the workshop, we coordinated our plans with David Heil and Sasha Burchuk, the State Collaborative on Assessment and Student Standards (SCASS) in Science. The SCASS is supported by the Council of Chief State School Officers (CCSSO) and includes 29 science instruction and assessment experts in 16 states. David and Sasha were key in arranging for the wide participation of those experts in the workshop and helped us select SCASS members to serve on workshop panels. We are very grateful for the time, effort, and insights David and Sasha contributed toward making the workshop a success. We also thank the CCSSO for covering their financial contribution for the workshop. We offer appreciation to all the state science instruction and assessments specialists who made presentations at the workshop, including Robin Anglin, West Virginia Department of Education; Anita Bernhardt, Maine Department of Education; Melinda Curless, Kentucky Department of Education; Jeff Greig, Connecticut State Department of Education; Susan Codere Kelly, Michigan Department of Education; Matt Krehbiel, Kansas State Department of Education; Shelley Lee, Wisconsin Department of Public Instruction; Yvette McCulley, Iowa Department of Education; Beverly Vance, North Carolina Department of Public Instruction; and James Woodland, Nebraska Department of Education. Their information and insights were very helpful to the committee. After the workshop, we followed up with state instruction and assessment specialists to learn more about their science assessments. These conversations provided a great deal of background information, and we are grateful for the information and insights we received. We thank Rachel Aazzerah, Oregon Department of Education; Catherine Bowler, Massachusetts Department of Education; Liz Butner, Connecticut Department of Education; Dawn Cameron, Minnesota Department of Education; Gail Hall, Vermont Department of Education; Saundra Hamon, Kentucky Department of Education; Lauren Monowar-Jones, Ohio Department of Education; Judy Pinnsonault, New York State Department of Education; and Brad Talbert, Utah Department of Education. The report includes numerous examples of assessment tasks that measure science learning as envisioned in the framework and the NGSS, most of which were originally developed by committee members. Three of these examples were developed by scholars outside of the committee: Geneva Haertel and Daisy Rutstein with SRI; Thomas Matts and Trevor Packer with the Advanced Placement Program at College Board; and Edys Quellmalz with WestEd. We thank them for their generosity in allowing us to use their examples. FM-x

OCR for page R1
PREPUBLICATION COPY- Uncorrected proofs We are especially indepted to Stephen Pruitt with Achieve, who coordinated the efforts to develop the Next Generation Science Standards. Stephen provided us with ongoing information about the development of the standards and and answered all of our questions. We sincerely appreciate his responsiveness. The committee gratefully acknowledges the dedicated effort provided by the staff of the Board on Testing and Assessment (BOTA) and the Board on Science Education (BOSE), who worked directly on this project. Stuart Elliott, director of BOTA, and Martin Storksdieck, director of BOSE, provided leadership in moving this project forward, and their insights and guidance throughout the course of the study were invaluable. We thank Heidi Schweingruber of BOSE for her insights about the NGSS and the implications for instruction and assessment. The committee also thanks Kelly Arrington, senior project assistant, for her exceptional organizational skills and her close attention to detail. Kelly handled all of the administrative details associated with four committee meetings, held in a variety of locations, and a workshop attended by more than 100 people, and she provided critical support in preparing the manuscript. Most especially, we express our appreciation for the extraordinary work done by Judy Koenig and Alix Beatty of BOTA in assembling critical information and in the drafting and editing of this report. Their efforts enabled the committee to push forward and meet multiple challenges related to project timelines, as well as the challenges of substantive issues regarding the design and use of educational assessments in general and for science in particular. We also thank member of the Office of Reports and Communication of the Division of Behavioral and Social Science for their dedicated work on this report. We are indebted to Eugenia Grohman for her sage advice in editing numerous versions of this manuscript. We thank Kirsten Sampson-Snyder for her work in coordinating a very intense review process and Yvonne Wise for shepherding the manuscript through myriad stages of production. This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the charge. The review comments and draft manuscript remain confidential to protect the integrity of the process. We thank the following individuals for their review of this report: Charles W. (Andy) Anderson, Department of Teacher Education, Michigan State University; William B. Bridges, Department of Engineering, Emeritus, California Institute of Technology; Derek Briggs, Research and Evaluation Methodology, School of Education, University of Colorado at Boulder; Angela DeBarger, Center for Technology in Learning, SRI International; George DeBoer, Project 2061, American Association for the Advancement of Science; Richard Duran, School of Education, University of California, Santa Barbara; Sean Elkins, Science Academic Program Consultant, Kentucky Department of Education; Brian Gong, Executive Director, Center for Assessment, Dover, New Hampshire; David Klahr, Department of Psychology, Carnegie Mellon University; Matt Krehbiel, Science Education Program, Kansas State Department of Education; Peter Labudde, Centre of Science and Technology Education, University of Applied Sciences and Arts Northwestern Switzerland; Richard C. Larson, Engineering Systems Division, Massachusetts Institute of Technology; Steven Long, Science Department, Rogers High School, Rogers, Arizona; Karen Mitchell, Research Area Director, SRI International; Mark D. Reckase, Department of Counseling, Educational Psychology and Special Education, Michigan State FM-xi

OCR for page R1
PREPUBLICATION COPY- Uncorrected proofs University; Eugenie C. Scott, Executive Director, National Center on Science Education, Oakland, California; Lorrie A. Shepard, School of Education, University of Colorado at Boulder; and Rebecca Zwick, Gevirtz Graduate School of Education, University of California, Santa Barbara, and Statistical Analysis, Data Analysis, and Psychometric Research, Educational Testing Service. Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the content of the report nor did they see the final draft of the report before its release. The review of this report was overseen by Lauress Wise, with HumRRO, and May Berenbaum, with the University of Illinois at Urbana-Champaign. Appointed by the NRC, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the committee and the institution. Finally, as cochairs of the committee, we thank all our fellow committee members for their dedication and outstanding contributions to this project. They actively assisted in all stages of this project, including planning the public workshop and making presentations, selecting and developing examples of assessment tasks, and writing and rewriting multiple drafts of this report. Their contributions during the late stages of the report’s development, when sections of the report had to be revised on very tight schedules, are especially appreciated. They gave generously of their time and intellects throughout the project. We believe their contributions ensure that the final product is understandable to a variety of audiences and fully portrays the complex issues associated with developing the new science assessments that will be needed. James W. Pellegrino and Mark R. Wilson, Cochairs Committee on Developing Assessments of Science Proficiency in K-12 FM-xii

OCR for page R1
PREPUBLICATION COPY- Uncorrected proofs Contents Summary 1 Introduction Context Need for Fundamental Change Committee’s Approach Guide to the Report 2 Assessments to Meet the Goals of the Framework The Framework’s Vision for K-12 Science Education Dimensions of the Framework Dimension 1: Scientific and Engineering Practices Dimension 2: Crosscutting Concepts Dimension 3: Disciplinary Core Ideas Integration: Three-Dimensional Science Learning Learning Progressions: Developing Proficiency Over Time Supporting Connections Across Disciplines Example 1: What is Going on Inside Me? Instructional Context Assessment Conclusions Assessing Three-Dimensional Learning Assessing the Development of Three-Dimensional Learning Over Time Breadth and Depth of Content 3 Assessment Design and Validation Assessment as a Process of Evidentiary Reasoning Construct-Centered Approaches to Assessment Design Illustrations of Task Design Approaches Evidence-Centered Design: Pinball Car Task Construct Modeling: Measuring Silkworms Validation Conclusion and Recommendation 4 Classroom Assessment Assessment Purposes: Formative or Summative Characteristics of NGSS-Aligned Assessments Variation in Assessment Activities Tasks with Multiple Components Making Connections Learning as a Progression Six Examples FM-xiii

OCR for page R1
PREPUBLICATION COPY- Uncorrected proofs Example 3: Measuring Silkworms Example 4: Behavior of Air Example 5: Movement of Water Example 6: Biodiversity in the Schoolyard Formative Assessment Tasks Summative Assessment Task The Set of Tasks Example 7: Climate Change Example 8: Ecosystems Lessons from the Examples Types of Assessment Activities Interpreting Results Using Multiple Practices Professional Development Conclusions and Recommendations 5 Assessment for Monitoring Current Science Monitoring Assessments Including Performance Tasks in Monitoring Assessments Measurement and Implementation Issues Examples Implications for Assessment of the NGSS Design Options Assumptions Two Classes of Design Options On-Demand Assessment Components Classroom-Embedded Assessment Components Maintaining the Quality of Classroom-Embedded Components Taking Advantage of Technology Variations in Item Response Formats Assessing Challenging Constructs Task Surrounds Conclusions and Recommendations 6 Designing an Assessment System Rationale for a Systems Approach Value of a System of Assessments Curriculum and Instruction Accountability Policies Communicating Assessment Results System Components Classroom Assessments Monitoring Assessments Indicators of Opportunity to Learn Understanding the System Components and Their Uses Examples of Alternative Science Assessment Systems FM-xiv

OCR for page R1
PREPUBLICATION COPY- Uncorrected proofs Example 1 Example 2 Conclusions and Recommendations 7 Implementing a Science Assessment System Gradual Implementation Equity and Fairness Technology Costs Conclusions and Recommendations References Appendix A Workshop Agenda Appendix B Biographical Sketches of Committee Members FM-xv