DEVELOPING ASSESSMENTS
FOR THE NEXT GENERATION
SCIENCE STANDARDS
Committee on Developing Assessments of Science Proficiency in K-12
James W. Pellegrino, Mark R. Wilson, Judith A. Koenig, and Alexandra S. Beatty, Editors
Board on Testing and Assessment
Board on Science Education
Division of Behavioral and Social Sciences and Education
NATIONAL RESEARCH COUNCIL
OF THE NATIONAL ACADEMIES
THE NATIONAL ACADEMIES PRESS
Washington, D.C.
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This study was supported by the S. D. Bechtel, Jr. Foundation, the Carnegie Corporation of New York under Contract No. B8834, and the William and Flora Hewlett Foundation under Contract No. 2012-7436. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the organizations or agencies that provided support for the project.
International Standard Book Number-13: 978-0-309-28951-1
International Standard Book Number-10: 0-309-28951-3
Library of Congress Control Number: 2014937574
Additional copies of this report are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2014 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Research Council. (2014). Developing Assessments for the Next Generation Science Standards. Committee on Developing Assessments of Science Proficiency in K-12. Board on Testing and Assessment and Board on Science Education, J.W. Pellegrino, M.R. Wilson, J.A. Koenig, and A.S. Beatty, Editors. Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. C. D. Mote, Jr., is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. C. D. Mote, Jr., are chair and vice chair, respectively, of the National Research Council.
COMMITTEE ON DEVELOPING ASSESSMENTS OF SCIENCE PROFICIENCY IN K-12
JAMES W. PELLEGRINO (Cochair), Learning Sciences Research Institute, University of Illinois at Chicago
MARK R. WILSON (Cochair), Graduate School of Education, University of California, Berkeley
RICHARD M. AMASINO, Department of Biochemistry, University of Wisconsin–Madison
EDWARD H. HAERTEL, School of Education, Stanford University
JOAN HERMAN, National Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles
RICHARD LEHRER, Department of Teaching and Learning, Peabody College of Vanderbilt University
SCOTT F. MARION, National Center for the Improvement of Educational Assessment, Inc.
PETER McLAREN, Rhode Island Department of Education
KNUT NEUMANN, Leibniz Institute for Science and Mathematics Education, University of Kiel
WILLIAM PENUEL, School of Education, University of Colorado at Boulder
HELEN R. QUINN, SLAC National Accelerator Laboratory, Stanford University
BRIAN J. REISER, School of Education and Social Policy, Northwestern University
KATHLEEN SCALISE, Educational Methodology, Policy and Leadership, University of Oregon
JEROME M. SHAW, Education Department, University of California, Santa Cruz
NANCY BUTLER SONGER, School of Education, University of Michigan, Ann Arbor
ROBERTA TANNER, Retired Physics Teacher, Thompson School District, Loveland, Colorado
CATHERINE J. WELCH, Department of Psychological and Quantitative Foundations, Educational Measurement, and Statistics, University of Iowa
JUDITH A. KOENIG, Study Director, Senior Program Officer
ALEXANDRA S. BEATTY, Senior Program Officer
STUART W. ELLIOTT, Director of Board on Testing and Assessment
HEIDI SCHWEINGRUBER, Senior Program Officer
MARTIN STORKSDIECK, Director of Board on Science Education
KELLY N. ARRINGTON, Senior Program Assistant
BOARD ON TESTING AND ASSESSMENT
EDWARD HAERTEL (Chair), School of Education, Stanford University
GARY CHAMBERLAIN, Department of Economics, Harvard University
MARK DYNARSKI, Pemberton Research, LLC
DAVID J. FRANCIS, Texas Institute for Measurement, Evaluation, and Statistics, University of Houston
JOAN HERMAN, National Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles
MICHAEL KANE, Test Validity, Educational Testing Service, Princeton, New Jersey
SHARON LEWIS, Council of the Great City Schools
ROBERT MARE, Department of Sociology, University of California, Los Angeles
DIANA C. PULLIN, School of Education, Boston College
ANN MARIE RYAN, Department of Psychology, Michigan State University
BRIAN STECHER, Education Program, The RAND Corporation, Santa Monica, California
JOHN ROBERT WARREN, Department of Sociology, University of Minnesota
MARK R. WILSON, Graduate School of Education, University of California, Berkeley
REBECCA ZWICK, Research and Development, Educational Testing Service, Santa Barbara, California
STUART W. ELLIOTT, Director
BOARD ON SCIENCE EDUCATION
HELEN R. QUINN (Chair), SLAC National Accelerator Laboratory, Stanford University
GEORGE BOGGS, Palomar College, San Marcos, California
MELANIE COOPER, Department of Chemistry, Michigan State University
RODOLFO DIRZO, Department of Biology, Stanford University
JACQUELYNNE ECCLES, Department of Psychology, University of Michigan, Ann Arbor
JOSEPH FRANCISCO, Department of Chemistry, Purdue University
MARGARET A. HONEY, New York Hall of Science
SUSAN KIEFFER, Department of Geology, University of Illinois, Urbana
MATTHEW KREHBIEL, Kansas State Department of Education
MICHAEL LACH, Urban Education Institute, University of Chicago
LYNN LIBEN, Department of Psychology, The Pennsylvania State University
BRIAN REISER¸ School of Education and Social Policy, Northwestern University
MARSHALL “MIKE” SMITH, Carnegie Foundation for the Advancement of Teaching
ROBERTA TANNER, Retired Physics Teacher, Thompson School District, Loveland, Colorado
SUZANNE WILSON, Department of Teacher Education, Michigan State University
YU XIE, Department of Sociology, University of Michigan, Ann Arbor
MARTIN STORKSDIECK, Director
HEIDI SCHWEINGRUBER, Deputy Director
Preface
The U.S. public education system has seen many reform efforts come and go, and the claim that few leave lasting benefits after they are out of fashion can be made with some credibility. The 2012 A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas offers the promise of something quite different. The framework proposed a dramatic rethinking of science education grounded in a thoughtful analysis of the reasons science education has fallen short. With its insistence that science education integrate the practices, disciplinary core ideas, and crosscutting concepts of science and engineering in a coherent fashion across the K-12 years, the framework established goals that cannot be achieved through tinkering. Implementing its vision will require a thorough rethinking of each element of science education, including science assessment.
Assessments, understood as tools for tracking what and how well students have learned, play a critical role in the education system—from classrooms to state houses. Frequent misapplication of these tools and misuse of their results have tarnished their reputation. But the new K-12 framework makes clear that such tools, reflecting new modes of assessment designed to measure the integrated learning it envisions, will be essential. Our committee was asked to develop an approach to science assessment that would support attainment of this vision as it has been elaborated in the Next Generation Science Standards: For States, By States, which were developed in response to the framework. Both documents are new, and the changes they call for are barely under way, but new assessments will be needed as soon as states and districts begin the process of implementing the
Next Generation Science Standards (NGSS) and changing their approach to science education. This meant that our committee had to work quickly to assemble and evaluate a wide range of information related to research and practice and to assimilate thinking and perspectives from across many disciplines.
With funding from the S. D. Bechtel, Jr. Foundation, the Carnegie Corporation of New York, and the William and Flora Hewlett Foundation, the National Research Council (NRC) established the Committee on Developing Assessment of Science Proficiency in K-12 to carry out a consensus study under the aegis of the Board on Testing and Assessment and the Board on Science Education. The committee was asked to recommend strategies for developing assessments that measure student proficiency in science as laid out in the new K-12 science education framework.
The committee benefited from the work of many others, and we wish to thank the many individuals who assisted us. We first thank the sponsors who supported this work: the S. D. Bechtel, Jr. Foundation, the Carnegie Corporation of New York, and the William and Flora Hewlett Foundation. We particularly thank the representatives from the sponsoring groups for their ongoing assistance and insights about the project: Andres Henriquez with Carnegie; Dennis Udall with Hewlett; and Soo Venkateson with Bechtel.
During the course of its work, the committee met four times, including two public sessions. The first public session was held in Palo Alto, California, at the offices of the Moore Foundation. We thank the staff at the Moore Foundation, particularly Janet Coffey, for their gracious hospitality in hosting this meeting. At this meeting, we heard from representatives of the two Race to the Top Assessment Program consortia with regard to their plans for using computer-based assessments, performance tasks, and other innovative approaches to assessing English language arts and mathematics that might be applied to assessment of science. We thank Jeff Nelhaus and Enis Dogan for their presentations on the work of the Partnership for Assessment of Readiness for College and Careers, and we thank Joe Wilhoft and Stan Rabinowitz for their presentations on the work of the Smarter Balanced Assessment Consortia.
The second meeting included a public workshop designed for the committee to learn more about innovative approaches to science assessment. We thank Alan Friedman, former member of the National Assessment Governing Board, and Peggy Carr, with the National Center for Education Statistics, for their presentation about the Computer Interactive and Hands-On Science Assessment of the National Assessment of Educational Progress; Rosemary Reshetar, with the
College Board, for her presentation about the newly revised Advanced Placement assessment in biology; Edys Quellmalz, with WestEd, for her presentation about the SimScientist Program; and Joseph Krajcik, with Michigan State University, for his presentation about the Investigating and Questioning our World through Science and Technology Program.
The workshop also provided time for the committee to learn more about science assessments that are currently used in some states, as well as time for state science instruction and assessment specialists to discuss the assessment challenges associated with the NGSS. To organize this part of the workshop, we coordinated our plans with David Heil and Sasha Burchuk, the State Collaborative on Assessment and Student Standards (SCASS) in science. The SCASS is supported by the Council of Chief State School Officers (CCSSO) and includes 29 science instruction and assessment experts in 16 states. David and Sasha were key in arranging for the wide participation of those experts in the workshop and helped us select SCASS members to serve on workshop panels. We are very grateful for the time, effort, and insights David and Sasha contributed toward making the workshop a success. We also thank the CCSSO for covering their financial contribution for the workshop.
We offer appreciation to all the state science instruction and assessment specialists who made presentations at the workshop, including Robin Anglin, West Virginia Department of Education; Anita Bernhardt, Maine Department of Education; Melinda Curless, Kentucky Department of Education; Jeff Greig, Connecticut State Department of Education; Susan Codere Kelly, Michigan Department of Education; Matt Krehbiel, Kansas State Department of Education; Shelley Lee, Wisconsin Department of Public Instruction; Yvette McCulley, Iowa Department of Education; Beverly Vance, North Carolina Department of Public Instruction; and James Woodland, Nebraska Department of Education. Their information and insights were very helpful to the committee.
After the workshop, we followed up with state instruction and assessment specialists to learn more about their science assessments. These conversations provided a great deal of background information, and we are grateful for the information and insights we received. We thank Rachel Aazzerah, Oregon Department of Education; Catherine Bowler, Massachusetts Department of Education; Liz Butner, Connecticut Department of Education; Dawn Cameron, Minnesota Department of Education; Gail Hall, Vermont Department of Education; Saundra Hamon, Kentucky Department of Education; Lauren Monowar-Jones, Ohio
Department of Education; Judy Pinnsonault, New York State Department of Education; and Brad Talbert, Utah Department of Education.
The report includes numerous examples of assessment tasks that measure science learning as envisioned in the framework and the NGSS, most of which were originally developed by committee members. Three of these examples were developed by scholars outside of the committee: Geneva Haertel and Daisy Rutstein with SRI International; Thomas Matts and Trevor Packer with the Advanced Placement Program at the College Board; and Edys Quellmalz with WestEd. We thank them for their generosity in allowing us to use their examples.
We are especially indebted to Stephen Pruitt with Achieve, Inc., who coordinated the efforts to develop the NGSS. Stephen provided us with ongoing information about the development of the standards and answered all of our questions. We sincerely appreciate his responsiveness.
The committee gratefully acknowledges the dedicated effort provided by the staff of the Board on Testing and Assessment (BOTA) and the Board on Science Education (BOSE), who worked directly on this project. Stuart Elliott, director of BOTA, and Martin Storksdieck, director of BOSE, provided leadership in moving this project forward, and their insights and guidance throughout the course of the study were invaluable. We thank Heidi Schweingruber of BOSE for her insights about the NGSS and the implications for instruction and assessment. The committee also thanks Kelly Arrington, senior project assistant, for her exceptional organizational skills and her close attention to detail. Kelly handled all of the administrative details associated with four committee meetings, held in a variety of locations, and a workshop attended by more than 100 people, and she provided critical support in preparing the manuscript.
Most especially, we express our appreciation for the extraordinary work done by Judy Koenig and Alix Beatty of BOTA in assembling critical information and in the drafting and editing of this report. Their efforts enabled the committee to push forward and meet multiple challenges related to project timelines, as well as the challenges of substantive issues regarding the design and use of educational assessments in general and for science in particular.
We also thank members of the Office of Reports and Communication of the Division of Behavioral and Social Sciences for their dedicated work on this report. We are indebted to Eugenia Grohman for her sage advice in editing numerous versions of this manuscript. We thank Kirsten Sampson Snyder for her work in coordinating a very intense review process and Yvonne Wise for shepherding the manuscript through myriad stages of production.
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the charge. The review comments and draft manuscript remain confidential to protect the integrity of the process.
We thank the following individuals for their review of this report: Charles W. (Andy) Anderson, Department of Teacher Education, Michigan State University; William B. Bridges, Department of Engineering (emeritus), California Institute of Technology; Derek Briggs, Research and Evaluation Methodology, School of Education, University of Colorado at Boulder; Angela DeBarger, Center for Technology in Learning, SRI International; George DeBoer, Project 2061, American Association for the Advancement of Science; Richard Duran, School of Education, University of California, Santa Barbara; Sean Elkins, Science Academic Program Consultant, Kentucky Department of Education; Brian Gong, Center for Assessment, Dover, New Hampshire; David Klahr, Department of Psychology, Carnegie Mellon University; Matt Krehbiel, Science Education Program, Kansas State Department of Education; Peter Labudde, Centre of Science and Technology Education, University of Applied Sciences and Arts Northwestern Switzerland; Richard C. Larson, Engineering Systems Division, Massachusetts Institute of Technology; Steven Long, Science Department, Rogers High School, Rogers, Arizona; Karen Mitchell, Association of American Medical Colleges; Mark D. Reckase, Department of Counseling, Educational Psychology and Special Education, Michigan State University; Eugenie C. Scott, National Center on Science Education, Oakland, California; Lorrie A. Shepard, School of Education, University of Colorado at Boulder; and Rebecca Zwick, Gevirtz Graduate School of Education, University of California, Santa Barbara, and Statistical Analysis, Data Analysis, and Psychometric Research, Educational Testing Service.
Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the content of the report nor did they see the final draft of the report before its release. The review of this report was overseen by Lauress Wise, with HumRRO, and May Berenbaum, with the University of Illinois at Urbana-Champaign. Appointed by the NRC, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review com-
ments were carefully considered. Responsibility for the final content of this report rests entirely with the committee and the institution.
Finally, as cochairs of the committee, we thank all our fellow committee members for their dedication and outstanding contributions to this project. They actively assisted in all stages of this project, including planning the public workshop and making presentations, selecting and developing examples of assessment tasks, and writing and rewriting multiple drafts of this report. Their contributions during the late stages of the report’s development, when sections of the report had to be revised on very tight schedules, are especially appreciated. They gave generously of their time and intellects throughout the project. We believe their contributions ensure that the final product is understandable to a variety of audiences and fully portrays the complex issues associated with developing the new science assessments that will be needed.
James W. Pellegrino and Mark R. Wilson, Cochairs
Committee on Developing Assessments of Science Proficiency in K-12
CONTENTS
2 Assessments to Meet the Goals of the Framework
The Framework’s Vision for K-12 Science Education
Dimension 1: Scientific and Engineering Practices
Dimension 2: Crosscutting Concepts
Dimension 3: Disciplinary Core Ideas
Integration: Three-Dimensional Science Learning
Learning Progressions: Developing Proficiency Over Time
Supporting Connections Across Disciplines
Assessing Three-Dimensional Learning
Assessing the Development of Three-Dimensional Learning Over Time
3 Assessment Design and Validation
Assessment as a Process of Evidentiary Reasoning
Construct-Centered Approaches to Assessment Design
Illustrations of Task-Design Approaches
Evidence-Centered Design—Example 2: Pinball Car Task
Construct Modeling: Measuring Silkworms
Assessment Purposes: Formative or Summative
Characteristics of NGSS-Aligned Assessments
Variation in Assessment Activities
Tasks with Multiple Components
Example 3: Measuring Silkworms
Example 6: Biodiversity in the Schoolyard
Current Science Monitoring Assessments
Including Performance Tasks in Monitoring Assessments
Measurement and Implementation Issues
Implications for Assessment of the NGSS
On-Demand Assessment Components
Classroom-Embedded Assessment Components
Collections of Performance Tasks
Maintaining the Quality of Classroom-Embedded Components
Taking Advantage of Technology
Variations in Item-Response Formats
Assessing Challenging Constructs
Conclusions and Recommendations
6 Designing an Assessment System
Rationale for a Systems Approach
Value of a System of Assessments
Communicating Assessment Results
Indicators of Opportunity to Learn
Understanding the System Components and Their Uses