SYSTEMS FOR STATE SCIENCE ASSESSMENT
Mark R. Wilson and Meryl W. Bertenthal, editors
THE NATIONAL ACADEMIES PRESS
THE NATIONAL ACADEMIES PRESS
500 Fifth Street, N.W. Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This study was supported by Master Agreement No. 9911018 between the National Academy of Sciences and the National Science Foundation, Grant No. EHR-0237653. Any opinions, findings, conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Library of Congress Cataloging-in-Publication Data
Systems for state science assessment / Committee on Test Design for K–12 Science Achievement ; Mark R. Wilson and Meryl W. Bertenthal, editors.
Includes bibliographical references and index.
ISBN 0-309-09662-6 (hardcover)—ISBN 0-309-55060-2 (pdf) 1. Science—Study and teaching (Elementary)—United States—Evaluation. 2. Science—Study and teaching (Secondary)—United States—Evaluation. 3. Science—United States—Examinations, questions, etc. I. Wilson, Mark R. II. Bertenthal, Meryl W. III. National Research Council (U.S.). Committee on Test Design for K–12 Science Achievement.
Additional copies of this report are available from
National Academies Press,
500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu.
Printed in the United States of America
Copyright 2006 by the National Academy of Sciences. All rights reserved.
Suggested citation: National Research Council. (2006). Systems for State Science Assessment. Committee on Test Design for K–12 Science Achievement. M.R. Wilson and M.W. Bertenthal, eds. Board on Testing and Assessment, Center for Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National Research Council.
COMMITTEE ON TEST DESIGN FOR K–12 SCIENCE ACHIEVEMENT
MARK R. WILSON (Chair),
University of California, Berkeley
J. MYRON ATKIN,
School of Education, emeritus, Stanford University
AUDREY B. CHAMPAGNE,
School of Education and Department of Chemistry, State University of New York at Albany
DAVID N. FIGLIO,
Department of Economics, University of Florida
GREGORY B. HALL,
Washington Department of Public Instruction, Olympia, WA
National Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles
HEINRICH D. HOLLAND,
Department of Earth and Planetary Sciences, Harvard University
School of Education, University of Michigan
School of Education, University of Pittsburgh
Peabody College Department of Teaching and Learning, Vanderbilt University
Council of the Great City Schools, Washington, DC
JAMES W. PELLEGRINO,
Department of Psychology and Center for the Study of Learning, Instruction, and Teacher Development, University of Illinois, Chicago
The RAND Corporation, Santa Monica, CA
GERALD M. STOKES,
Joint Global Change Research Institute, University of Maryland
Alternative School for Math and Science, Corning, NY
MERYL BERTENTHAL, Study Director
ALEXANDRA BEATTY, Senior Program Officer (from June 2004)
JUDITH KOENIG, Senior Program Officer (until February 2004)
SUSAN McCUTCHEN, Research Associate
M. JANE PHILLIPS, Senior Program Assistant (until February 2004)
ANDREW TOMPKINS, Research Associate
TERESIA WILMORE, Senior Program Assistant (from April 2004)
Working Group of Assessment Directors
C. SCOTT TRIMBLE (Acting Chair),
Kentucky Department of Education (retired)
Virginia Department of Education
HERMAN W. MEYERS,
Vermont State Department of Education (retired)
Nebraska Department of Education
Texas Education Agency (retired)
Working Group of Science Supervisors
THOMAS E. KELLER (Chair),
Maine Department of Education
California Department of Education
Tennessee Department of Education
SHELLEY A. LEE,
Wisconsin Department of Public Instruction
Utah State Office of Education
Florida Department of Education
Working Group of Science Teachers
VALDINE McLEAN (Chair),
Pershing County High School, Lovelock, NV
George Washington Carver High School of Engineering and Science, Philadelphia, PA
Williamston Explorer Elementary School, Williamston, MI
West Branch Middle School, West Branch, IA
Guilford County Middle College High School, Jamestown, NC
Mountain Ridge Middle School, Castle Rock, CO
CAROL A. SHESTOK,
Westford Public School System, Westford, MA
BOARD ON TESTING AND ASSESSMENT
LAURESS L. WISE (Chair),
Human Resources Research Organization, Alexandria, VA
LYLE F. BACHMAN,
Department of Applied Linguistics and TESL, University of California, Los Angeles
EVA L. BAKER,
Center for the Study of Evaluation, University of California, Los Angeles
STEPHEN B. DUNBAR,
College of Education, University of Iowa
DAVID J. FRANCIS,
Department of Psychology, University of Houston
MILTON D. HAKEL,
Department of Psychology, Bowling Green State University
ANDREW J. HARTMAN,
Bell Policy Center, Denver
ROBERT M. HAUSER,
Institute for Research on Poverty, Center for Demography, University of Wisconsin–Madison
DANIEL M. KORETZ,
Graduate School of Education, Harvard University
EDWARD P. LAZEAR,
Graduate School of Business, Stanford University
RICHARD J. LIGHT,
Graduate School of Education and John F. Kennedy School of Government, Harvard University
ROBERT J. MISLEVY,
Department of Measurement and Statistics, University of Maryland, College Park
MICHAEL T. NETTLES,
Educational Testing Service, Princeton, NJ
JAMES W. PELLEGRINO,
Center for the Study of Learning, Instruction, and Teacher Development, University of Illinois, Chicago
DIANA C. PULLIN,
School of Education, Boston College
LORETTA A. SHEPARD,
School of Education, University of Colorado, Boulder
STUART ELLIOTT, Board Director
LISA ALSTON, Administrative Coordinator
This report could not have been produced without the support of a number of people, and we are grateful for their contributions. First, we would like to acknowledge the support of our sponsor, the National Science Foundation, and, in particular, thank program officer Janice Earle for the encouragement she provided over the course of this study. Her belief in the importance of this project was highly valued by the committee.
We owe a debt of gratitude to the individuals who served on the working groups of state science supervisors, state directors of assessment, and science teachers, whose names are listed in this front matter. Members of these groups attended and contributed to committee meetings, participated in workshops, and reviewed drafts of the committee’s and design teams’ reports. Throughout the study they kept the committee informed about important practical issues related to No Child Left Behind (NCLB) and states’ efforts to meet the legislative requirements for science standards and assessments. Several of the working group members also participated directly in the work of the design teams, providing the wisdom of practice to teams’ work.
Our sincerest thanks go to state directors of assessment: Shelley Loving-Ryder, Virginia Department of Education; Herman W. Meyers, Vermont State Department of Education; Pat Roschewski, Nebraska Department of Education; Ann Smisko, Texas Education Agency; C. Scott Trimble (retired), Kentucky Department of Education; state science supervisors: Thomas E. Keller, Maine Department of Education; Diane Hernandez, California Department of Education; Linda Jordan, Tennessee Department of Education; Shelley A. Lee, Wisconsin Department of Public Instruction; Brett Moulding, Utah State Office of Educa-
tion; Marsha Winegarner, Florida Department of Education; and outstanding science teachers: Valdine McLean, Pershing County High School, Lovelock, NV; Amitabha Basu, George Washington Carver High School of Engineering and Science, Philadelphia, PA; Conni Crittenden, Williamston Explorer Elementary School, Williamston, MI; Hector Ibarra, West Branch Middle School, West Branch, IA; Patricia LeGrand, Guilford County Middle College High School, Jamestown, NC; John McKinney, Mountain Ridge Middle School, Castle Rock, CO; and Carol A. Shestok, Westford Public School System, Westford, MA.
Because the committee wanted its findings to be based on a broad understanding of the ways in which science assessment systems could be designed and implemented, we asked six design teams of experts to help us conceptualize a variety of assessment system models. Individually and collectively, the members of the six groups provided great insight into the complexity of the issues involved in creating quality assessment systems. We are particularly grateful to the people who led four of these design teams: W. James Popham, University of California, Los Angeles (emeritus); Barbara Plake, Buros Institute, University of Nebraska; Edys Quellmalz, SRI International; Mark Moody, Hillcrest Associates; and Richard Patz, R.J. Patz, Inc. Also serving on these design teams were Mark Reckase, Michigan State University; Joseph Martineau, Michigan State University; Chad W. Buckendal, University of Nebraska–Lincoln; James C. Impara, University of Nebraska–Lincoln; Ian MacGregor, National Science Resources Center; Scott Marion, National Center for the Improvement in Educational Assessment; Carol Shestok, Westford Public School System; John McKinney, Mountain Ridge Middle School; Teresa Eckhout, Lincoln Public Schools; Cindy Gray, Elkhorn Public Schools; Thomas Neumann, Millard South High School; Arthur Zygielbaum, National Center for Information Technology in Education; Gwen Nugent, National Center for Information Technology in Education; Roger Bruning, University of Nebraska–Lincoln; Albert Steckelberg, University of Nebraska–Lincoln; Myron Atkin, Stanford University; James W. Pellegrino, University of Illinois at Chicago; Thomas E. Keller, Maine Department of Education; Brett Moulding, Utah Office of Education; Paul D. Sandifer, South Carolina Department of Education; James Beall, St. John’s College, Annapolis; Henry Heikkinen, University of Northern Colorado; Smith Holt, Oklahoma State University; John Layman, University of Maryland; A. Truman Schwartz, Macalester College; and Christos Zahopolous, Northeastern University.
We also acknowledge with great appreciation the work done by Carol Smith, University of Massachusetts, Boston; Marianne Wiser, Clark University; Andy Anderson, Michigan State University; Brian Coppola, University of Michigan; Brian Reiser, Northwestern University; and Kefyn Catley, Vanderbilt University, who constituted the two design teams charged with developing assessments based on research on children’s learning.
The committee was aided greatly by individuals who participated in our meetings and helped us to understand the complex issues involved in designing
and implementing science assessment systems to meet the requirements of the No Child Left Behind Act of 2002. A portion of the committee’s first meeting in August 2003 was an information gathering workshop at which Sue Rigney, U.S. Department of Education, Office of Elementary and Secondary Education, made a detailed presentation of the legislation and its requirements for science standards and assessments. Rolf Blank, Council of Chief State School Officers; Michael Hill, National Association of State Boards of Education; Bruce Hunter, American Association of School Administrators; Mary Kusler, American Association of Schools Administrators, Rural Schools Initiative; Michelle McGlaughlin, American Federation of Teachers; and Scott Young, National Conference of State Legislatures provided the committee with an overview of the challenges that states face in implementing science assessments under NCLB. We are grateful for the guidance these individuals provided.
We thank Robert Rothman, Annenberg Institute for School Reform, Brown University, for describing the many methods used by states for aligning their science standards and assessments, and Dylan Wiliam, Educational Testing Service, and Paul Black, Kings College, Cambridge, England, for providing the committee with a deeper understanding of international approaches to science education and assessment. We are also grateful to Mark Reckase and Joseph Martineau for sharing their thinking on the ways in which students’ achievement in science might be measured over time and across grades, and to Stan Metzenberg, California State University, Northridge, for sharing early drafts of his paper “Improving State Science Assessments,” with the committee. We also thank Alice Fu, a graduate student at Stanford University, for her assistance with gathering examples to illustrate state science standards.
At the committee’s third meeting, Gerald Tindal, University of Oregon, and Mark Reckase helped the committee understand many of the issues involved in developing and implementing an assessment program that is highly inclusive. We are grateful for their perspectives and are appreciative of their participation in our work.
In May 2004, the committee held a workshop at which more than 100 participants representing educational organizations, states, test publishers, and educational researchers participated in detailed discussions of the design teams’ work and gave their reactions to the usefulness of the various models. We are grateful to all of the design team members who participated and to the members of the working groups who served as discussion leaders at the workshop. At that same workshop, the committee also explored the ways in which technology could be used for improving the design, implementation, and administration of science assessments. We thank Randy Bennett, Educational Testing Service, and Edys Quellmalz for sharing their work on this topic. We also thank Geneva Haertel, SRI International, who coauthored the paper that Edys presented.
We have been most fortunate to have an outstanding staff working with us on this project. We are especially grateful to Alix Beatty, who joined the project
just as we were beginning to write this report and whose assistance was invaluable in getting the volume ready for publication. Susan McCutchen and Andrew Tompkins have provided ongoing research support to the committee since its inception, and Judy Koenig helped us early on to organize the work of our design teams and to develop their charge. The committee benefited tremendously from having two excellent senior project assistants. Jane Phillips supported the committee extremely well until her retirement. We are very grateful to Teresia Wilmore, who joined the project after Jane left and who has served ever since, unflappably and flawlessly. She has dealt smoothly with the logistics of our meetings, with our enormous collections and distributions of materials, and with the many aspects of managing a committee, three working groups, six design teams, and multiple authors of commissioned papers.
The Board on Testing and Assessment (BOTA) and the Board on Science Education (BOSE) were instrumental in shaping early discussions about the project and in providing general guidance and support along the way. Their insights and knowledge at critical points in our work are much appreciated. We are especially grateful to BOTA members Lorrie Shepard, Eva Baker, and Bob Mislevy for their contributions to our thinking on possible assessment system models. We thank Stuart Elliott, BOTA director, and Jean Moon, BOSE director, for their support and encouragement. We note in particular our deep gratitude to Stuart, who attended all of our committee meetings and provided wise counsel at every stage of the committee’s work.
Also at the National Research Council (NRC), many people contributed to the successful completion of this project. Martin Orland and Patricia Morison, director and associate director, respectively, of the Center for Education, provided the committee with encouragement and guidance along the way. Viola Horek, administrative coordinator for the center, and Lisa Alston, administrative coordinator for BOTA, were always there to ensure that the project ran smoothly and that the committee had everything it needed to do its work. We are extremely grateful to Kirsten Sampson Snyder who shepherded the committee’s report through review and to Steve Olson and Christine McShane whose expert editing of the text of this report markedly improved its readability.
Above all, we would like to thank the committee members who gave enormous amounts of time and energy to this endeavor. The committee’s work covered an exceedingly broad array of complex topics and issues, and committee members exhibited a deep commitment to learning from each other’s expertise and producing a final report that reflects a consensus among all members. Everyone on the committee drafted text and contributed constructive, critical thinking to the deliberations. We are most grateful to them, as they made this report possible.
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC’s Report Review Committee. The purpose of this indepen-
dent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: Eva L. Baker, Center for the Study of Evaluation, University of California, Los Angeles; Glenn A. Crosby, Department of Chemistry (emeritus), Washington State University; David Hammer, Departments of Physics and Curriculum & Instruction, University of Maryland, College Park; Edward J. Hendry, Interdisciplinary Curriculum Specialist; Social Studies/Art/Music, Nashua School District, Nashua, NH; Norman G. Lederman, Department of Mathematics and Science Education, Illinois Institute of Technology, Chicago; Robert Linn, School of Education, University of Colorado; Stephen L. Pruitt, Science and Mathematics, Georgia State Department of Education; Edward D. Roeber, Office of Educational Assessment and Accountability, Michigan Department of Education, Lansing; Norman H. Sleep, Department of Geophysics, Stanford University; Nancy Butler Songer, Science Education and Learning Technologies, University of Michigan; Anne Tweed, Mathematics and Science Division, Mid-continent Research for Education and Learning, Aurora, CO; Carl E. Wieman, Department of Physics, University of Colorado; and Wendy M. Yen, Center for K–12 Research, Educational Testing Service, Monterey, CA.
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations nor did they see the final draft of the report before its release. The review of this report was overseen by Adam Gamoran, Department of Sociology, University of Wisconsin–Madison, and R. Duncan Luce, Institute for Mathematical Behavioral Science, University of California, Irvine. Appointed by the NRC, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.
Mark R. Wilson, Chair
Meryl W. Bertenthal, Study Director
Committee on Test Design for K–12 Science Achievement