NATIONAL ACADEMY PRESS
2101 Constitution Avenue, N.W. Washington, DC 20418
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. William A. Wulf is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. William A. Wulf are chairman and vice chairman, respectively, of the National Research Council.
The study was supported by Grant No. ESI-9355774 between the National Academy of Sciences and the National Science Foundation. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the organizations or agencies that provided support for this project.
International Standard Book Number 0-309-06428-7
Additional copies of this report are available from
National Academy Press,
2101 Constitution Avenue, N.W., Washington, DC 20418 Call (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area) This report is also available on line at http://www.nap.edu
Printed in the United States of America
Copyright 1999 by the National Academy of Sciences. All rights reserved.
BOARD ON INTERNATIONAL COMPARATIVE STUDIES IN EDUCATION
Michael W. Kirst (Chair),
School of Education, Stanford University
Gordon M. Ambach (Ex officio),
Council of Chief State School Officers, Washington, D.C.
Christopher T. Cross,
Council for Basic Education, Washington, D.C.
John A. Dossey,
Department of Mathematics, Illinois State University
Ronald K. Hambleton,
School of Education, University of Massachusetts
Ruth E.S. Hayhoe,
Hong Kong Institute of Education, Hong Kong
Paul G. LeMahieu,
Delaware Education Research and Development Center, University of Delaware and Delaware Department of Education
Mary M. Lindquist,
School of Education, Columbus State University
Marlaine E. Lockheed,
The World Bank, Washington, D.C.
Lynn W. Paine,
Department of Teacher Education, Michigan State University
Andrew C. Porter,
Wisconsin Center for Educational Research, School of Education, University of Wisconsin, Madison
Francisco O. Ramirez,
School of Education, Stanford University
Patricia L. Morison, Director
Alexandra Beatty, Program Officer
M. Jane Phillips, Senior Project Assistant
BOARD ON TESTING AND ASSESSMENT
Robert L. Linn (Chair),
School of Education, University of Colorado
Carl F. Kaestle (Vice Chair),
Department of Education, Brown University
Richard C. Atkinson, President,
University of California
Iraline G. Barnes,
Washington, D.C.
Paul J. Black,
School of Education, King's College, London
Richard P. Durán,
Graduate School of Education, University of California, Santa Barbara
Christopher F. Edley, Jr.,
Harvard Law School
Paul W. Holland,
Graduate School of Education, University of California, Berkeley
Michael W. Kirst,
School of Education, Stanford University
Alan M. Lesgold,
Learning Research and Development Center, University of Pittsburgh
Lorraine M. Mcdonnell,
Department of Political Sciences, University of California, Santa Barbara
Kenneth Pearlman,
Lucent Technologies, Inc., Warren, N.J.
Paul R. Sackett,
Industrial Relations Center, University of Minnesota
Richard J. Shavelson,
School of Education, Stanford University
Catherine E. Snow,
Graduate School of Education, Harvard University
William L. Taylor, Attorney at Law,
Washington, D.C.
William T. Trent,
Office of the Chancellor, University of Illinois
Jack Whalen,
Xerox Palo Alto Research Center, Palo Alto, Calif.
Kenneth I. Wolpin,
Department of Economics, University of Pennsylvania
Michael J. Feuer, Director
Viola Horek, Administrative Associate
Preface
The Board on International Comparative Studies in Education (BICSE) was established by the National Research Council (NRC) in 1988 at the request of the U.S. Department of Education's National Center for Education Statistics and the National Science Foundation. The board's mandate over the past nine years has been to monitor the overall quality of international comparative studies, and to foster understanding about how such comparisons can be developed and used. Not surprisingly, the board has devoted considerable energy to monitoring each phase of The Third International Mathematics and Science Study (TIMSS), a study conducted by the International Association for the Evaluation of Educational Achievement. BICSE has followed debates about conceptual, technical, and management issues and has offered both counsel and a forum for discussion as the study has proceeded. In early 1997, as the first study results were released, the board cosponsored (with several other NRC bodies) a symposium at which leading scholars, policy makers, and TIMSS researchers reflected on the first set of results, methodological issues raised by the study, and the study's implications for education reform (see Learning from TIMSS: Results of the Third International Mathematics and Science Study, National Research Council, 1997).
Now that the initial results have been released, the board has turned its attention to what happens next. The TIMSS data are potentially useful to researchers, policy makers, practitioners, and others interested in evidence regarding factors that influence student learning. But although the study has produced a remarkable volume of intriguing data, it is by no means complete. Scholarly review of the initial data, evaluations of claims based on the data, and follow-up secondary analysis based on the primary findings are all integral parts of a study of this magnitude, but the bulk of this very important work has not yet begun. Because of the board's serious concern that this necessary work has not been undertaken, or funded, it held a workshop on June 17 and 18, 1998, to explore different perspectives on possible next steps.
The workshop was an invaluable opportunity for the board to explore issues and questions it has addressed over the years and to solidify its thinking about many of them. Because the board is convinced of the importance of moving forward with the TIMSS data, it presents in this report both recommendations as to what ought to be done and many of the innovative specific ideas that emerged from the workshop. These recommendations reflect the board's conviction, based on its many years of involvement with and deliberations about TIMSS, that this study is an extremely rich resource for the policy, scholarly, and practice communities, and that all of these
groups have a responsibility to take full advantage of it. The recommendations and discussion in this report are intended to assist both researchers and funders who are considering further work with TIMSS, and a broader audience of researchers, policy makers, practitioners, and others who have followed the TIMSS results and are eager to use them. This report is, in a sense, the culmination of many years of effort for the board.
On behalf of the entire board, I extend particular thanks to a number of people whose help in this undertaking has been invaluable. Two board members, Lynn Paine and Francisco Ramirez, took the lead in framing the issues the board needed to address and in planning the workshop. They devised a workshop format that would make stimulating interactions among a diverse group, and on a complex set of issues, possible. It was challenging for the planners and participants alike, but the results were well worth it. Two other board members, Andrew Porter and John Dossey, also played key roles in the planning and in the workshop itself, leading discussions and providing support throughout the process. The board also owes a debt of gratitude to Harry Judge and Stephen Heynemann, who, by serving as rapporteurs at the workshop, took on the challenge of helping the board to synthesize the discussion. Support from staff members Patricia Morison, who has guided the board throughout the process, and Alix Beatty, who assisted with the workshop planning and drafted this document, is also gratefully acknowledged. Jane Phillips provided very able administrative support throughout the workshop, as she does for all BICSE activities.
BICSE carries out its work under the auspices of the NRC's Board on Testing and Assessment and several of its members have assisted in shaping the workshop and report. BICSE is particularly grateful to Richard Shavelson, former chair of BOTA, for his help at several stages of the process. Thanks are also due to our sponsors, NCES and NSF, for their support during the planning of the workshop. In particular, Eugene Owen at NCES and Larry Suter at NSF have been instrumental in facilitating BICSE's involvement in important issues throughout the design and execution of TIMSS.
The key to the success of this board project, however, lay in the contributions of 13 scholars who grounded the discussion with thoughtfully written responses to questions the board had framed. The board extends its deep appreciation to David Baker, Douglas Grouws, Jeremy Kilpatrick, Elizabeth King, Gerald LeTendre, Mary Metz, Heinrich Mintrop, Richard Murnane, Gary Natriello, Aaron Pallas, Senta Raizen, Mavis Sanders, and James Shymansky. By supplying their responses in advance of the workshop so that all participants could read them before convening, these scholars made it possible for the group to plunge quickly into substantive discussions of the issues about which the board was concerned. This diverse group also broadened the discussion and provided countless reminders of the many perspectives
that need to be considered. The board asked a number of individuals who had not previously studied the TIMSS results to take on this task, and particularly appreciates their willingness to review a large volume of material in preparation for the workshop.
I also wish to thank all of the board members for the thoughtful ideas and stimulating discussions that led to the workshop and the production of this document. Their persistence in both tracking the complexities of TIMSS and thinking carefully about its implications over the years has yielded many valuable contributions to the process.
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC's Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making the published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.
We wish to thank the following individuals for their participation in the review of this report: Paul J. Black, School of Education, King's College, London; Andrea A. diSessa, Graduate School of Education, University of California, Berkeley; Edward Hiler, vice chancellor for agriculture and life sciences, Texas A&M University; Michael Martinez, Education Department, University of Calfornia, Irvine; Stephen W. Raudenbush, Department of Counseling, Educational Psychology, and Special Education, Michigan State University; Catherine Snow, Graduate School of Education, Harvard University; and P. Michael Timpane, RAND, Washington, D.C. While the individuals listed above have provided constructive comments and suggestions, it must be emphasized that responsibility for the final content of this report rests entirely with the authoring board and the institution.
Michael W. Kirst, Chair
Board on International Comparative Studies in Education