Advisers to the Nation on Science, Engineering, and Medicine
National Academy of Sciences
National Academy of Engineering
Institute of Medicine
National Research Council
Center for Education
Board on Testing and Assessment
Board on International Comparative Studies in Education
July 19, 2000
Dr. Gary W. Phillips
National Center for Education Statistics
1990 K Street, N.W., Room 9116 Washington, D.C. 20006
Dr. Judith S. Sunley
Interim Assistant Director
Directorate for Education and Human Resources
National Science Foundation 4201 Wilson Boulevard Arlington, Virginia 22230
Dear Dr. Phillips and Dr. Sunley:
With funding from your agencies, the Board on International Comparative Studies in Education (BICSE) at the National Academies was established in 1988 to advise you on the conduct of the Third International Mathematics and Science Study (TIMSS) and other international comparative studies of education. Since that time, BICSE has been involved with all aspects of TIMSS, from elements of its original design to aspects of its conduct and oversight, and from release of reports to concerns about data availability and secondary analysis. Our recent reports document some of BICSE 's efforts related to TIMSS.1 BICSE continues to be interested in the effects that TIMSS results are having on practice.
The board writes to you now because we believe that there is an urgent need to evaluate the effects of TIMSS on education practice, as well as the process by which TIMSS has been used to try to bring about change in the United States. If, as Mislevy (1995:419) suggests, “in the broadest sense, international assessment is meant to gather information about schooling in a number of countries and somehow use it to improve student learning,”2 then educators need to
2101 Constitution Avenue, NW, Washington, DC 20418 Telephone (202) 334 3010 Fax (202) 334 1294 national-academies.org
Board on International Comparative Studies in Education, Learning from TIMSS: Results of the Third International Mathematics and Science Study, National Academy Press, 1997; Next Steps for TIMSS: Directions for Secondary Analysis, National Academy Press, 1999.
Mislevy, R.J. (1995), What can we learn from international assessments? Educational Evaluation and Policy Analysis 17:410-437.
understand more about this “somehow” – the process by which they can start to move from a rank-ordering of countries to changing learning in classrooms.
The board commends the long-standing collaborative investment and effort that both the Department of Education and the National Science Foundation have made in TIMSS. The study's methodological rigor, sampling and participation rates, as well as the collection of a rich array of explanatory variables all contribute to its value as a research and statistical tool that allows examination and comparison of the United States with other countries.
In addition, recognizing the potential of such international data to affect practice, additional investments were made to ensure data reports that were easier to understand and interpret, to achieve wider and more accurate media coverage, and to promote broad public interest in the results. Publicly released video tapes of two mathematics lessons in each of three countries (Japan, Germany, and the United States) seemed to pique the interest and enthusiasm of their many viewers. Several states and one consortium of districts participated in an experiment with using state- and district-level TIMSS data to affect practice in those places. Production and quick release of materials from TIMSS – from test items to tool kits to the data itself – were facilitated in the hope that both researchers and educators could use them to understand the results more deeply and that practitioners would find them valuable for examining their own practice. Almost daily, stories trickle back, suggesting that administrators in districts and teachers in classrooms are talking the language of international standards in mathematics and science. All these efforts represent creative attempts on an unprecedented scale to use solid international data to inform and influence practice across our nation. Yet evidence of their actual use and effects remains almost entirely anecdotal.
At a recent BICSE meeting, a number of the key participants in the above-mentioned efforts to use TIMSS results to influence practice came to discuss the results of those efforts with the board. For example, we heard from the First in the World Consortium, a group of districts located in Chicago's northern suburbs, that has used TIMSS participation as a key part of its strategy to ensure its students are first in the world in mathematics and science by the year 2000. The strategies these districts have employed to use their TIMSS results include the development of teacher learning networks, summer institutes, lesson-study groups (using TIMSS videotapes), and a topical curriculum framework benchmarked to the highest achieving TIMSS countries by topic. However, no evaluation data are available either to help other districts understand the process of participating in such a collaboration or about the effectiveness of these various strategies in improving classroom practice and student achievement.
Another example is the Department of Education's TIMSS tool kit, which was developed to present the TIMSS data in a way that would be accessible and relevant to a broader range of policy makers and educators than would normally follow international results. The principal audience for the tool kit was the providers of professional development for teachers. The tool kit represented an attempt to translate research findings into a form that would be readily accessible to practitioners. For example, a curriculum module included a guidebook that allows educators at the local level to use TIMSS as a starting point to evaluate their own curricula. The tool kit 's designers hoped that by presenting the information about TIMSS in a clear, consistent manner, it could help to prevent misunderstanding of this complex dataset. Tool kit dissemination efforts have included national conferences to train trainers, as well as efforts on the part of several professional organizations to distribute it to their membership widely. For example, the National Council for Teachers of Mathematics (NCTM) purchased one kit for each of over 200 of their affiliate groups at the state and local level and used them as the focus for the annual leadership meeting. Anecdotal reports suggest that the kit has proved useful to many in the field; yet no systematic evidence is available about how widely the tool kits were distributed, who has used them and how, or what the effects have been of professional development efforts based on the kits.
CONCLUSIONS AND RECOMMENDATIONS
BICSE has come to several conclusions about these efforts:
There is considerable enthusiasm, as well as a lot of activity, directed toward using TIMSS as a strategy for reform and improvement throughout the United States.
From our examination of some of the more visible of these activities, we conclude that they are potentially very useful strategies to pursue.
No research, evaluation, or systematic policy analyses are being conducted to try to document or understand the effects of these strategies. This is a missed opportunity.
A timely commitment to evaluation is important because interest and enthusiasm in international data remain high. As you know, a repeat of TIMSS with the grade 8 cohort (TIMSS-R) was funded and mounted quickly; the achievement data were collected in fall 1999 and the video data have followed. In TIMSS-R, a number of states and districts are participating in a Benchmarking Project and are already meeting together to plan for the release and use of their data in 2001. There is a unique opportunity to understand and learn from these promising efforts to use international data to shape practice at the state and district level.
Such data are needed now so that, in the future, people can understand which of these efforts worked and which did not, how the successful efforts achieved their goals, and the magnitude and nature of the effects they have had. Evaluation will benefit the design of future international studies, making them more useful to reformers, especially to efforts to include states and districts in such data collection. Such information will be needed by those in states
and districts who are participating in the international studies and working on educational improvement. Without such evidence, decisions about future policy strategies and investments – aimed at making international data useful in educational improvement – will be guided by anecdotes, best guesses, and personal preferences. We recommend that the Department of Education and the National Science Foundation commission some well-designed studies that assess the effects of TIMSS on practice: both the process of how TIMSS is being used as well as the effects of those efforts.
An additional investment now in such studies can allow the nation to benefit even more from the major investment that has already been made in the collection of international data about science and mathematics achievement.
Board on International Comparative Studies in Education