National Academies Press: OpenBook
Suggested Citation:"Letter Report." National Research Council. 2000. BICSE Letter Report on Evaluating the Effects of TIMSS. Washington, DC: The National Academies Press. doi: 10.17226/9903.
×
THE NATIONAL ACADEMIES

Advisers to the Nation on Science, Engineering, and Medicine

National Academy of Sciences

National Academy of Engineering

Institute of Medicine

National Research Council

Center for Education

Board on Testing and Assessment

Board on International Comparative Studies in Education

July 19, 2000

Dr. Gary W. Phillips

Acting Commissioner

National Center for Education Statistics

1990 K Street, N.W., Room 9116 Washington, D.C. 20006

Dr. Judith S. Sunley

Interim Assistant Director

Directorate for Education and Human Resources

National Science Foundation 4201 Wilson Boulevard Arlington, Virginia 22230

Dear Dr. Phillips and Dr. Sunley:

With funding from your agencies, the Board on International Comparative Studies in Education (BICSE) at the National Academies was established in 1988 to advise you on the conduct of the Third International Mathematics and Science Study (TIMSS) and other international comparative studies of education. Since that time, BICSE has been involved with all aspects of TIMSS, from elements of its original design to aspects of its conduct and oversight, and from release of reports to concerns about data availability and secondary analysis. Our recent reports document some of BICSE 's efforts related to TIMSS.1 BICSE continues to be interested in the effects that TIMSS results are having on practice.

The board writes to you now because we believe that there is an urgent need to evaluate the effects of TIMSS on education practice, as well as the process by which TIMSS has been used to try to bring about change in the United States. If, as Mislevy (1995:419) suggests, “in the broadest sense, international assessment is meant to gather information about schooling in a number of countries and somehow use it to improve student learning,”2 then educators need to

2101 Constitution Avenue, NW, Washington, DC 20418 Telephone (202) 334 3010 Fax (202) 334 1294 national-academies.org

1  

Board on International Comparative Studies in Education, Learning from TIMSS: Results of the Third International Mathematics and Science Study, National Academy Press, 1997; Next Steps for TIMSS: Directions for Secondary Analysis, National Academy Press, 1999.

2  

Mislevy, R.J. (1995), What can we learn from international assessments? Educational Evaluation and Policy Analysis 17:410-437.

Suggested Citation:"Letter Report." National Research Council. 2000. BICSE Letter Report on Evaluating the Effects of TIMSS. Washington, DC: The National Academies Press. doi: 10.17226/9903.
×

understand more about this “somehow” – the process by which they can start to move from a rank-ordering of countries to changing learning in classrooms.

BACKGROUND

The board commends the long-standing collaborative investment and effort that both the Department of Education and the National Science Foundation have made in TIMSS. The study's methodological rigor, sampling and participation rates, as well as the collection of a rich array of explanatory variables all contribute to its value as a research and statistical tool that allows examination and comparison of the United States with other countries.

In addition, recognizing the potential of such international data to affect practice, additional investments were made to ensure data reports that were easier to understand and interpret, to achieve wider and more accurate media coverage, and to promote broad public interest in the results. Publicly released video tapes of two mathematics lessons in each of three countries (Japan, Germany, and the United States) seemed to pique the interest and enthusiasm of their many viewers. Several states and one consortium of districts participated in an experiment with using state- and district-level TIMSS data to affect practice in those places. Production and quick release of materials from TIMSS – from test items to tool kits to the data itself – were facilitated in the hope that both researchers and educators could use them to understand the results more deeply and that practitioners would find them valuable for examining their own practice. Almost daily, stories trickle back, suggesting that administrators in districts and teachers in classrooms are talking the language of international standards in mathematics and science. All these efforts represent creative attempts on an unprecedented scale to use solid international data to inform and influence practice across our nation. Yet evidence of their actual use and effects remains almost entirely anecdotal.

CURRENT STATUS

At a recent BICSE meeting, a number of the key participants in the above-mentioned efforts to use TIMSS results to influence practice came to discuss the results of those efforts with the board. For example, we heard from the First in the World Consortium, a group of districts located in Chicago's northern suburbs, that has used TIMSS participation as a key part of its strategy to ensure its students are first in the world in mathematics and science by the year 2000. The strategies these districts have employed to use their TIMSS results include the development of teacher learning networks, summer institutes, lesson-study groups (using TIMSS videotapes), and a topical curriculum framework benchmarked to the highest achieving TIMSS countries by topic. However, no evaluation data are available either to help other districts understand the process of participating in such a collaboration or about the effectiveness of these various strategies in improving classroom practice and student achievement.

Suggested Citation:"Letter Report." National Research Council. 2000. BICSE Letter Report on Evaluating the Effects of TIMSS. Washington, DC: The National Academies Press. doi: 10.17226/9903.
×

Another example is the Department of Education's TIMSS tool kit, which was developed to present the TIMSS data in a way that would be accessible and relevant to a broader range of policy makers and educators than would normally follow international results. The principal audience for the tool kit was the providers of professional development for teachers. The tool kit represented an attempt to translate research findings into a form that would be readily accessible to practitioners. For example, a curriculum module included a guidebook that allows educators at the local level to use TIMSS as a starting point to evaluate their own curricula. The tool kit 's designers hoped that by presenting the information about TIMSS in a clear, consistent manner, it could help to prevent misunderstanding of this complex dataset. Tool kit dissemination efforts have included national conferences to train trainers, as well as efforts on the part of several professional organizations to distribute it to their membership widely. For example, the National Council for Teachers of Mathematics (NCTM) purchased one kit for each of over 200 of their affiliate groups at the state and local level and used them as the focus for the annual leadership meeting. Anecdotal reports suggest that the kit has proved useful to many in the field; yet no systematic evidence is available about how widely the tool kits were distributed, who has used them and how, or what the effects have been of professional development efforts based on the kits.

CONCLUSIONS AND RECOMMENDATIONS

BICSE has come to several conclusions about these efforts:

  1. There is considerable enthusiasm, as well as a lot of activity, directed toward using TIMSS as a strategy for reform and improvement throughout the United States.

  2. From our examination of some of the more visible of these activities, we conclude that they are potentially very useful strategies to pursue.

  3. No research, evaluation, or systematic policy analyses are being conducted to try to document or understand the effects of these strategies. This is a missed opportunity.

A timely commitment to evaluation is important because interest and enthusiasm in international data remain high. As you know, a repeat of TIMSS with the grade 8 cohort (TIMSS-R) was funded and mounted quickly; the achievement data were collected in fall 1999 and the video data have followed. In TIMSS-R, a number of states and districts are participating in a Benchmarking Project and are already meeting together to plan for the release and use of their data in 2001. There is a unique opportunity to understand and learn from these promising efforts to use international data to shape practice at the state and district level.

Such data are needed now so that, in the future, people can understand which of these efforts worked and which did not, how the successful efforts achieved their goals, and the magnitude and nature of the effects they have had. Evaluation will benefit the design of future international studies, making them more useful to reformers, especially to efforts to include states and districts in such data collection. Such information will be needed by those in states

Suggested Citation:"Letter Report." National Research Council. 2000. BICSE Letter Report on Evaluating the Effects of TIMSS. Washington, DC: The National Academies Press. doi: 10.17226/9903.
×

and districts who are participating in the international studies and working on educational improvement. Without such evidence, decisions about future policy strategies and investments – aimed at making international data useful in educational improvement – will be guided by anecdotes, best guesses, and personal preferences. We recommend that the Department of Education and the National Science Foundation commission some well-designed studies that assess the effects of TIMSS on practice: both the process of how TIMSS is being used as well as the effects of those efforts.

An additional investment now in such studies can allow the nation to benefit even more from the major investment that has already been made in the collection of international data about science and mathematics achievement.

Sincerely yours,

Andrew Porter

Chair

Board on International Comparative Studies in Education

Suggested Citation:"Letter Report." National Research Council. 2000. BICSE Letter Report on Evaluating the Effects of TIMSS. Washington, DC: The National Academies Press. doi: 10.17226/9903.
×
Page 1
Suggested Citation:"Letter Report." National Research Council. 2000. BICSE Letter Report on Evaluating the Effects of TIMSS. Washington, DC: The National Academies Press. doi: 10.17226/9903.
×
Page 2
Suggested Citation:"Letter Report." National Research Council. 2000. BICSE Letter Report on Evaluating the Effects of TIMSS. Washington, DC: The National Academies Press. doi: 10.17226/9903.
×
Page 3
Suggested Citation:"Letter Report." National Research Council. 2000. BICSE Letter Report on Evaluating the Effects of TIMSS. Washington, DC: The National Academies Press. doi: 10.17226/9903.
×
Page 4
Next: Board on International Comparative Studies in Education - Roster »
BICSE Letter Report on Evaluating the Effects of TIMSS Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!