CHAPTER SIX
Frequently Asked Questions about TIMSS

TIMSS has produced a storehouse of information that will take many years to analyze thoroughly. It also has provided a snapshot of the educational systems in the United States and many other countries. As these systems change in future years, the results of TIMSS will provide a baseline against which improvements can be measured.

This report highlights many of the key findings from TIMSS and relates those findings to the implementation of standards-based education in the United States. The goal of this report is not to recommend specific courses of action or to define a research agenda that can address remaining questions. Instead, by offering information from TIMSS as an analytical tool, this report seeks to further local improvements in schools made by the many different individuals with a stake in U.S. education.

This final chapter offers a set of questions that are often asked about TIMSS along with brief responses to those questions.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 80
Global Perspectives for Local Action: Using TIMSS to Improve U.S. Mathematics and Science Education CHAPTER SIX Frequently Asked Questions about TIMSS TIMSS has produced a storehouse of information that will take many years to analyze thoroughly. It also has provided a snapshot of the educational systems in the United States and many other countries. As these systems change in future years, the results of TIMSS will provide a baseline against which improvements can be measured. This report highlights many of the key findings from TIMSS and relates those findings to the implementation of standards-based education in the United States. The goal of this report is not to recommend specific courses of action or to define a research agenda that can address remaining questions. Instead, by offering information from TIMSS as an analytical tool, this report seeks to further local improvements in schools made by the many different individuals with a stake in U.S. education. This final chapter offers a set of questions that are often asked about TIMSS along with brief responses to those questions.

OCR for page 80
Global Perspectives for Local Action: Using TIMSS to Improve U.S. Mathematics and Science Education What Information Was TIMSS Designed to Gather? TIMSS gathered information in five different categories. It gave timed paper-and-pencil tests to more than half a million children in 41 different countries to test their understanding and skills in mathematics and science (a more limited number of students also participated in a performance assessment consisting of 12 hands-on tasks). It administered questionnaires to students, teachers, and administrators to gauge such factors as student attitudes toward mathematics and science, teacher backgrounds, and school policies and practices. It analyzed more than 1,000 textbooks and curriculum guides from participating countries to investigate curricular patterns. It videotaped a total of 231 representative eighth-grade mathematics classrooms in the United States, Germany, and Japan to analyze mathematics teaching in practice. And it conducted case studies of educational policies and practices in the same three countries. Which Groups of Students Were Studied? TIMSS assessed the achievement of students at three stages of their education. At the elementary school level, TIMSS administered achievement tests to students in the two adjacent grades containing the most 9 year olds (a group referred to as population 1, corresponding to grades three and four in the United States). At the lower secondary school level, students were studied in the two adjacent grades containing the most 13 year olds (population 2, or seventh and eighth grade in the United States). At the upper secondary school level, TIMSS assessed the mathematical and scientific proficiency of students in their last year of secondary school (population 3, corresponding to U.S. high school seniors). In a Nutshell, How Did U.S. Students Compare to Their International Peers on the Student Achievement Assessments in Mathematics and Science? The youngest U.S. students had the highest comparative achievement. In elementary school science and mathematics, U.S. students scored near the top of all students in science and among a band of countries scoring above the international average in mathematics. U.S. middle school students continued to exceed the international average in science but fell below it in mathematics. High school seniors in the United States were below the international average in general knowledge of both subjects and even further behind in separate tests of physics and advanced mathematics. Another way to compare is to ask what percentage of each nation's students rank among the top 10 percent internationally. In elementary school mathematics, 9 percent of U.S. fourth graders would make that cut—almost a representative share but far fewer than in the strongest countries; 39 percent of the upper-grade population 1 students in Singapore are in the top 10 percent internationally. In elementary school science, 16 percent of U.S. fourth graders make the top 10 percent. In middle school, 13 percent of U.S. eighth graders are in the top echelon for science, and 5 percent are for mathematics—again, well behind top-ranked Singapore, which had 45 percent of its

OCR for page 80
Global Perspectives for Local Action: Using TIMSS to Improve U.S. Mathematics and Science Education upper-grade students rank in the top 10 percent internationally in mathematics. The pattern revealed by TIMSS has essentially held true in international comparisons over the past decade and a half—not stellar performance, not last, and stronger among younger students, especially in science. In fact, it is unlikely that U.S. students' standing has changed much over the past 30 years. In What Subjects or Educational Characteristics Is the United States Strongest Relative to Other Nations? Where Is It Weakest? U.S. strengths tend to match what is emphasized in U.S. curricula. U.S. elementary and middle school students do comparatively well in earth science, life science, and environmental issues and the nature of science. They do less well in physical science at this age, and performance in this area continued to fade in later years. By the final year of secondary school, U.S. scores in physics are among the lowest of the nations participating in TIMSS. In mathematics, U.S. elementary and middle school students are comparatively strong in number sense and data representation and analysis, with the younger group also excelling in patterns, relations, functions, and geometry. U.S. students are weak in measurement at both grade levels, and from elementary school to middle school performance in geometry moves from above to below the international average. One explanation suggested by TIMSS for U.S. students' slide in rank is that they are not challenged enough. Eighth-grade mathematics in the United States is covered in seventh grade in high-achieving countries, and U.S. mathematics teachers demand less sophisticated thinking than do teachers in Germany and Japan. As a result, U.S. students learn more slowly than is the norm abroad, making less progress from one grade to the next than students in most countries. Did TIMSS Compare All U.S. Students with Just the Best Students in Other Countries? The designers of TIMSS sought to have all countries, as much as possible, test comparable groups of students. Drawing on critiques of past international comparisons, they established rigorous new procedures to make sure that students taking the tests were randomly selected to represent all students in their nations. An international committee scrutinized the selection process in each country for compliance with these procedures. When nations did not-meet the standards—for example, because too many schools, teachers, or students declined to participate—these exceptions were noted to allow researchers to take them into account. Even so, meeting the selection criteria did not necessarily disadvantage a country in the rankings. Countries that met the criteria for the fourth-and eighth-grade tests included Korea and Japan, which generally outperformed the United States, as well as the Czech Republic, Hong Kong, and Singapore, which performed well in many areas. The concern that all U.S. students are being compared unfairly to elite students abroad runs deepest in regard to the last year of secondary school. The conventional wisdom is that in some other countries only the best students are still in school at that age. In fact, to the extent

OCR for page 80
Global Perspectives for Local Action: Using TIMSS to Improve U.S. Mathematics and Science Education that biases can be identified in the population 3 samples, most of them should have favored the United States. For example, the United States has proportionally fewer 17 year olds enrolled than the average of other TIMSS countries with enrollment data. Since students who drop out presumably tend to be lower achievers, this phenomenon may enhance the relative rank of the United States. It is true that most countries participating in the end-of-secondary school tests, including the United States, did not meet the selection criteria for representative samples. Nevertheless, important conclusions can be drawn from the results. The fact that some of the highest-achieving U.S. students had not taken equally advanced courses as their foreign peers is itself a point to consider. Furthermore, a particularly weak area of performance among seniors was in geometry, a curricular area that is emphasized in U.S. high schools. Aren't Students at the End of Secondary School Too Varied in Age, from One Country to Another, to Compare Them? The U.S. students in population 3 were somewhat younger than the international average: 18.1 years old compared to an average of 18.7 among all 21 nations participating in this level of TIMSS achievement testing. The disparity was half as great for the subset of students who took the advanced science and mathematics tests: the average U.S. age was 18.0, while the average age among all nations participating in this special test was 18.4 years for physics and 18.3 years for mathematics. In addition, age was not strictly correlated with performance. Australia and New Zealand did better than the United States on the mathematics and science general assessment, but their students were on average younger than those in the United States. The performance of Russian students was on a par with that of U.S. students despite being a year younger and having had a year less of formal schooling. Several other factors also minimize the importance of the age differences among students in population 3. First, the TIMSS general tests for secondary school seniors are essentially literacy tests, not examinations on advanced knowledge. The mathematics topics were similar to topics covered by the seventh grade in most countries, and the science topics generally were covered by the ninth grade. High school seniors all should have been exposed to the material. Finally, unlike the tests of the two younger populations, the test of high school seniors was not intended to compare students of the same age. Secondary school graduation is a turning point in all countries—the threshold of adulthood and of important choices among work and further study. International variation in the ages and knowledge levels at which adolescents cross that threshold is not only inevitable but interesting in and of itself. Isn't TIMSS Just a ''Horserace'' That Puts Too Much Emphasis on Test Scores Without Revealing Substantive Insights into Our Educational System? Some people may choose to focus selectively on the quantitative country scores derived from the TIMSS data, but TIMSS as a whole pro-

OCR for page 80
Global Perspectives for Local Action: Using TIMSS to Improve U.S. Mathematics and Science Education duced much more than just measures of student achievement. It yielded qualitative and quantitative information about the attitudes, lifestyles, teaching, and educational policies that contribute to academic strengths and weaknesses in each country. As such, the study is a tool for comparing educational policies and practices and finding the roots of relatively strong and weak performance. At the same time, even the achievement test scores that prompt the "horserace" complaint are valuable because they demonstrate a shortfall between U.S. performance and that of the highest-achieving countries. Furthermore, TIMSS presented the country scores in a more statistically meaningful way than the typical newspaper summary, grouping countries in broad bands of achievement instead of making much of small differences between countries that are effectively on a par with each other. Don't Cultural Differences among the TIMSS Countries Render Test Score Comparisons Meaningless? Differences are what make TIMSS valuable and interesting. Cultural differences are an argument against simplistic interpretations of TIMSS, not an argument against TIMSS itself. Data from the student questionnaires illustrate why. For example, U.S. high school seniors study less, with more optimism, than do their peers in many countries. TIMSS does not say these cultural traits determine our standing internationally, but the data bring to light possible contributing factors and suggest questions for further research. Educational goals and philosophies, social values, school safety, teachers' ideas about student ability, how students spend their time—countless factors that stem from culture affect student performance. One goal of TIMSS was therefore to identify and examine these differences. Did TIMSS Prove That the U.S. Curriculum Is "A Mile Wide and an Inch Deep"? On the face of it, TIMSS data suggest that U.S. schools cover an exceptionally high number of topics every year at most levels relative to mathematics and science classes in other countries. This has led educators to ask: Is U.S. teaching more superficial? Would U.S. students do better with fewer topics covered in more depth? U.S. science and mathematics textbooks do cover many more topics than is typical in other countries' books. Furthermore, the United States tends to repeat topics over more years than do other countries, at least in mathematics. The data sketch a picture of repeated exposure without time for mastery. However, the overall picture is not necessarily black and white. For one thing, the number of topics covered is not necessarily a sign of bad teaching. A teacher who covers many topics may be spending a lot of time on a few topics while briefly touching on others. A teacher also may cover many topics to draw connections among different areas of the curriculum. Or a teacher who revisits topics may go deeper each time. Other indicators, however, continue to point to problems in U.S. curricula. The videotapes of eighth-grade mathematics classes in three

OCR for page 80
Global Perspectives for Local Action: Using TIMSS to Improve U.S. Mathematics and Science Education countries showed U.S. teachers presenting less sophisticated topics and evoking a lower level of student reasoning than their German and Japanese peers. Depth of coverage also may suffer from the fact that U.S. teachers changed topics more often than did teachers in Germany and Japan, and their lessons were more likely to be interrupted. What Did TIMSS Discover about the Conventional Ways of Teaching in Different Countries? Teaching practices reflect particular instructional goals, beliefs about science and mathematics and how they are learned, and assumptions about what a normal lesson looks like. Lessons vary, obviously, but TIMSS suggests that each culture may have typical "scripts" for teaching mathematics and science. As demonstrated in the videotapes and case studies, the typical U.S. script emphasizes incremental mastery of a sequence of skills. A frustrated student is a signal that previous material was not conveyed adequately. In Japan, in contrast, frustration can be normal and useful in a lesson. Students there may be asked to invent ways to solve a new problem, and their struggle sets the stage on which the teacher brings out new concepts and shows relationships among concepts and facts. In each country a typical sequence of actions carries this script forward. In a U.S. middle school mathematics class, the teacher typically reviews previous material, often by checking homework; then demonstrates how to solve the day's new problems; has the students practice on their own; and, finally, corrects their practice problems. The equivalent teacher in Japan reviews by lecturing or questioning the students, poses problems and has students work on them individually and perhaps in groups, leads a class discussion of how to solve the problems, and concludes by clarifying and summarizing the main concepts. Germany is closer in practice to the United States than Japan. German and U.S. students do mathematics by following the teacher's lead. Japanese students do the same at times, but at other times their job is to think creatively about the subject. Open-ended questions and close-ended quizzing both have their place in Japan, but teachers in Germany and the United States do much more of the latter. Consistent with the U.S. emphasis on skill, U.S. teachers give more time to review than in the highest-achieving countries. Most U.S. students received only 10 minutes of instruction on new material in a typical eighth-grade mathematics class, and a similar pattern seems to hold true for science. U.S. teachers also tend to vary lessons by changing topics, while Japanese teachers are more apt to stay on one topic but shift often from classwork to individual work to small-group work and back again. U.S. elementary and middle school teachers are unusual in giving students class time to begin their homework, and they give more quizzes than other teachers. Did TIMSS Find Any One Factor That Causes Higher Student Performance? This is what everyone would love to find—a sure-fire prescription for improving education. TIMSS demonstrates that there isn't one. Education and learning involve countless

OCR for page 80
Global Perspectives for Local Action: Using TIMSS to Improve U.S. Mathematics and Science Education variables, from teacher education and parental attitudes to class size and students' after-school jobs. The variables all affect learning, and no single one has been shown to be overwhelmingly influential. Education is generally too complex to link causes and effects conclusively. TIMSS turned up many counterexamples for arguments attempting to ascribe achievement to particular variables. For instance, while many people believe that smaller class size is associated with higher achievement, classes in Korea, one of the top-performing countries, average more than 40 students. The results of TIMSS also reiterate that some factors affecting achievement are not immediately within schools' control. For example, parents' education levels and the number of books in the home were related to student achievement.