Click for next page ( 124


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 123
6 Subject-Matter Experts' Perceptions of the Relevance of the NAEP Long-Term Trend Items in Science and Mathematics Jennifer R. Zieleskiewicz The National Assessment of Educational Progress (NAEP), a congression- ally mandated project of the National Center for Education Statistics (NCES), evaluates American students' educational accomplishments in a variety of disci- plines. Since 1969, assessments have been conducted periodically in reading, mathematics, science, writing, and other subjects. There are two types of trend assessments at the national level: the main NAEP or short-term trend NAEP assessments and the long-term trend NAEP assessments. The main NAEP assess- ments are given every few years and were designed to adapt to changes in assess- ment approaches. The long-term trend NAEP assessments were designed to be stable and measure specific trends in educational performance over time. Because the long-term trend items were developed well before the current main NAEP frameworks and assessments, some have questioned whether the long-term trend items are up-to-date and relevant measures of student achieve- ment. Thus, the purpose of this study was to evaluate long-term trend NAEP for its relevance compared with the classroom, national standards, and main NAEP assessments using item-level data. This study is only a first step toward evaluat- ing the relevance of long-term trend NAEP. BACKGROUND The NAEP assessments used in this study were the main NAEP and the long- term trend NAEP assessments for mathematics and science. The main NAEP mathematics assessment was first administered in 1990 and then again in 1992 and 1996. The main NAEP science assessment was first administered in 1996. 123

OCR for page 123
24 LONG-TERM TREND ITEMS IN SCIENCE AND MATHEMATICS The long-term trend NAEP assessments are based on a collection of items origi- nally administered in NAEP assessments during the 1970s and 1980s. The cur- rent form of long-term trend NAEP was first administered in 1988 and has been administered every two years through 1999. Both the mathematics and science NAEPs are scheduled to be administered in 2000. The development processes were different for each of the NAEP trend assess- ments. The main NAEP trend assessments framework was developed through a consensus-building process that resulted in a very specific definition of what is important to assess in education. Prior NAEP evaluators provided evidence of relevance by evaluating the framework; relying on groups of subject-matter experts to provide evidence that the consensus process resulted in a "good" and current framework; and conducting item classification/content congruence stud- ies, such as the Sireci et al. analysis of the 1996 NAEP science assessment (Chapter 4, this volume), to show whether the items reflect the goals in the framework. However, long-term trend NAEP was not based on a consensus- building process or development of frameworks but was based on educational standards from the 1950s. Thus, for the purposes of this study it was suggested that some of the long-term trend NAEP items be evaluated to provide evidence that the long-term trend assessment was a "good" and current method of evaluat- ing education, like that of main NAEP. Researchers were interested in answering the following questions: 1. Do the long-term trend items in mathematics and science adequately assess knowledge and skills taught in grade 8 classrooms today? 2. Do the long-term trend items in mathematics and science adequately assess what national standards should be taught in grade 8 classrooms today? 3. How do long-term and main NAEP items compare with one another? A survey was developed to assess these research questions. METHOD Sample The survey sample consisted of two groups of 30 raters selected to evaluate the mathematics and science items. Each discipline had three groups of raters, containing 10 participants each. The three groups were eighth-grade district faculty, eighth-grade reform faculty, and disciplinary specialists. The groups were defined as follows: Eighth-Grade District Faculty Eighth-grade schoolteachers who were representative of typical schools and classrooms were used as raters. The participating states were chosen based on the

OCR for page 123
JENNIFER R. ZIELESKIEWICZ 125 random selection of one state from each of the Census Bureau divisions. These states were Kansas, Arkansas, Indiana, Georgia, New York, Oregon, Idaho, South Dakota, Vermont, and Alabama. One school district was randomly selected from each of these states, regardless of population size. All participating school dis- tricts are identical for the mathematics and science surveys, but for one instance in the science survey, where the Platt school district in South Dakota was used. The mathematics and/or science coordinator for each school district was contacted and asked to nominate an eighth-grade school teacher for each discipline. The coordinator was asked to nominate someone believed to be a good representative of the science or mathematics taught in the school district and who would also be interested in participating in a study on education. Eighth-Grade Reform Faculty Eighth-grade schoolteachers currently in the classroom or on leave, who had some knowledge of local and state reform issues, were selected to participate as raters. The National Research Council (NRC) provided the names and contact information for the science participants in this group because they participated in a prior study in the summer of 1997. The states included were Maryland, Florida, Virginia, Ohio, Kentucky, Texas, Delaware, California, Washington, and South Carolina. From these same states the NRC provided the names of state assess- ment directors who were asked to nominate a mathematics teacher with some knowledge of local and state reform issues. Disciplinary Specialists Disciplinary specialists consisted of professionals, such as university profes- sors and leaders in professional organizations, who are considered experts in national mathematics or science standards assessments, were nominated by the NRC. Disciplinary specialists for science were from the University of Minnesota; Assessment Curriculum & Teaching Systems; Science Examination for New Standards Project in California; Bedford, New York Public Schools; University of Oklahoma; Alaska Department of Education; West ED; University of Califor- nia; Vanderbilt University; and Colorado Department of Education. Disciplinary specialists for mathematics were from the University of Wisconsin, University of Iowa, University of Georgia, University of Delaware, Michigan State University, Colgate University, University of California at Los Angeles, Connecticut Depart- ment of Education, and San Diego State University. Instruments Two surveys were developed: one for mathematics and one for science. Each included a subset of items from the long-term trend NAEP assessments as well as the released items from the main NAEP assessments.

OCR for page 123
126 The Math Survey LONG-TERM TREND ITEMS IN SCIENCE AND MATHEMATICS There were 22 main NAEP and 37 trend mathematics items, making a total of 59 items for the mathematics survey. The main NAEP items were released items from the 1990, 1992, and 1996 assessments. The trend items were from the current long-term trend assessments. All selected items from the main and long- term trend assessments were randomly ordered in the survey. The Science Survey There were 16 main NAEP and 25 trend science items, making a total of 41 items for the science survey. The main NAEP science items were released from the 1996 assessment. The trend items were from the current trend assessment. All selected items from the main and long-term trend assessments were randomly ordered in the survey. Survey Questions For each item a set of two or four questions was asked of each rater. Questions A and B Questions A and B provided information to answer the question, "Do the long-term trend items in mathematics and science adequately assess knowledge and skills taught in today's grade 8 classrooms?" If the rater answered "no" to question A, the rater did not proceed to question B. But if the rater answered "yes," they did proceed to B (see Appendix). Questions C and D Questions C and D provided information to answer the question, "Do the long-term trend items in mathematics and science adequately assess what national standards state should be taught in today's grade 8 classrooms?" If the answer was "no" to question C, the rater did not proceed to question D. But if the answer was "yes," they did proceed to D (see Appendix). The eighth-grade district faculty and the eighth-grade reform faculty were given questions A through D, whereas the disciplinary specialists were given only questions C and D. The two groups of faculty were given the four questions because this information requires eighth-grade teaching experience. The disci- plinary specialists were given only questions C and D because it was thought they would have a good understanding of their field and educational national standards but did not currently teach eighth grade. Surveys were sent out and participants

OCR for page 123
JENNIFER R. ZIELESKIEWICZ 127 were asked to return the materials within two weeks of receipt. Sixty surveys were mailed, and there was a 100 percent response rate. Statistical Analysis and Results Means and standard deviations were computed for each group in mathematics and science for the main and long-term NAEPs. It should be noted that, while the sample size for questions A and C is either 8, 9, or 10 for each of the subgroups, the sample sizes are smaller for questions B and D and vary from item to item. The reason for the reduced and variable sample sizes is that only those who answered "yes" to question A(C) answered question B(D). Since there is no direct item-level correspondence between the main and long-term NAEP items, the mean and standard deviations of item means were computed for items in the main and long-term NAEP, separately, for comparison across the two NAEPs. Table 6-1 displays means for mathematics for questions A through D, separately for the three groups. Similar information is shown for science in Table 6-2. TABLE 6-1 Means and Standard Deviations of Responses to Questions A, B. C, and D Across Groups and Combined for Mathematics District Reform Disciplinary Teachers Teachers Specialists Combined Question Short- Long- Term Term (22)C (37) 0.832 0.946 0.164 0.128 10 9-10 Short- Long Term Term (22) (37) 0.954 0.061 Short- Long- Term Term (22) (37) Short- Long- Term Term A Meana SD N (Range)b B C D 0.940 0.050 9-10 X X X X 0.893 0.943 0.085 0.058 Mean 3.916 4.252 4.174 3.876 X X 4.045 4.064 SD 0.438 0.430 0.390 0.298 X X 0.324 0.305 N 6-10 4-10 8-10 9-10 Mean 0.941 0.911 0.968 0.943 0.950 0.915 0.953 0.923 SD 0.067 0.062 0.048 0.051 0.074 0.084 0.039 0.035 N 10 9-10 9-10 9-10 9-10 9-10 Mean 4.043 4.375 4.155 3.820 3.937 4.045 4.045 4.080 SD 0.356 0.386 0.347 0.277 0.485 0.314 0.272 0.225 N 8-10 8-10 9-10 9-10 8-10 6-10 NOTE: Means and standard deviations by item are also available by short-term and long-term, respectively. aMean of item means and standard deviation of item means. bNumber of respondents for questions B and D are conditional on responses to questions A and C. See Appendix for exact wording of these questions. CNumber of items.

OCR for page 123
28 LONG-TERM TREND ITEMS IN SCIENCE AND MATHEMATICS TABLE 6-2 Means and Standard Deviations of Responses to Questions A, B. C, and D Across Groups and Combined for Science District Reform Disciplinary Teachers Teachers Specialists Combined Question Short- Long- Term Term (16)C (25) 0.724 0.824 0.173 0.188 9-10 10 4.067 3.943 0.447 0.364 4_9 4-10 Short- Long Term Term (16) (25) 0.869 0.917 0.087 0.161 10 9-10 Short- Long- Term Term (16) (25) Short- Long- Term Term A Meana SD N (Range)b B C D Mean SD N X X 3.994 3.959 0.400 0.342 2-10 X X X X X X 0.797 0.870 0.121 0.156 4.030 3.951 0.400 0.221 Mean 0.831 0.868 0.900 0.873 0.863 0.868 0.865 0.870 SD 0.154 0.144 0.097 0.149 0.154 0.111 0.101 0.101 N 9-10 10 10 9-10 10 9-10 Mean 3.991 3.986 4.070 4.044 4.027 3.617 4.029 3.882 SD 0.449 0.325 0.388 0.313 0.459 0.477 0.314 0.217 N 5-10 5-10 7-10 5-10 5-10 7-10 NOTE: Means and standard deviations by item are also available by short-term and long-term, respectively. aMean of item means and standard deviation of item means. bNumber of respondents for questions B and D are conditional on responses to questions A and C. See Appendix for exact wording of these questions. CNumber of items. Means in Tables 6-1 and 6-2 are quite similar for main and long-term NAEP. The means for questions A and C vary between .724 and .968, with most of the means in the .80s and .9Os. Standard deviations for questions A and C range from .048 to .188. The differences between means for A and C range from zero to 10 percentage points for science, with most at five percentage points or less. The differences between means for A and C range from zero to 10 percentage points for mathematics, with most at three percentage points or less. These high and comparable mean results indicate that study participants strongly feel that both frameworks, as reflected in the selected items, adequately assess mathematics and science concepts taught in today's classrooms as well as what national stan- dards say should be taught in today's classrooms. The results also indicate a high degree of comparability across the two NAEPs. Means for questions B and D are around 4.0, again indicating that study participants strongly feel that both frameworks, as reflected in the selected items, adequately assess the importance of mathematics and science concepts taught in

OCR for page 123
JENNIFER R. ZIELESKIEWICZ 129 today's classrooms as well as what national standards say should be taught. Standard deviations for questions B and D range from .277 to .485. The results also indicate a high degree of comparability across the two NAEPs. The differ- ences between the main and the long-term NAEP items are in the second decimal place on a scale ranging from 1 to 5. Because of the small sample sizes involved, no significance tests were performed. DISCUSSION The research conducted on the subject-matter experts' perceptions of the relevance of the NAEP long-term trend items in mathematics and science for eighth graders has provided relevant information on the importance of the long- term trend assessment. These findings can be summarized as follows: The long-term trend and main NAEP mathematics and science items evaluated in this study appear to reflect important content and skills that grade 8 teachers cover in their classes. This subset of items also appears to reflect what grade 8 teachers and disciplinary specialists believe are important parts of national standards in math- ematics and science. For both mathematics and science there do not appear to be any meaning- ful differences in grade 8 teachers' or disciplinary specialists' perceptions of the relevance of long-term trend items and main NAEP items, as measured by cover- age in current classrooms or reflection in national standards. . The results taken as a whole suggest that these long-term trend items are up to date in measuring student achievement. Findings suggest that these long-term trend and main NAEP items reflect what is important for eighth-grade students to know and be able to do as well as what national standards say should be taught in today's eighth-grade classrooms. It also appears that teachers and disciplinary specialists agree on the relevance of the selected long-term trend and main NAEP items as measured by coverage in current classrooms or reflection in national standards. Although this study is only a first step toward evaluating the relevance of the long-term trend NAEP, one could suggest that the long-term trend NAEP, as a whole, may be relevant in today's classroom, reflects current national stan- dards, and is as equally valuable as the main NAEP assessments. However, more research is needed to clarify this conclusion. No meaningful differences were found between rating groups on questions A through D, suggesting that the selected long-term trend and main NAEP items are reflective of what is being taught in the classroom as well as what national standards say should be taught. This implies that these long-term NAEP trend items are not out of date for use in the 1990s and provide the same information as the main NAEP items. These findings also suggest that national standards are

OCR for page 123
130 LONG-TERM TREND ITEMS IN SCIENCE AND MATHEMATICS visible to individuals in the teaching professions, although to what degree is uncertain. It is likely that some teachers will recognize the national standards, but it is unclear whether all of those in the teaching profession have knowledge of them. These results have important implications for the future of the long-term trend assessment. For several decades the NAEP trends have provided informa- tion about the educational achievements of students in American schools. The goal has been to assess information on what American students know and can do in the classroom and to compare their current performance with that of similar students assessed in the past. The main and long-term NAEP trends were designed to assess this achievement but using different frameworks. The results of this study give some suggestion that both this subset of main and long-term NAEPs items are equally valuable in their assessment and provide similar information. Perhaps one day the two trends may be combined into one assessment, eliminat- ing the use of multiple measures over time and simplifying the testing process in general. More research is needed to investigate this possibility. Because the results are preliminary, any interpretation or speculation on their use should be weighed against a small sample size and the use of a subset of trend items. As a result, caution should be taken in generalizing these results to popu- lations outside the parameters of this study. A small sample may not adequately assess the differences in educational content from one state to the next, particu- larly in those states where there is no mandated curriculum or textbook adoption. Also, a subset of trend items was used, thus providing information about those national standards and content reflected only in those items. The results have shown that those items are reflective of what is important for eighth-grade students to know and be able to do as well as what national standards say should be taught in today's eighth-grade classrooms. It would be premature to assume, since only a portion of the national standards and content were represented in this study, that all national standards and content are reflected in classroom curriculum. To learn more about the long-term trend NAEP assessment, research should be performed to focus on several aspects of it. Future research should consider the use of a larger sample of raters and the inclusion of a larger, more representa- tive sample of items from the mathematics and science assessments. It may also be appropriate to include other disciplines in the research. SUMMARY Researchers were interested in knowing if the long-term NAEP is an up-to- date and relevant measure of student achievement in mathematics and science compared with the main NAEP frameworks and assessments developed more recently. Subsets of items from the long-term and main NAEP assessments were combined into two surveys, one for mathematics and one for science, and were sent to 60 subject-matter experts.

OCR for page 123
JENNIFER R. ZIELESKIEWICZ 131 The long-term trend and main NAEP science and mathematics items evalu- ated in this study appear to reflect important content and skills that grade 8 teachers cover in their classes. These select items also appear to reflect what grade 8 teachers and disciplinary specialists believe to be important parts of national standards in science and mathematics. Lastly, in both science and math- ematics there do not appear to be any meaningful differences in grade 8 teachers' or disciplinary specialists' perceptions of the relevance of long-term trend items and main NAEP items as measured by coverage in current classrooms or reflec- tion in national standards. APPENDIX Question A: Does this item assess knowledge and/or skills that students in your school will have covered in science (mathematics) by the end of the eighth grade? (Please check one) No Yes Question B.: If yes, relative to all of what students cover in science (mathematics) in your school, how important is it for students to know/be able to do what is covered in the item by the end of the eighth grade? (Please circle one) 2 Not important 3 Somewhat important 4 5 Very important Question C: Does this item assess knowledge and/or skills that students should have covered in science (mathematics) by the end of the eighth grade according to your best understanding of national standards? (Please check one) No Yes Question D: If yes, relative to all of what national standards say students should cover, how important is it for students to know and be able to do what is covered by this item by the end of the eighth grade? (Please circle one) 2 Not important 3 Somewhat important 4 5 Very important