Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 123
6
Subject-Matter Experts' Perceptions of the
Relevance of the NAEP Long-Term Trend
Items in Science and Mathematics
Jennifer R. Zieleskiewicz
The National Assessment of Educational Progress (NAEP), a congression-
ally mandated project of the National Center for Education Statistics (NCES),
evaluates American students' educational accomplishments in a variety of disci-
plines. Since 1969, assessments have been conducted periodically in reading,
mathematics, science, writing, and other subjects. There are two types of trend
assessments at the national level: the main NAEP or short-term trend NAEP
assessments and the long-term trend NAEP assessments. The main NAEP assess-
ments are given every few years and were designed to adapt to changes in assess-
ment approaches. The long-term trend NAEP assessments were designed to be
stable and measure specific trends in educational performance over time.
Because the long-term trend items were developed well before the current
main NAEP frameworks and assessments, some have questioned whether the
long-term trend items are up-to-date and relevant measures of student achieve-
ment. Thus, the purpose of this study was to evaluate long-term trend NAEP for
its relevance compared with the classroom, national standards, and main NAEP
assessments using item-level data. This study is only a first step toward evaluat-
ing the relevance of long-term trend NAEP.
BACKGROUND
The NAEP assessments used in this study were the main NAEP and the long-
term trend NAEP assessments for mathematics and science. The main NAEP
mathematics assessment was first administered in 1990 and then again in 1992
and 1996. The main NAEP science assessment was first administered in 1996.
123
OCR for page 124
24
LONG-TERM TREND ITEMS IN SCIENCE AND MATHEMATICS
The long-term trend NAEP assessments are based on a collection of items origi-
nally administered in NAEP assessments during the 1970s and 1980s. The cur-
rent form of long-term trend NAEP was first administered in 1988 and has been
administered every two years through 1999. Both the mathematics and science
NAEPs are scheduled to be administered in 2000.
The development processes were different for each of the NAEP trend assess-
ments. The main NAEP trend assessments framework was developed through a
consensus-building process that resulted in a very specific definition of what is
important to assess in education. Prior NAEP evaluators provided evidence of
relevance by evaluating the framework; relying on groups of subject-matter
experts to provide evidence that the consensus process resulted in a "good" and
current framework; and conducting item classification/content congruence stud-
ies, such as the Sireci et al. analysis of the 1996 NAEP science assessment
(Chapter 4, this volume), to show whether the items reflect the goals in the
framework. However, long-term trend NAEP was not based on a consensus-
building process or development of frameworks but was based on educational
standards from the 1950s. Thus, for the purposes of this study it was suggested
that some of the long-term trend NAEP items be evaluated to provide evidence
that the long-term trend assessment was a "good" and current method of evaluat-
ing education, like that of main NAEP. Researchers were interested in answering
the following questions:
1. Do the long-term trend items in mathematics and science adequately
assess knowledge and skills taught in grade 8 classrooms today?
2. Do the long-term trend items in mathematics and science adequately
assess what national standards should be taught in grade 8 classrooms today?
3. How do long-term and main NAEP items compare with one another? A
survey was developed to assess these research questions.
METHOD
Sample
The survey sample consisted of two groups of 30 raters selected to evaluate
the mathematics and science items. Each discipline had three groups of raters,
containing 10 participants each. The three groups were eighth-grade district
faculty, eighth-grade reform faculty, and disciplinary specialists. The groups
were defined as follows:
Eighth-Grade District Faculty
Eighth-grade schoolteachers who were representative of typical schools and
classrooms were used as raters. The participating states were chosen based on the
OCR for page 125
JENNIFER R. ZIELESKIEWICZ
125
random selection of one state from each of the Census Bureau divisions. These
states were Kansas, Arkansas, Indiana, Georgia, New York, Oregon, Idaho, South
Dakota, Vermont, and Alabama. One school district was randomly selected from
each of these states, regardless of population size. All participating school dis-
tricts are identical for the mathematics and science surveys, but for one instance
in the science survey, where the Platt school district in South Dakota was used.
The mathematics and/or science coordinator for each school district was contacted
and asked to nominate an eighth-grade school teacher for each discipline. The
coordinator was asked to nominate someone believed to be a good representative
of the science or mathematics taught in the school district and who would also be
interested in participating in a study on education.
Eighth-Grade Reform Faculty
Eighth-grade schoolteachers currently in the classroom or on leave, who had
some knowledge of local and state reform issues, were selected to participate as
raters. The National Research Council (NRC) provided the names and contact
information for the science participants in this group because they participated in
a prior study in the summer of 1997. The states included were Maryland, Florida,
Virginia, Ohio, Kentucky, Texas, Delaware, California, Washington, and South
Carolina. From these same states the NRC provided the names of state assess-
ment directors who were asked to nominate a mathematics teacher with some
knowledge of local and state reform issues.
Disciplinary Specialists
Disciplinary specialists consisted of professionals, such as university profes-
sors and leaders in professional organizations, who are considered experts in
national mathematics or science standards assessments, were nominated by the
NRC. Disciplinary specialists for science were from the University of Minnesota;
Assessment Curriculum & Teaching Systems; Science Examination for New
Standards Project in California; Bedford, New York Public Schools; University
of Oklahoma; Alaska Department of Education; West ED; University of Califor-
nia; Vanderbilt University; and Colorado Department of Education. Disciplinary
specialists for mathematics were from the University of Wisconsin, University of
Iowa, University of Georgia, University of Delaware, Michigan State University,
Colgate University, University of California at Los Angeles, Connecticut Depart-
ment of Education, and San Diego State University.
Instruments
Two surveys were developed: one for mathematics and one for science.
Each included a subset of items from the long-term trend NAEP assessments as
well as the released items from the main NAEP assessments.
OCR for page 126
126
The Math Survey
LONG-TERM TREND ITEMS IN SCIENCE AND MATHEMATICS
There were 22 main NAEP and 37 trend mathematics items, making a total
of 59 items for the mathematics survey. The main NAEP items were released
items from the 1990, 1992, and 1996 assessments. The trend items were from the
current long-term trend assessments. All selected items from the main and long-
term trend assessments were randomly ordered in the survey.
The Science Survey
There were 16 main NAEP and 25 trend science items, making a total of 41
items for the science survey. The main NAEP science items were released from
the 1996 assessment. The trend items were from the current trend assessment.
All selected items from the main and long-term trend assessments were randomly
ordered in the survey.
Survey Questions
For each item a set of two or four questions was asked of each rater.
Questions A and B
Questions A and B provided information to answer the question, "Do the
long-term trend items in mathematics and science adequately assess knowledge
and skills taught in today's grade 8 classrooms?" If the rater answered "no" to
question A, the rater did not proceed to question B. But if the rater answered
"yes," they did proceed to B (see Appendix).
Questions C and D
Questions C and D provided information to answer the question, "Do the
long-term trend items in mathematics and science adequately assess what national
standards state should be taught in today's grade 8 classrooms?" If the answer
was "no" to question C, the rater did not proceed to question D. But if the answer
was "yes," they did proceed to D (see Appendix).
The eighth-grade district faculty and the eighth-grade reform faculty were
given questions A through D, whereas the disciplinary specialists were given
only questions C and D. The two groups of faculty were given the four questions
because this information requires eighth-grade teaching experience. The disci-
plinary specialists were given only questions C and D because it was thought they
would have a good understanding of their field and educational national standards
but did not currently teach eighth grade. Surveys were sent out and participants
OCR for page 127
JENNIFER R. ZIELESKIEWICZ
127
were asked to return the materials within two weeks of receipt. Sixty surveys
were mailed, and there was a 100 percent response rate.
Statistical Analysis and Results
Means and standard deviations were computed for each group in mathematics
and science for the main and long-term NAEPs. It should be noted that, while the
sample size for questions A and C is either 8, 9, or 10 for each of the subgroups,
the sample sizes are smaller for questions B and D and vary from item to item.
The reason for the reduced and variable sample sizes is that only those who
answered "yes" to question A(C) answered question B(D). Since there is no
direct item-level correspondence between the main and long-term NAEP items,
the mean and standard deviations of item means were computed for items in the
main and long-term NAEP, separately, for comparison across the two NAEPs.
Table 6-1 displays means for mathematics for questions A through D, separately
for the three groups. Similar information is shown for science in Table 6-2.
TABLE 6-1 Means and Standard Deviations of Responses to Questions A, B.
C, and D Across Groups and Combined for Mathematics
District Reform Disciplinary
Teachers Teachers Specialists Combined
Question
Short- Long-
Term Term
(22)C (37)
0.832 0.946
0.164 0.128
10 9-10
Short- Long
Term Term
(22) (37)
0.954
0.061
Short- Long-
Term Term
(22) (37)
Short- Long-
Term Term
A Meana
SD
N (Range)b
B
C
D
0.940
0.050
9-10
X X
X X
0.893 0.943
0.085 0.058
Mean 3.916 4.252 4.174 3.876 X X 4.045 4.064
SD 0.438 0.430 0.390 0.298 X X 0.324 0.305
N 6-10 4-10 8-10 9-10
Mean 0.941 0.911 0.968 0.943 0.950 0.915 0.953 0.923
SD 0.067 0.062 0.048 0.051 0.074 0.084 0.039 0.035
N 10 9-10 9-10 9-10 9-10 9-10
Mean 4.043 4.375 4.155 3.820 3.937 4.045 4.045 4.080
SD 0.356 0.386 0.347 0.277 0.485 0.314 0.272 0.225
N 8-10 8-10 9-10 9-10 8-10 6-10
NOTE: Means and standard deviations by item are also available by short-term and long-term,
respectively.
aMean of item means and standard deviation of item means.
bNumber of respondents for questions B and D are conditional on responses to questions A and C.
See Appendix for exact wording of these questions.
CNumber of items.
OCR for page 128
28
LONG-TERM TREND ITEMS IN SCIENCE AND MATHEMATICS
TABLE 6-2 Means and Standard Deviations of Responses to Questions A, B.
C, and D Across Groups and Combined for Science
District Reform Disciplinary
Teachers Teachers Specialists Combined
Question
Short- Long-
Term Term
(16)C (25)
0.724 0.824
0.173 0.188
9-10 10
4.067 3.943
0.447 0.364
4_9 4-10
Short- Long
Term Term
(16) (25)
0.869 0.917
0.087 0.161
10 9-10
Short- Long-
Term Term
(16) (25)
Short- Long-
Term Term
A Meana
SD
N (Range)b
B
C
D
Mean
SD
N
X
X
3.994 3.959
0.400 0.342
2-10
X
X
X
X
X
X
0.797 0.870
0.121 0.156
4.030 3.951
0.400 0.221
Mean 0.831 0.868 0.900 0.873 0.863 0.868 0.865 0.870
SD 0.154 0.144 0.097 0.149 0.154 0.111 0.101 0.101
N 9-10 10 10 9-10 10 9-10
Mean 3.991 3.986 4.070 4.044 4.027 3.617 4.029 3.882
SD 0.449 0.325 0.388 0.313 0.459 0.477 0.314 0.217
N 5-10 5-10 7-10 5-10 5-10 7-10
NOTE: Means and standard deviations by item are also available by short-term and long-term,
respectively.
aMean of item means and standard deviation of item means.
bNumber of respondents for questions B and D are conditional on responses to questions A and C.
See Appendix for exact wording of these questions.
CNumber of items.
Means in Tables 6-1 and 6-2 are quite similar for main and long-term NAEP.
The means for questions A and C vary between .724 and .968, with most of the
means in the .80s and .9Os. Standard deviations for questions A and C range from
.048 to .188. The differences between means for A and C range from zero to 10
percentage points for science, with most at five percentage points or less. The
differences between means for A and C range from zero to 10 percentage points
for mathematics, with most at three percentage points or less. These high and
comparable mean results indicate that study participants strongly feel that both
frameworks, as reflected in the selected items, adequately assess mathematics
and science concepts taught in today's classrooms as well as what national stan-
dards say should be taught in today's classrooms. The results also indicate a high
degree of comparability across the two NAEPs.
Means for questions B and D are around 4.0, again indicating that study
participants strongly feel that both frameworks, as reflected in the selected items,
adequately assess the importance of mathematics and science concepts taught in
OCR for page 129
JENNIFER R. ZIELESKIEWICZ
129
today's classrooms as well as what national standards say should be taught.
Standard deviations for questions B and D range from .277 to .485. The results
also indicate a high degree of comparability across the two NAEPs. The differ-
ences between the main and the long-term NAEP items are in the second decimal
place on a scale ranging from 1 to 5. Because of the small sample sizes involved,
no significance tests were performed.
DISCUSSION
The research conducted on the subject-matter experts' perceptions of the
relevance of the NAEP long-term trend items in mathematics and science for
eighth graders has provided relevant information on the importance of the long-
term trend assessment. These findings can be summarized as follows:
· The long-term trend and main NAEP mathematics and science items
evaluated in this study appear to reflect important content and skills that grade 8
teachers cover in their classes.
· This subset of items also appears to reflect what grade 8 teachers and
disciplinary specialists believe are important parts of national standards in math-
ematics and science.
For both mathematics and science there do not appear to be any meaning-
ful differences in grade 8 teachers' or disciplinary specialists' perceptions of the
relevance of long-term trend items and main NAEP items, as measured by cover-
age in current classrooms or reflection in national standards.
.
The results taken as a whole suggest that these long-term trend items are up
to date in measuring student achievement. Findings suggest that these long-term
trend and main NAEP items reflect what is important for eighth-grade students to
know and be able to do as well as what national standards say should be taught in
today's eighth-grade classrooms. It also appears that teachers and disciplinary
specialists agree on the relevance of the selected long-term trend and main NAEP
items as measured by coverage in current classrooms or reflection in national
standards. Although this study is only a first step toward evaluating the relevance
of the long-term trend NAEP, one could suggest that the long-term trend NAEP,
as a whole, may be relevant in today's classroom, reflects current national stan-
dards, and is as equally valuable as the main NAEP assessments. However, more
research is needed to clarify this conclusion.
No meaningful differences were found between rating groups on questions A
through D, suggesting that the selected long-term trend and main NAEP items are
reflective of what is being taught in the classroom as well as what national
standards say should be taught. This implies that these long-term NAEP trend
items are not out of date for use in the 1990s and provide the same information as
the main NAEP items. These findings also suggest that national standards are
OCR for page 130
130
LONG-TERM TREND ITEMS IN SCIENCE AND MATHEMATICS
visible to individuals in the teaching professions, although to what degree is
uncertain. It is likely that some teachers will recognize the national standards,
but it is unclear whether all of those in the teaching profession have knowledge of
them.
These results have important implications for the future of the long-term
trend assessment. For several decades the NAEP trends have provided informa-
tion about the educational achievements of students in American schools. The
goal has been to assess information on what American students know and can do
in the classroom and to compare their current performance with that of similar
students assessed in the past. The main and long-term NAEP trends were designed
to assess this achievement but using different frameworks. The results of this
study give some suggestion that both this subset of main and long-term NAEPs
items are equally valuable in their assessment and provide similar information.
Perhaps one day the two trends may be combined into one assessment, eliminat-
ing the use of multiple measures over time and simplifying the testing process in
general. More research is needed to investigate this possibility.
Because the results are preliminary, any interpretation or speculation on their
use should be weighed against a small sample size and the use of a subset of trend
items. As a result, caution should be taken in generalizing these results to popu-
lations outside the parameters of this study. A small sample may not adequately
assess the differences in educational content from one state to the next, particu-
larly in those states where there is no mandated curriculum or textbook adoption.
Also, a subset of trend items was used, thus providing information about those
national standards and content reflected only in those items. The results have
shown that those items are reflective of what is important for eighth-grade students
to know and be able to do as well as what national standards say should be taught
in today's eighth-grade classrooms. It would be premature to assume, since only
a portion of the national standards and content were represented in this study, that
all national standards and content are reflected in classroom curriculum.
To learn more about the long-term trend NAEP assessment, research should
be performed to focus on several aspects of it. Future research should consider
the use of a larger sample of raters and the inclusion of a larger, more representa-
tive sample of items from the mathematics and science assessments. It may also
be appropriate to include other disciplines in the research.
SUMMARY
Researchers were interested in knowing if the long-term NAEP is an up-to-
date and relevant measure of student achievement in mathematics and science
compared with the main NAEP frameworks and assessments developed more
recently. Subsets of items from the long-term and main NAEP assessments were
combined into two surveys, one for mathematics and one for science, and were
sent to 60 subject-matter experts.
OCR for page 131
JENNIFER R. ZIELESKIEWICZ
131
The long-term trend and main NAEP science and mathematics items evalu-
ated in this study appear to reflect important content and skills that grade 8
teachers cover in their classes. These select items also appear to reflect what
grade 8 teachers and disciplinary specialists believe to be important parts of
national standards in science and mathematics. Lastly, in both science and math-
ematics there do not appear to be any meaningful differences in grade 8 teachers'
or disciplinary specialists' perceptions of the relevance of long-term trend items
and main NAEP items as measured by coverage in current classrooms or reflec-
tion in national standards.
APPENDIX
Question A: Does this item assess knowledge and/or skills that students in your
school will have covered in science (mathematics) by the end of the eighth grade?
(Please check one)
No Yes
Question B.: If yes, relative to all of what students cover in science (mathematics)
in your school, how important is it for students to know/be able to do what is
covered in the item by the end of the eighth grade? (Please circle one)
2
Not
important
3
Somewhat
important
4
5
Very
important
Question C: Does this item assess knowledge and/or skills that students should
have covered in science (mathematics) by the end of the eighth grade according
to your best understanding of national standards? (Please check one)
No Yes
Question D: If yes, relative to all of what national standards say students should
cover, how important is it for students to know and be able to do what is covered
by this item by the end of the eighth grade? (Please circle one)
2
Not
important
3
Somewhat
important
4
5
Very
important
Representative terms from entire chapter:
trend items
