Grading the Nation's Report Card Research from the Evaluation of NAEP (2000) / Chapter Skim
Currently Skimming:
7 Issues in Phasing Out Trend NAEP
7 Issues in Phasing Out Trend NAEP
Pages 132-151
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 132... ...
The recommendations focus on two designs. In one promising design, longterm trends are monitored with main NAEP, and overlapping main NAEP assessments are used whenever an assessment is modified.
|
From page 133... ...
This paper was commissioned by the National Research Council to discuss ways in which long-term trend NAEP could be phased out and replaced by the main NAEP assessments while still maintaining a long-term trend line. One significant question to be addressed is the following: How can a single assessment be developed that is stable enough to provide long-term trends while still being flexible enough to adapt to changes in assessment approaches?
|
From page 134... ...
(1996~. The evaluation of these designs includes considering how their implementation would affect the measurement of long-term trends as well as the effect on the main NAEP assessments.
|
From page 135... ...
The state NAEP assessments use representative subsets of main NAEP booklets. The two programs differ in administration procedures and other operational procedures, such as who is included in the assessments.
|
From page 136... ...
has considered combining the state NAEP and main NAEP assessments to increase efficiency, these differences make combining them challenging (Mullis, 1997; Rust, 1996; Spencer, 1996~. The use of state NAEP likely will increase pressure for changing the assessment's content because a wider group of people in states and school districts have a stake in NAEP.
|
From page 137... ...
The current practice is to make the changes in the main NAEP assessment called for by content experts and supported by the National Assessment Governing Board, but to leave the trend assessment frameworks undisturbed. (Barron and Koretz, 1996:215)
|
From page 138... ...
In an example provided by Barron and Koretz (1996) , the main NAEP assessments indicated greater relative gains in writing achievement in high school than did the long-term trend NAEP writing assessment.
|
From page 139... ...
/2. TABLE 7-3 Differences in Selected Percentiles Between Males and Females in Long-Term Trend NAEP Sciencea All Male Female Difference Difference/Q Age 9 Plo 174.5 176.5 172.3 4.2 P25 201.3 202.9 200.0 2.9 P50 231.0 232.7 229.6 3.1 P75 258.9 262.1 256.2 5.9 P9O 283.6 286.9 279.0 7.9 Age 13 Plo 105.3 208.9 202.4 6.5 .248 P25 230.4 233.9 227.6 6.3 .240 P50 257.7 262.4 253.6 8.8 .335 P75 282.9 288.6 277.3 11.3 .430 P9O 304.4 309.3 298.4 10.9 .415 Age 17 Plo 235.1 234.0 235.8 -1.8 -.059 P25 265.9 268.9 263.3 5.6 .182 P50 298.2 303.9 293.3 10.6 .345 P75 327.3 333.2 321.7 11.5 .375 P9O 351.7 358.6 344.1 14.5 .472 aPercentiles are from Campbell et al.
|
From page 140... ...
140 0.50 0.40 Standardized 0 30 Male - Female 0.20 Difference 0.10 ~ 0.00 -0.10 0.50 0.40 Standardized 0 30 Male - Female 0 20 D'fference 0.10 0.00 ~ -0.10 0.50 0.40 ~ Standardized 0 30 Male- Female Difference 0.20 0.10 0.00 -0.10 ISSUES IN PHASING OUT TREND NAEP Age 9 or Grade 4 10 B W:~ / B J 25 50 75 90 Selected Percentiles Age 13 Or Grade If 10 10 25 25 50 75 90 Selected Percentiles ~'J''/~' Age170rGrade12 / / / / 50 75 90 Selected Percentiles B Long-Term Trend J Main B Long-Term Trend J Main B Long-Term Trend J Main FIGURE 7-1 Standardized male-female differences in 1996 long-term trend NAEP and main NAEP selected percentiles.
|
From page 141... ...
In this section, alternative designs are discussed for the main NAEP and long-term trend NAEP assessments. Design 1: Keep the Current Design One possibility is to continue with the current design, which for long-term trend NAEP uses the same exercises and operational procedures from one assessment to the next.
|
From page 142... ...
With this design, main NAEP could continue to evolve to reflect curricular trends, unimpeded by the necessity to maintain long-term trends. However, unlike the current design, periodic modest changes are allowed in the content of long-term trend NAEP in an attempt to avoid problems associated with "the relation of item performance to overall proficiency .
|
From page 143... ...
The general concepts of developing alternate forms of an assessment and equating could be used in a new long-term trend NAEP assessment design. One difference between NAEP and assessments that routinely use equating processes is that NAEP uses a set of booklets, with different students administered different booklets.
|
From page 144... ...
In this event, special studies would be required to link the two assessments if long-term trends were to be followed from one long-term trend assessment to another. Design 3: Eliminate Long-Term Trend NAEP and Use Main NAEP for Trend Assessment NAEP faces two formidable challenges if long-term trend NAEP is eliminated.
|
From page 145... ...
Along with differences in content and exercise types, these include differences between the two assessments in operational and analysis procedures. For example, as discussed earlier, the main NAEP assessment uses "derived race," whereas the long-term trend assessment uses "observed race." Barron and Koretz (1996)
|
From page 146... ...
If, as implied by Figure 7-2, all NAEP data are reported on the scale of the current main NAEP assessment, the trends are estimated for the current main NAEP assessment. Clearly, estimating such trends entails strong statistical assumptions, since main NAEP was not administered in previous years.
|
From page 147... ...
An extensive research program that involves development of methodology and empirical research is needed before NAEP adopts this linking design. Design 4: Eliminate Long-Term Trend NAEP and Maintain Two Main NAEPs for Trend Assessment Zwick (1992)
|
From page 148... ...
Possibly, procedures could be developed that allow for linking assessments as different as long-term trend NAEP and main NAEP or as different as new and old main NAEP. However, it is also possible that the results of the research will indicate that changes in main NAEP assessments need to be much more tightly constrained than is presently the case.
|
From page 149... ...
Another potentially useful alternative is: Design 2: periodically update long-term trend NAEP while maintaining main NAEP. In this design, main NAEP is allowed to change, in small ways, to better reflect current curncula.
|
From page 150... ...
National Assessment Governing Board (NAGB) 1996 Policy Statement on Redesigning the National Assessment of Educational Progress.
|
From page 151... ...
1992 Statistical and psychometric issues in the measurement of educational achievement trends: Examples from the National Assessment of Educational Progress. Journal of Educational Statistics 17(2)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.