Grading the Nation's Report Card Research from the Evaluation of NAEP (2000) / Chapter Skim
Currently Skimming:
8 Issues in Combining State NAEP and Main NAEP
8 Issues in Combining State NAEP and Main NAEP
Pages 152-171
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 152... ...
Significant administration differences between main NAEP and state NAEP exist, which make combining difficult. These differences currently are addressed by adjusting state NAEP scores.
|
From page 153... ...
COMPARISON OF MAIN NAEP AND STATE NAEP The main NAEP and long-term trend NAEP assessments were not designed to produce state-level data. To explore the possibility of NAEP providing data at the state level, in 1990, 1992, and 1994 voluntary trial state NAEP assessments were conducted that produced state-level data to compare states to one another and to the nation as a whole.
|
From page 154... ...
TABLE 8-1 Main NAEP and State NAEP Assessments by Year Since 1986a Main NAEP (grades 4, 8, and 12 Year except where noted) State (or Trial State)
|
From page 155... ...
Thus, although the same items are used in state NAEP and main NAEP, concerns about the effects of differences in administration procedures led to the decision to independently scale the two assessments. Linking studies that have been conducted use a common person design, in which a sample of examiners from main NAEP is matched to the state NAEP sample.
|
From page 156... ...
However, when these linking studies have been conducted, it has been found repeatedly that the average proportion correct on state NAEP tends to be higher than the average proportion correct on main NAEP for the matched samples. This finding suggests that, on average, students can be expected to correctly answer more items when an assessment is administered under state NAEP administration procedures than when an assessment with identical questions is administered under main NAEP administration conditions.
|
From page 157... ...
Implications of Differences for Score Interpretation Apparently, the linking studies that adjust for differences in administration conditions have the following as their goal: the scale scores reported for a particular state should reflect the scale scores that state would have received had the state assessment been administered under the conditions used to administer the main NAEP assessment. Various assumptions are implicit in conducting these linking studies, and a single set of linking constants is applied for all jurisdictions.
|
From page 158... ...
State NAEP is then linked to main NAEP. Based on this study, a state with an average percentage of items correct in state NAEP of 62 percent will have an average scale score of 270.
|
From page 159... ...
One of the related issues is that IRT (item response theory) parameter estimates for a given item could differ considerably from the main NAEP to the state NAEP assessment.
|
From page 160... ...
Main NAEP data are used to construct the item maps. Recall that students tend to score higher when using state NAEP administration than when using main NAEP administration conditions.
|
From page 161... ...
These problems seem most serious when attempting to make criterion-referenced interpretations of scores, such as those made with item maps and achievement levels. DESIGNS FOR COMBINING STATE AND MAIN NAEP SAMPLES In this section, issues in developing designs for combining state and main NAEP samples are discussed.
|
From page 162... ...
, "the NAEP sampling and weighting procedures are designed to obtain sample data that permit estimates of subpopulation characteristics of reasonably high precision." The precision targets are stated ahead of time, and samples are designed to meet these targets. Current Design for Main NAEP The goal of the main NAEP sample design is to adequately represent the population of students in the United States in a particular grade as well as certain subpopulations.
|
From page 163... ...
Apart from using main NAEP data to estimate linking functions, data analysis for state NAEP is conducted using the state data only. Some possibilities for combining the main and state NAEP sample designs and/or data analyses follow.
|
From page 164... ...
Note that under this design, to meet target precision for the states, it is necessary to pool data from the state and main NAEP samples. These precision targets could be met only if the random error and bias due to linking are negligible.
|
From page 165... ...
suggested that this change would enhance participation in main NAEP. However, they did not discuss how this enhanced participation, if it did exist, might affect the comparability of main NAEP scores between current main NAEP and main NAEP after the change in recruitment procedures was made.
|
From page 166... ...
As with the other designs that involve an integration of main and state NAEP data, a major issue concerning this design is that it requires a linking study to adjust state results for differences in state and national administration conditions. The gain in precision for main NAEP and the state precision targets likely could be achieved only if the random error and bias due to linking are negligible.
|
From page 167... ...
This issue needs to be addressed before further consideration of Rust and Shaffer' s Approach 2. A major issue with both Spencer's Approach 2 and Rust and Shaffer's Approach 2 is that both rely heavily on there being little random error or bias in adjusting state NAEP results for operational differences between state NAEP and main NAEP.
|
From page 168... ...
A significant problem in each approach arises from the operational differences between main and state NAEP that cause complications potentially difficult to overcome. Unless the operational procedures for main NAEP and state NAEP can be made much more similar to one another, the potential complications caused by these approaches might lead to severe problems in combining NAEP samples.
|
From page 169... ...
Question 5: Do the differences in administration and recruitment conditions affect the constructs that are being measured by the NAEP assessments? To what extent does bias or systematic error influence the These questions should be thoroughly addressed before any design for combining the state and main NAEP samples is implemented under current recruitment and administration conditions.
|
From page 170... ...
McLaughlin 1993 A Study of the Administration of the 1992 National Assessment of Educational Progress Trial State Assessment Program. Palo Alto, Calif.: American Institutes for Research.
|
From page 171... ...
In NAEP Reconfigured: An Integrated Redesign of the National Assessment of Educational Progress, E.G. Johnson, S
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.