Grading the Nation's Report Card Research from the Evaluation of NAEP (2000) / Chapter Skim
Currently Skimming:
9 Difficulties Associated with Secondary Analysis of NAEP Data
9 Difficulties Associated with Secondary Analysis of NAEP Data
Pages 172-194
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 172... ...
Second, providing data for secondary analysis is only one function of NAEP. Thus, the developers of NAEP must try to balance the anticipated needs of secondary analysts with other, sometimes competing, NAEP functions.
|
From page 173... ...
The information presented in this paper was collected through informal interviews with a number of NAEP secondary analysts as well as a number of staff members at NCES and ETS. This paper provides an overview of the potential difficulties one may encounter when using NAEP data and makes recommendations for improving the usability of NAEP data for secondary analysis in the future.
|
From page 174... ...
. The resulting item parameter estimates are used along with the item responses and background information collected on the examiners to estimate the distribution of student proficiency.
|
From page 175... ...
To outline the difficulties that secondary analysts have using the data, interviews were conducted (either by phone or e-mail) with researchers who have conducted secondary analyses of NAEP data or who have received training on secondary analysis of NAEP data but have not used the data outside the training.
|
From page 176... ...
In addition to collecting information from secondary analysts, three people who work on NAEP two from ETS and one from NCES and are knowledgeable about issues concerning secondary analysis of NAEP data were interviewed. They were asked about training opportunities and other assistance available to secondary analysts as well as other efforts NCES and ETS have made to facilitate NAEP research.
|
From page 177... ...
The median number of problems reported was three, and the maximum was seven. Discussions with NAEP secondary analysts identified six areas of concern: (1)
|
From page 178... ...
It takes much longer for the data and the accompanying technical documentation to be available for secondary analysts. According to the NCES Web site, the technical report for the 1996 science assessment was released in January 1998, almost two years after the assessment was administered.
|
From page 179... ...
Estimates of the design effects are provided to the secondary analyst in the NAEP documentation. Jackknife estimates of standard errors are much more precise but historically have been difficult for many secondary analysts to compute.
|
From page 180... ...
. However, there are other ways to facilitate analysis of NAEP data, and this is one example of how NCES and ETS have listened to the problems of secondary analysts and worked to find solutions.
|
From page 181... ...
Conditioning The aspect of the methodology that appears to cause secondary analysts the most concern is conditioning student achievement on background information. The process of conditioning on student background information in NAEP is also called "multiple imputations" or "plausible values methodology." For the purposes of this paper the scaling methodology and the resulting plausible values will be discussed separately.
|
From page 182... ...
Another problem that conditioning causes for secondary analysts is more fundamental. Researchers with years of experience with NAEP and strong backgrounds in statistics said that they still do not understand the methodology used to scale NAEP in anything more than general terms Another advantage of conditioning is that is allows estimates of proficiency to be obtained for individuals who answered all of the items either incorrectly or correctly something that is problematic with traditional scaling methods.
|
From page 183... ...
The technical report also states that conditioning allows key population features to be estimated consistently even when item booklet composition, format, and content balance change over time (Allen et al., 1996~. However, it is not known to what degree changes to the item booklet composition, format, and content balance also change the degree to which the model has been correctly specified.
|
From page 184... ...
In addition, software for handling plausible values in HEM analyses was developed using an NCES grant. However, NAEP staff have been less responsive to researchers who need a single optimal proficiency estimate for each examined, and ETS has not made its software available so that secondary analysts could produce their own estimates of proficiency.5 Form and Organization of NAEP Data Several problems were reported concerning the form and organization of NAEP data on the data files from researchers interested in item-level data.
|
From page 185... ...
It would save money for secondary analysts (some of which comes through NCES in the form of research grants)
|
From page 186... ...
They liked the opportunity to meet and interact with NAEP staff. They liked getting hands-on experience with the data and being able to try out special software written to help secondary analysts with the special features of NAEP.
|
From page 187... ...
Also, including SPSS and SAS code along with the data has improved the usability of the data. In addition, two researchers I spoke with received funding from NCES to develop special software for secondary analysts to use software that would automatically handle things like plausible values and jackknife weights.
|
From page 188... ...
There are a number of recommendations to facilitate secondary analysis of NAEP data that resulted from interviews with NAEP secondary analysts and NAEP staff at NCES and ETS. Several of them would be relatively easy to implement.
|
From page 189... ...
Implementing this recommendation would also require that better communication be established. NAEP staff, specifically ETS staff who write the NAEP technical documentation, currently reported knowing little about the specific interests of secondary analysts.
|
From page 190... ...
it could lessen confusion about the data; (3) it could reduce the amount of documentation needed to explain the data and result in documentation that is easier for secondary analysts to use; and (4)
|
From page 191... ...
The NAEP technical report states that conditioning allows key population features to be estimated consistently even when item booklet composition, format, and content balance change over time (Allen et al., 1996~.7 However, even though, theoretically, conditioning allows such changes over time, not even the designers of NAEP believe it to the degree that they would rely on it with the long-term NAEP trend assessment.8 The mantra of the long-term trend assessment is "when measuring change, don't change the measure." And even though the content balance of the long-term trend assessment is widely thought to be outdated and the content balance is uneven, that is exactly what has happened for 15 years the exact same set of items has been administered in every assess 6It is important to note that more of this time is used to administer performance items (large items) than was the case in earlier NAEP assessments.
|
From page 192... ...
" Eliminating the plausible values would also simplify secondary analysis of NAEP data. Plausible values are used to make apparent to researchers that the estimates of student proficiency are just that estimates.
|
From page 193... ...
The changes outlined aboveclearer communication between NAEP staff and secondary analysts, documentation and data files geared toward different types of secondary analysts, and a simpler NAEP design have the potential to dramatically increase the amount of research that is conducted using NAEP data, research that could be used to improve education and help students achieve to their potential. ACKNOWLEDGMENTS The author thanks all of the NAEP secondary analysts who shared their NAEP experiences; Larry Ogle from NCES and John Mazzeo and Al Rogers from ETS for providing information about NAEP; and Dean Cotton, Dan Koretz, and Kris Waltman for reviewing drafts of the paper.
|
From page 194... ...
Allen 1992 The NAEP Technical Report: 1990. Washington, D.C.: U.S.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.