Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress (2017) / Chapter Skim
Currently Skimming:
8 Conclusions and Recommendations
8 Conclusions and Recommendations
Pages 235-248
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 235... ...
4. Did the process yield a reasonable set of cut scores?
|
From page 236... ...
Reliability of Achievement Levels Three kinds of reliability evidence were reported by NAGB and ACT: interpanelist agreement, intrapanelist consistency across items of different types, and the stability of cut scores across occasions. The materials ACT prepared to document the standard settings were very detailed and included the kinds of reliability information one would expect to see.
|
From page 237... ...
who studied this issue recommended additional studies before reporting achievement-level results. NAGB chose not to conduct additional studies; it did lower the cut scores for mathematics by 1 standard error, but it accepted the cut scores for reading.
|
From page 238... ...
ACT conducted several studies to obtain feedback from content-area experts regarding the validity of ALDs and exemplar items, and the content experts who participated in these studies suggested changes to both the descriptors and the items. The descriptors were further revised by NAGB, and those final, official versions are quite different from the ones used for setting the cut scores (see the Annex to Chapter 5)
|
From page 239... ...
But, to date, there has been no effort to set new cut scores using the most current ALDs.2 CONCLUSION 5-2 Changes in the National Assessment of Educational Progress mathematics frameworks in 2005 led to new achievement-level descriptors and a new scale and cut scores for the achievement levels at the 12th grade, but not for the 4th and 8th grades. These changes create a perceived or actual break between 12th-grade mathematics and 4th- and 8th grade mathematics.
|
From page 240... ...
We were not able to determine whether this evidence was considered when the final cut scores were adopted for the National Assessment of Educational Progress. More recently, there has been a focus on external validity and predictive validity: that is, setting benchmarks on NAEP that are related to concurrent or future performance on measures external to NAEP.
|
From page 241... ...
CONCLUSION 6-1 The National Assessment of Educational Progress achievement levels are widely disseminated to and used by many audiences, but the interpretive guidance about the meaning and appropriate uses of those levels provided to users is inconsistent and piecemeal. Without appropriate guid ance, misuses are likely.
|
From page 242... ...
SETTING NEW STANDARDS: WEIGHING THE OPTIONS3 In this report, we explore a number of important issues with the NAEP achievement levels established in 1992. Researchers and others have raised questions about the standard setting methodology, the validity of the ALDs, and the appropriateness of the cut scores. The committee finds some of these criticisms to be compelling and to bear on our judgments of the extent to which the achievement levels are valid, reliable, reasonable, and informative.
|
From page 243... ...
.4 CONCLUSION 7-2 Although there is evidence to support con ducting a new standard setting at this time for all grades in reading and mathematics, setting new cut scores would disrupt the National Assessment of Educational Progress trend line 4This conclusion was added after the report was initially transmitted to the U.S. Depart ment of Education; see Chapter 1 ("Data Sources")
|
From page 244... ...
The more audiences understand the scale scores, the less likely they are to misuse the achievement levels. Setting new cut scores at this time, when so many things are in flux, would likely create considerable confusion about their meaning. We do not encourage a new standard setting at this time.
|
From page 245... ...
Moreover, additional work to verify alignment for grade-4 reading and grade-12 mathematics is needed.6 RECOMMENDATION 2 Once satisfactory alignment among the frameworks, the item pools, the achievement-level descrip tors, and the cut scores in National Assessment of Educational Progress mathematics and reading has been demonstrated, their designation as trial should be discontinued. This work should be completed and the results evaluated as stipulated by law: 20 U.S.
|
From page 246... ...
Like the research that led to setting scale scores that represent academic preparedness for college, new research should focus on other measures of future performance, such as being on track for a college-ready high school diploma for 8th-grade students and readiness for middle school for 4th-grade students.8 Actions are needed to improve the interpretation and use of NAEP reports, maintain the validity and usefulness of NAEP data, and ensure the currency of the NAEP achievement levels. The first step is to develop more concrete guidance for users on appropriate and inappropriate interpretations of achievement levels, to avoid NAEP's audiences attaching their own understandings to them.
|
From page 247... ...
This information should be communicated to users with clear guidance on substantiated and unsubstantiated interpretations.9 Since 1983, NAEP results have been reported using statistics based on the scale score metric, including the mean, median, mode, standard deviation, and percentiles. Achievement levels were adopted later as an additional reporting device to serve specific communication and policy purposes.
|
From page 248... ...
New linking studies have interpreted NAEP results in terms of the results of international assessments, with possibilities for linking NAEP 4th- and 8th-grade results to indicate being on track for future learning. External, but connected to NAEP, major national initiatives have significantly altered state standards in reading and mathematics.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.