Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress (2017) / Chapter Skim
Currently Skimming:
Summary
Summary
Pages 1-14
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 1... ...
Over time, there has been growing interest in comparing educational progress across the states. At the same time, there has been increasing interest in having the results reported in a way that policy makers and the public could understand and so that they could be used to examine students' achievement in relation to high, world-class standards.
|
From page 2... ...
After a major standard setting process in 1992, NAEP began reporting results in relation to the three achievement levels. However, the use of achievement levels has provoked controversy and disagreement, and evaluators have identified numerous concerns.
|
From page 3... ...
Hence, NAGB sought feedback from a wide range of experts and stakeholders in setting the standards: educators, administrators, subject-matter specialists, policy makers, parent groups, and professional organizations, as well as the general public. Through the standard setting process, NAGB adopted a set of achievement levels for each subject area and grade.
|
From page 4... ...
In spite of the NAEd evaluators' concerns, NAGB moved forward with achievement-level reporting for the 1992 assessments of mathematics and reading. Since then, NAGB and NCES have sponsored research conferences, sought advice from experts in standard setting, commissioned research, formed standing advisory groups, held training workshops, and published materials on standard setting.
|
From page 5... ...
The achievement levels defined by these cut scores provide the basis for using and interpreting test results, and thus, the validity of test score interpretations hinges on the appropriateness of the cut score. In evaluating these outcomes, the committee examined evidence of their reliability and validity.
|
From page 6... ...
Content-Related Validity Evidence With regard to content-related validity evidence, the studies focused on the alignment between the ALDs and cut scores, the frameworks, and the test questions. For these studies, a second and sometimes third group of panelists were asked to review the ALDs and cut scores produced by the initial standard setting.
|
From page 7... ...
It is not clear why NAGB did not pursue studies. In contrast, the NAEd reports include a variety of studies, such as comparisons with state assessments, international assessments, advanced placement tests, and college admissions tests.
|
From page 8... ...
Recent research has focused on validity evidence based on relationships with external variables, that is, setting benchmarks on NAEP that are related to concurrent or future performance on measures external to NAEP. The findings from this research can be used to evaluate the validity of new interpretations of the existing achievement levels, suggest possible adjustments to the cut scores or descriptors, or enhance understanding and use of the achievement-level results.
|
From page 9... ...
CONCLUSION 6-1 The National Assessment of Educational Progress achievement levels are widely disseminated to and used by many audiences, but the interpretive guidance about the meaning and appropriate uses of those levels provided to users is inconsistent and piecemeal. Without appropriate guid ance, misuses are likely.
|
From page 10... ...
We also recognize the difficulties that might be created by setting new standards, particularly the disruptions that would result from breaking people's interpretations of the trends. We also note that during their 24 years they have acquired meaning for NAEP's various audiences and stakeholders: they serve as stable benchmarks for monitoring achievement trends, and they are widely used to inform public discourse and policy decisions.
|
From page 11... ...
.2 CONCLUSION 7-2 Although there is evidence to support con ducting a new standard setting at this time for all grades in reading and mathematics, setting new cut scores would disrupt the National Assessment of Educational Progress trend line at a time when many other contextual factors are changing. In the short term, the disruption in the trend line could be avoided by continuing to follow the same cut scores but ensur ing the descriptions are aligned with them.
|
From page 12... ...
Similar research is needed to evaluate alignment for the grade-4 and grade-8 math ematics assessments and to revise them as needed to ensure that they represent the knowledge and skills of students at each achievement level. Moreover, additional work to verify align ment for grade-4 reading and grade-12 mathematics is needed.4 RECOMMENDATION 2 Once satisfactory alignment among the frameworks, item pools, achievement-level descriptors, and cut scores in National Assessment of Educational Progress mathematics and reading has been demonstrated, their desig nation as trial should be discontinued.
|
From page 13... ...
Like the research that led to setting scale scores that represent academic preparedness for college, new research should focus on other measures of future performance, such as being on track for a college-ready high school diploma for 8th-grade students and readiness for middle school for 4th-grade students.7 RECOMMENDATION 5 Research is needed to articulate the intended interpretations and uses of the achievement levels and to collect validity evidence to support these interpretations and uses. In addition, research is needed to identify the actual inter pretations and uses commonly made by the National Assess ment of Educational Progress's various audiences and to evalu ate the validity of each of them.
|
From page 14... ...
Factors to consider include, but are not limited to sub stantive changes in the constructs, item types, or frameworks; innovations in the modality for administering assessments; advances in standard setting methodologies; and changes in the policy environment for using NAEP results. These factors should be weighed against the downsides of interrupting the trend data and information.10 10This recommendation was revised after the report was initially transmitted to the U.S.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.