Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress (2017) / Chapter Skim
Currently Skimming:
6 Interpretations and Uses of NAEP Achievement Levels
6 Interpretations and Uses of NAEP Achievement Levels
Pages 189-218
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 189... ...
We consider the findings and conclusions presented in Chapters 3-5 in light of the inferences that are based on achievement levels. We discuss interpretations and uses in terms of those intended by the developers of the achievement levels -- the National Assessment Governing Board (NAGB)
|
From page 190... ...
. The committee examined the following materials to gather information about interpretations and uses: • NAEP policy statements (National Assessment Governing Board, 1990, 1993, 1995, 2006)
|
From page 191... ...
A more recent policy statement addresses NAEP data broadly, that is, all reports rather than just achievement-level reporting: "The Nation's Report Card shall seek to encourage wide public attention to NAEP results and clear understanding of their meaning and significance" (p.
|
From page 192... ...
. BOX 6-1 Users and Uses of NAEP Achievement Level Data and Actions Based on Them Users • Federal officials • State officials • Local district officials • Local school officials • Media • Education researchers • General public Uses and Examples of Interpretations • Trends or comparisons of successive cohorts, such as proficiency rates in reading (mathematics)
|
From page 193... ...
Uses NAEP materials are not entirely explicit about the intended uses of achievement-level results. The initial policy statement (National Assessment Governing Board, 1990, p.
|
From page 194... ...
1) : Foreword A policy on setting achievement levels on the National Assessment of Educational Progress (NAEP)
|
From page 195... ...
It is difficult to discern a theory of use from the many NAEP documents that attempt to describe and designate appropriate uses of NAEP achievement-level descriptors (ALDs)
|
From page 196... ...
However, we were not able to locate any studies specifically designed to support the interpretations and uses discussed above, for example, that achievement levels make the NAEP data more understandable to general users, parents, policy makers, and educators. What is lacking is a comprehensive validity investigation that connects specific evidence to each of the intended interpretations and uses discussed above.
|
From page 197... ...
Prior to the release of NAEP results, NCES convenes representatives from each state: states are encouraged to send assessment, communication, and subject-area specialists. These state teams work with national assessment specialists to review their state's data, and they often draft preliminary press releases, which are then reviewed by a nationally recognized communications firm.
|
From page 198... ...
. Clicking on the gold rectangle takes a user to an Interactive Report Card dashboard, where the user can customize the information provided.
|
From page 199... ...
Returning to the main menu shown in Figure 6-2 and selecting "Knowledge and Skills" (see arrow on the left side of the page) brings up a page that shows examples of what students are "likely" to be able to do at various score levels, shown in Figure 6-5.
|
From page 200... ...
In addition, the many users of NAEP vary in both their knowledge of NAEP and their reasons for seeking NAEP information. Users with limited knowledge of NAEP, including interested parents, face a wide array of NAEP information, which includes undefined technical terms, such as ALDs, trend lines, sample items, and state-by-state comparisons.
|
From page 201... ...
Our search was not intended to be a random or representative sampling of the universe of uses and users: instead, it was a targeted search for examples illustrating a variety of NAEP uses and users. We selected examples of press and journal coverage of achievement-level results, state and local press releases and "state report cards" that are required by the No Child Left Behind Act (NCLB)
|
From page 202... ...
For this activity, the committee selected panelists to represent a diverse set of perspectives, organizing the panelists into four groups: state and local policy makers, education writers, education policy researchers and advocacy groups, and the designers
|
From page 203... ...
INTERPRETATIONS AND USES OF NAEP ACHIEVEMENT LEVELS 203 FIGURE 6-4 Continued
|
From page 204... ...
. of the common core state standards the Partnership for Assessment of -- Readiness for College and Careers (PARCC)
|
From page 205... ...
, analyze their state's scale score trends, interpret results in light of their own state curriculum, and use item formats to suggest changes to state assessments and to aid in defining levels of student performance. State policy makers also use NAEP data to validate state standards and to build the case for educational reform and drive change in their states (DeVito, 1997, as cited in Jaeger, 2003; Simmons and Mwalimu, 2000; Zenisky et al., 2009)
|
From page 206... ...
Education Policy and Advocacy Groups Education organizations and interest groups at the committee's public forum reported using NAEP data as evidence to support their advocacy of specific needs of students, teachers, and schools. Although NAEP does not provide student-level data, one panelist reported that the groups use NAEP achievement levels to set aspirational goals for student learning and better communicate success targets to parents.
|
From page 207... ...
Common Core Assessment Consortia For the two national assessment consortia, Smarter Balanced Assessment Consortium6 and the Partnership for Assessment of Readiness for College and Careers, NAEP items are one of several components they use in their own standard setting methods to aid in defining proficiency levels for their assessments. PARCC uses NAEP as an external source of validity evidence.
|
From page 208... ...
• Interpreting "proficient" as equivalent to grade-level perfor mance, such as relating a scale score increase of X to a Y's (months, year) worth of learning or interpreting NAEP achievement levels as grade-level work.
|
From page 209... ...
a cut score, the outcome will be different for a group in which roughly 50 percent of examinees are proficient and a group in which a much lower or a much higher percentage are proficient. The result in such a comparison may conflict with one based on relative scale score changes in those two groups.
|
From page 210... ...
The percentage of students who scored Proficient or above increased more for African American students than for other students. At the same time, the average scale score increased more for other students than African Americans.
|
From page 211... ...
INTERPRETATIONS AND USES OF NAEP ACHIEVEMENT LEVELS 211 FIGURE 6-7 Comparison of 4th-grade mathematics performance for black and white students: Percentage of students at each achievement level and percentage at or above Proficient. SOURCE: The Nation's Report Card (2015b)
|
From page 212... ...
and the kinds of interpretations and uses that are made of NAEP achievement-level reports. The validity evidence documents the integrity and accuracy of the procedures used to set the achievement levels, the extent to which the ALDs are consistent with the assessment framework, and the relationships between the achievement levels and external criteria.
|
From page 213... ...
Furthermore, NAEP reports of achievement-level results include examples of inappropriate uses, such as tracking the percentage of students at or above a cut score to evaluate progress by population groups. Considerable information is provided to state and district education personnel and media representatives in preparation for a release of NAEP results, and NAGB provided us with examples of these materials.
|
From page 214... ...
Such an approach would be useful for communicating the difficulty levels of reading passages presented to test takers. CONCLUSION 6-1 The National Assessment of Educational Progress achievement levels are widely disseminated to and used by many audiences, but the interpretive guidance about the meaning and appropriate uses of those levels provided to users is inconsistent and piecemeal.
|
From page 215... ...
. The Valid Use of NAEP Achievement Level Scores to Confirm State Testing Results in the No Child Left Behind Act.
|
From page 216... ...
. Student Performance Standards on the National Assessment of Educational Progress: Affirmations and Improvements.
|
From page 217... ...
. NAEP Validity Studies: Reporting the Results of the National Assessment of Educational Progress.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.