Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress (2017) / Chapter Skim
Currently Skimming:
1 Introduction
1 Introduction
Pages 15-34
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 15... ...
By 1988, there was considerable support for these changes. The Elementary and Secondary Education Act of 1988 authorized the formation of the National Assessment Governing Board (NAGB)
|
From page 16... ...
Three achievement levels are used: "Basic," "Proficient," and "Advanced." Results are reported according to the percentage of test takers whose scores are at each achievement level, and brief descriptions of the levels are provided with the results.1 The percentage of test takers who score below the Basic level is also reported: Figures 1-1a through 1-6a in the Annex to this chapter show an example of this type of reporting. The purpose for setting achievement levels was explicitly stated (National Assessment Governing Board, 1993, p.
|
From page 17... ...
To address its charge, the committee held three in-person meetings and four half-day virtual meetings during 2015. Before discussing our approach to the study, we provide some background on the process for developing achievement levels, or, more generally, standard setting and on the key features of NAEP.
|
From page 18... ...
In these education and BOX 1-1 Key Terms in Standard Setting Unless specifically noted, the definitions below are excerpted and adapted from Standards for Educational and Psychological Testing (American Educational Re search Association et al., 2014)
|
From page 19... ...
. Standard setting: The process, often judgment based, of setting cut scores using a structured procedure that seeks to map test scores into discrete performance levels that are usually specified by performance-level descriptors (p.
|
From page 20... ...
, various characteristics of the assessment and its uses, and often the experiential base of those conducting the standard setting. Regardless of the method chosen, the most critical issue is that the process is carried out carefully and systematically, following accepted procedures.
|
From page 21... ...
of interest. The ALDs guide standard setting so that panelists can operationalize them in terms of cut scores with the same conceptualization used by item writers.
|
From page 22... ...
student population. Historically, there have been two samples: one for the national assessment and one for the state assessments, initially referred to as the Trial State Assessment Program.
|
From page 23... ...
Students receive different but overlapping sets of NAEP items using a form of matrix subsampling known as balanced incomplete block spiraling. This design requires highly complicated analyses and does not permit the performance of a particular student to be accurately measured.
|
From page 24... ...
Results are tabulated over time to provide trend information. As described above, NAEP also reports performance using achievement levels.
|
From page 25... ...
This public forum was organized as five panel discussions, each focused on a type of audience for NAEP results: journalists and education writers, state policy users, developers of the assessments designed to be aligned with the Common Core State Standards, research and advocacy groups, and a synthesis panel with two experts in standard setting. See Appendix A for the forum agenda.
|
From page 26... ...
, and cut scores -- reasonable and sensible? Did it follow generally accepted procedures (at the time the achievement levels were set and also according to the current state of the field and knowledge base)
|
From page 27... ...
to the criticisms made in the external evaluations; • empirical studies of the reliability, validity, usefulness, and "infor mativeness" of the achievement levels and achievement-level reporting, including studies of the effects of using various proce dures for establishing the cut scores; • professional standards with regard to standard setting (e.g., those of the American Educational Research Association, the American Psychological Association, and the National Council on Measure ment in Education) ; • edited volumes and other scholarly literature from well-known experts in the field of standard setting, summarizing research and empirical studies and offering insights and advice on best practice; • subjective accounts about achievement-level reporting, including opinion pieces, commentaries, newspaper columns and articles, blogs, tweets, conference and workshop presentations, and public forums; and • other reports prepared by research and policy organizations about specific aspects of achievement-level reporting for NAEP.
|
From page 28... ...
In order to be able to address our charge within the allotted time period, we reviewed several compendia and edited volumes by well-respected experts in the field of standard setting. These documents are well known in the measurement field, and the various chapters cover an array of views.
|
From page 29... ...
In Chapter 7, we explore the issues to consider in deciding whether a new standard setting is needed, and we present our recommendations in Chapter 8. ANNEX: EXAMPLE OF ACHIEVEMENT LEVEL REPORTING FOR NAEP The six bar charts in this Annex show the percentage of test takers who scored at each achievement level -- Basic, Proficient, and Advanced -- for mathematics and reading for grades 4, 8, and 12.
|
From page 30... ...
SOURCE: The Nation's Report Card (2015b)
|
From page 31... ...
SOURCE: The Nation's Report Card (2015d)
|
From page 32... ...
. Below Basic Basic Proficient Advanced Percentage FIGURE 1-4a Trend in 4th-grade National Assessment of Educational Progress reading achievement-level results.
|
From page 33... ...
. Percentage FIGURE 1-6a Trend in 12th-grade National Assessment of Educational Progress reading achievement-level results.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.