Evaluation of the Achievement Levels for Mathematics and Reading on the National Assessment of Educational Progress (2017) / Chapter Skim
Currently Skimming:
5 Validity of the Achievement Levels
5 Validity of the Achievement Levels
Pages 101-188
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 101... ...
The final section presents the committee's conclusions about both kinds of validity. CONCEPTS OF VALIDITY AND VALIDATION Content and Criterion Validity Evidence In the context of setting standards for NAEP, content-related validity evidence focuses on the extent to which the achievement levels and exemplar items reflect the content and skills embodied in the assessment 101
|
From page 102... ...
The report should contain a description of the test, what is measured, the conclu sions and decision that are based on the test results, the obtained score, information on how to interpret the reported score, and any cut score used for classification. The 1999 version included the following guidance relevant to achievement levels, in Standard 8.8: When score reporting includes assigning individuals to categories, the categories should be chosen carefully and described precisely.
|
From page 103... ...
describe it as an argument-based approach. That is, validation involves developing a scientifically sound argument to support the intended interpretation of test scores and their relevance to the proposed use, as discussed in the Standards (American Educational Research Association et al., 1999, p.
|
From page 104... ...
CONTENT-RELATED VALIDITY EVIDENCE Content-related validity evidence for the NAEP achievement levels is presented in the ACT documentation and technical reports (ACT, Inc., 1993a, 1993b, 1993c, 1993d, 1993e; Allen et al., 1996, App.
|
From page 105... ...
For NAEP, prior to the process of setting cut scores, the ALDs were interpreted as aspirational: that is, they define the things that students should know and be able to do. Once the standard setting was completed, the draft descriptors were revised and exemplar items were selected to reflect the things students at each level actually know and can do.
|
From page 106... ...
The items can be mapped to the achievement level at which the likelihood of a correct response is 0.67. Thus, the exemplar items demonstrate the kinds of tasks that students with proficiency at the cut score are likely to get correct (e.g., two out of three times)
|
From page 107... ...
Of the 18 panelists that participated in the review, 10 had participated in the original standard setting, and 8 were new. Panelists who had participated in the standard setting had been selected from among the teachers for each of the three grade levels at the original standard setting.
|
From page 108... ...
Working in grade groups, the panelists agreed on several items to include as exemplars for each achievement level. The numbers of items varied, and there was some difficulty in reaching agreement on items for some achievement levels, particularly the advanced level.
|
From page 109... ...
Moreover, there was consensus that there was a mismatch between the descriptors and the items. The panelists agreed that exemplar items were critical to understanding the achievement levels, but their review of the released items was generally negative (Silver and Kenney, p.
|
From page 110... ...
They found that students at each achievement level had an average percentage correct of about 65 percent for items mapped to that achievement level.3 From this analysis, the researchers concluded that the NAGBproposed final descriptors in mathematics were reasonably clear and that the cut scores reflect the kinds of achievement included in the descriptors (ACT, Inc., 1993c, p.
|
From page 111... ...
Second, a panel of math ematics experts was convened. They reviewed all items that anchored to each of the three achievement level ranges and wrote individual descrip tions of the mathematics skills measured by each item.
|
From page 112... ...
, assign individual test takers to achievement levels. • Compute the probability of each student in that achievement level answering each item correctly (or, for an open-ended question, reaching a given score level)
|
From page 113... ...
Overall, a total of 171 items (approximately 76%) mapped to one of the achievement levels, and 54 items (approximately 24%)
|
From page 114... ...
At the end of the anchor study, panelists prepared and settled on draft descriptions for each of the achievement levels. Panelists also responded to three evaluation questions about their level of satisfaction with the item-level descriptors, the anchor descriptions for each achievement level, and the final ALDs.
|
From page 115... ...
presents results from a study to examine the congruence between the item pool and the ALDs. The 1992 Assessment Review of the Descriptors from the Standard Setting A total of 19 panelists participated in the initial review -- 10 from the original standard setting and 9 new panelists who were state-level reading curriculum supervisors or assessment directors or university faculty teaching in disciplines related to the subject area (see Allen et al., 1996, App.
|
From page 116... ...
One group used a procedure called item difficulty categorization; the other used a procedure called judgmental item categorization.6 The item-difficulty categorization procedure examined the level of support for the descriptors as justified by empirical performance data for the NAEP items. The items were selected for each achievement level using a response probability criterion of 0.50 at the lower borderline score.
|
From page 117... ...
The same criteria were set for the response probability value and the discrimination index used to map items onto specific achievement levels. The analyses were done for all three grades, and results appear in Table 5-2.
|
From page 118... ...
. The 16 panelists worked in grade groups to develop item-level descriptions and anchor descriptions for each achievement level.
|
From page 119... ...
But for 4th grade, none of the panelists thought the alignment was strong. For the Basic level, all the panelists judged it to be moderate; at the Proficient level, two panelists rated it as moderate, and three judged it as weak; for the Advanced level, one panelist rated it as moderate, and four rated it as weak.
|
From page 120... ...
made use of some of the same data sources as NAEd in evaluating the reasonableness and criterion-related validity of the achievement levels, including performance on AP tests; college admission tests; the international assessments in place by that time (IAEP was only administered in 1988 and 1991) : mathematics and reading tests for 15-year-olds on the Programme for International Student Assessment (PISA)
|
From page 121... ...
This result can be compared with NAEP results from 2011 and 2013, where the percentages scoring at the advanced level were 8 and 9 percent, respectively. Similarly, 8 percent of U.S.
|
From page 122... ...
Exact cut scores are as follows: below level 1 (a score less than or equal to 357.77) ; level 1 (a score greater than 357.77 and less than or equal to 420.07)
|
From page 123... ...
Figures 5-3 and 5-4 show the percentages of students at each of the levels for 2011, and Figures 5-5 and 5-6 show them for 2007. For those years, 13 percent of 4th graders and 7 percent of 8th graders scored at the advanced level; these results compare with 7 percent of 4th graders and 8 percent of 8th graders who were at the advanced level in mathematics on NAEP in 2011.
|
From page 124... ...
Exact cut scores are as follows: below level 1b (a score less than or equal to 262.04) ; level 1b (a score greater than 262.04 and less than or equal to 334.75)
|
From page 125... ...
students do on international assessments. They provide data that are useful comparisons with NAEP, particularly in judging the reasonableness of the percentages of students that score at the Proficient and Advanced levels.
|
From page 126... ...
126 EVALUATION OF ACHIEVEMENT LEVELS ON NAEP FIGURE 5-3 Percentage of 4th-grade students reaching the Trends in International Mathematics and Science Study international benchmarks in mathematics, by education system: 2011.
|
From page 127... ...
7The TIMSS International Study Center has reservations about the reliability of the average achievement score because the percentage of students with achievement too low for estimation is greater than 25 percent. 8National Defined Population covers less than 90 percent but at least 77 percent of the National Target Population (see Appendix A)
|
From page 128... ...
128 EVALUATION OF ACHIEVEMENT LEVELS ON NAEP FIGURE 5-4 Percentage of 8th-grade students reaching the Trends in International Mathematics and Science Study international benchmarks in mathematics, by education system: 2011.
|
From page 129... ...
. 2National Defined Population covers less than 90 percent, but at least 77 per cent of National Target Population (see Appendix A)
|
From page 131... ...
* FIGURE 5-5 Percentage of students reaching the Trends in International Mathematics and Science Study 2007 international benchmarks of mathematics achievement, 4th grade.
|
From page 133... ...
* FIGURE 5-6 Percentage of students reaching the Trends in International Mathematics and Science Study 2007 international benchmarks of mathematics achievement, 4th grade.
|
From page 134... ...
roughly equivalent to the cut scores on NAEP for similar grades and subject areas. Once the TIMSS and PISA cut scores are determined, it is possible to calculate the percentage of students in other countries who would likely score at each of the NAEP achievement levels.
|
From page 135... ...
Countries are rank ordered by their estimated performance: in Table 5-4 by the percentage of students in the advanced level; in Table 5-5 by the percentage of students at or above the Basic level. The two analyses show fairly consistent results, with the same set of countries appearing in the top 10.
|
From page 136... ...
NAEP = National Assessment of Educational Progress, TIMSS = Trends in International Mathematics Science Study. SOURCE: Data from Hambleton et al.
|
From page 137... ...
In reading, two states' proficiency standards were in NAEP's Proficient range. In many cases, the NAEP scale equivalent for state standards, especially in grade-4 reading, mapped below the NAEP achievement level for Basic.
|
From page 138... ...
TABLE 5-5 Percentage of Students at or Azbove Basic, Proficient, and Advanced in Grade-8 2003 TIMSS 138 Mathematics: Estimated by Linking the Grade-8 2000 NAEP Mathematics Achievement Levels to the Grade-8 1999 TIMSS Mathematics Scale Percentage Percentage Percentage at or Above Margin of Error at or Above Margin of Error at or Above Margin of Error Nation Basic for Basic Proficient for Proficient Advanced for Advanced Singapore 96+ 1.5 73+ 4.6 35+ 6.4 Hong Kong, SAR 95+ 1.7 66+ 5.5 24+ 6.0 Korea, Republic of 92+ 1.8 65+ 4.6 29+ 5.4 Chinese Taipei 88+ 2.4 61+ 4.5 30+ 5.0 Japan 90+ 2.3 57+ 5.1 20+ 4.7 Belgium (Flemish) 82+ 3.7 40 5.6 9 3.0 Netherlands 83+ 4.0 38 6.2 7 3.0 Hungary 77 3.9 37 5.1 9 2.9 Estonia 82+ 4.0 36 5.8 6 2.6 Slovak Republic 68 4.5 28 4.5 6 2.1 Australia 67 4.9 27 4.7 5 2.2 Russian Federation 69 4.8 27 4.8 5 2.0 Malaysia 70 5.1 26 5.0 4 1.9 United States 67 4.7 26 4.4 5 1.9 Latvia 70 4.9 25 4.8 4 1.8 Lithuania 66 4.7 24 4.3 4 1.7 Israel 63 4.6 24 4.0 5 1.8 England 65 5.4 22 4.7 4 1.8 Scotland 65 5.2 22 4.4 3 1.5 New Zealand 63 5.6 21 4.7 3 1.8 Sweden 66 5.2 21 4.3 3 1.3 Serbia 54- 4.5 19 3.2 4 1.3 Slovenia 63 5.2 19 4.0 2 1.1 Romania 53- 5.0 18 3.6 4 1.5 Armenia 54 4.8 18 3.4 3 1.2
|
From page 139... ...
indicates the 95 percent confidence level that the nation's percentage at and above the projected achievement level is greater or less than that in the United States. TIMSS =Trends in International Mathematics and Science Study.
|
From page 140... ...
. The AP courses and tests most closely related to NAEP reading are English language and composition and English literature and composition.
|
From page 141... ...
We compared the percentages of students who scored of 3 or higher and 4 or higher on each AP test with the percentages of students who scored at the Proficient and Advanced levels on NAEP: see Tables 5-8 and 5-9. It is important to note that the percentages of students shown in the tables do not represent the percentages of AP test takers that scored at each level: rather, they 13For details, see http://apcentral.collegeboard.com/apc/public/courses/220300.html [March 2016]
|
From page 142... ...
142 FIGURE 5-7 National Assessment of Educational Progress scale equivalents of states' grade-4 reading standards for Proficient performance, by state: 2009. SOURCE: Bandeira de Mello (2011)
|
From page 143... ...
FIGURE 5-8 National Assessment of Educational Progress scale equivalents of states' grade-8 reading standards for Proficient performance, by state: 2009. SOURCE: Bandeira de Mello (2011)
|
From page 144... ...
144 FIGURE 5-9 National Assessment of Educational Progress scale equivalents of states' grade-4 mathematics standards for Proficient performance, by state: 2009. SOURCE: Bandeira de Mello (2011)
|
From page 145... ...
FIGURE 5-10 National Assessment of Educational Progress scale equivalents of states' grade-8 mathematics standards for Proficient performance, by state: 2009. SOURCE: Bandeira de Mello (2011)
|
From page 146... ...
reported for the 1992 results, thus allowing comparisons with the baseline in 1992.15 For mathematics, Table 5-8 compares data for 3 years: 2005, 2009, and 2013.16 Overall, the percentage of students scoring in the advanced level on NAEP is lower than the percentage of students scoring 3 or higher or 4 or higher for each year shown. The differences are in the range of 2 to 3 percentage points.
|
From page 147... ...
AP = Advanced Placement, NAEP = National Assessment of Educational Progress. SOURCE: Data from the College Board (2009, 2013, 2015)
|
From page 148... ...
AP = Advanced Placement, NAEP = National Assessment of Educational Progress. SOURCE: Moran et al.
|
From page 149... ...
Three values are reported from the projection analysis: the NAEP score at which 50 percent of SAT test takers score 500 or above; the NAEP score at which 67 percent of SAT test takers score 500 or above; and the NAEP score at which 80 percent of SAT test takers score 500 or above. The different percentages (50, 67, and 80, respectively)
|
From page 150... ...
The NAEP cut scores for Proficient are 176 for mathematics and 302 for reading. NAEP = National Assessment of Educational Progress, SAT = Scholastic Aptitude Test.
|
From page 151... ...
CONCLUSIONS Content-Related Validity Evidence ACT conducted studies to collect evidence of content validity for the achievement levels and exemplar items. The content experts that participated in these studies suggested changes to the ALDs and exemplars.
|
From page 152... ...
Since 1992, there have been additional changes to the frameworks, item pools, assessments, and studies to identify needed revisions to the ALDs. But, to date, there has been no effort to set new cut scores using the most current ALDs.21 CONCLUSION 5-2 Changes in the National Assessment of Educational Progress mathematics frameworks in 2005 led to new achievement-level descriptors and a new scale and cut scores for the achievement levels at the 12th grade, but not for the 4th and 8th grades.
|
From page 153... ...
Beaton and colleagues (2012) discuss the use of a predictive approach to determine achievement levels, an approach that places a high priority on the external validity of the cut scores in relation to predetermined criteria.
|
From page 154... ...
achievement levels were set, new research has investigated the relationships between NAEP scores and external measures, such as academic preparedness for college. The findings from this research can be used to eval uate the validity of new interpretations of the existing perfor mance standards, suggest possible adjustments to the cut scores or descriptors, and or enhance understanding and use of the achievement-level results.
|
From page 155... ...
Official -- 1992 Fourth-grade students performing at the Basic level should show some evidence of understanding the mathematical concepts and procedures in the five NAEP content areas. Fourth graders performing at the Basic level should be able to estimate and use basic facts to perform simple computations with whole numbers; show some understanding of fractions and decimals; and solve some simple real-world problems in all NAEP content areas.
|
From page 156... ...
And, there is an ability to explain how the solution was achieved. Official -- 1992 Fourth-grade students performing at the Proficient level should consistently apply integrated procedural knowledge and conceptual understanding to problem solving in the five NAEP content areas.
|
From page 157... ...
Official -- 1992 Fourth-grade students performing at the Advanced level should apply integrated procedural knowledge and conceptual understanding to problem solving in the five NAEP content areas. Fourth graders performing at the Advanced level should be able to solve complex and nonroutine real-world problems in all NAEP content areas.
|
From page 158... ...
correctly, as are one-step problems. However, problems involving the use As students approach the Proficient level, of available data, and determinations they will solve problems involving the of what is necessary and sufficient to use of available data and determine what solve the problem, are generally quite is necessary and sufficient for a correct difficult.
|
From page 159... ...
Eighth graders performing at the Basic level should complete problems correctly with the help of structural prompts such as diagrams, charts, and graphs. They should be able to solve problems in all NAEP content areas through the appropriate selection and use of strategies and technological tools -- including calculators, computers, and geometric shapes.
|
From page 160... ...
The Proficient level denotes a thorough understanding of the arithmetic The Proficient level denotes a thorough operations listed at the Basic level. understanding of the arithmetic This understanding is sufficient to operations listed at the Basic level.
|
From page 161... ...
Quantity and spatial relationships in problem solving and reasoning should be familiar to them, and they should be able to convey underlying reasoning skills beyond the level of arithmetic. They should be able to compare and contrast mathematical ideas and generate their own examples.
|
From page 162... ...
Official -- 1992 Eighth-grade students performing at the Advanced level should be able to reach beyond the recognition, identification, and application of mathematical rules in order to generalize and synthesize concepts and principles in the five NAEP content areas. Eighth graders performing at the Advanced level should be able to probe examples and counterexamples in order to shape generalizations from which they can develop models.
|
From page 163... ...
Calculators are used appropriately to solve problems. Official -- 1992 Twelfth-grade students performing at the Basic level should demonstrate procedural and conceptual knowledge in solving problems in the five NAEP content areas.
|
From page 164... ...
Official -- 2009 Twelfth-grade students performing at the Basic level should be able to solve mathematical problems that require the direct application of concepts and procedures in familiar mathematical and real-world settings. Students performing at the Basic level should be able to compute, approximate, and estimate with real numbers, including common irrational numbers.
|
From page 165... ...
Basic algebraic concepts, measurement, and constructive geometry concepts are mastered at this level. Official -- 1992 Twelfth-grade students performing at the Proficient level should consistently integrate mathematical concepts and procedures to the solutions of more complex problems in the five NAEP content areas.
|
From page 166... ...
Official -- 2009 Twelfth-grade students performing at the Proficient level should be able to recognize when particular concepts, procedures, and strategies are appropriate, and to select, integrate, and apply them to solve problems. They should also be able to test and validate geometric and algebraic conjectures using a variety of methods, including deductive reasoning and counterexamples.
|
From page 167... ...
Twelfth graders performing at the Proficient level should also be able to translate between representations of functions, including verbal, graphical, tabular, and symbolic representations; to use appropriate representations to solve problems; and to use graphing tools and to construct formulas for spreadsheets. Students performing at this level should be able to use technology to calculate summary statistics for distributions of data.
|
From page 168... ...
Official -- 1992 Twelfth-grade students performing at the Advanced level should consistently demonstrate the integration of procedural and conceptual knowledge and the synthesis of ideas in the five NAEP content areas. Twelfth-grade students performing at the Advanced level should understand the function concept; and be able to compare and apply the numeric, algebraic, and graphical properties of functions.
|
From page 169... ...
and make inferences or predictions using the mathematical model. Official -- 2009 Twelfth-grade students performing at the Advanced level should demonstrate in-depth knowledge of and be able to reason about mathematical concepts and procedures.
|
From page 170... ...
Twelfth-grade students performing at the Advanced level should be able to solve problems and analyze properties of three-dimensional figures. They should be able to describe the effects of transformations of geometric figures in a plane or in three dimensions, to reason about geometric properties using coordinate geometry, and to do computations with vectors and to use vectors to represent magnitude and direction.
|
From page 171... ...
Official -- 2009 Fourth-grade students performing at the Basic level should be able to locate relevant information, make simple inferences, and use their understanding of the text to identify details that support a given interpretation or conclusion. Students should be able to interpret the meaning of a word as it is used in the text.
|
From page 172... ...
When reading informational texts such as articles and excerpts from books, fourth-grade students performing at the Basic level should be able to identify the main purpose and an explicitly stated main idea, as well as gather information from various parts of a text to provide supporting information.
|
From page 173... ...
For example, when reading literary text, Proficientlevel fourth graders should be able to summarize the story, draw conclusions about the characters or plot, and recognize relationships such as cause and effect. When reading informational text, Proficient-level students should be able to summarize the information and identify the author's intent or purpose.
|
From page 174... ...
When reading informational texts such as articles and excerpts from books, fourth-grade students performing at the Proficient level should be able to locate relevant information, integrate information across texts, and evaluate the way an author presents information. Student performance at this level should demonstrate an understanding of the purpose for text features and an ability to integrate information from headings, text boxes, graphics and their captions.
|
From page 175... ...
For example, when reading literary text, Advanced-level students should be able to make generalizations about the point of the story and extend its meaning by integrating personal experiences and other readings with the ideas suggested by the text. They should be able to identify literary devices such as figurative language.
|
From page 176... ...
Students should be able to interpret characteristics of poems and evaluate aspects of text organization. When reading informational texts, such as articles and excerpts from books, fourth-grade students performing at the Advanced level should be able to make complex inferences about main ideas and supporting ideas.
|
From page 177... ...
When reading text appropriate to eighth grade, they should be able to identify specific aspects of the text that reflect the overall meaning, recognize and relate interpretations and connections among ideas in the text to personal experience, and draw conclusions based on the text. For example, when reading literary text, Basic-level eighth graders should be able to identify themes and make inferences and logical predictions about aspects such as plot and characters.
|
From page 178... ...
They should be able to state and provide some support for judgments about the way an author presents content and about character motivation. When reading informational texts such as exposition and argumentation, eighth-grade students performing at the Basic level should be able to recognize inferences based on main ideas and supporting details.
|
From page 179... ...
For example, when reading literary text, students at the Proficient level should be able to give details and examples to support themes that they identify. They should be able to use implied as well as explicit information in articulating themes; to interpret the actions, behaviors, and motives of characters; and to identify the use· of literary devices such as personification and foreshadowing.
|
From page 180... ...
They should be able to identify how figurative language is used. When reading informational texts such as exposition and argumentation, eighthgrade students performing at the Proficient level should be able to locate and provide facts and relevant information that support a main idea or purpose, interpret causal relations, provide and support a judgment about the author's argument or stance, and recognize rhetorical devices.
|
From page 181... ...
-- Explaining an author's use of -- Responding personally and critically to literary devices a text -- Analyzing an author's purpose and viewpoint -- sing cultural or historical information U to develop perspectives on a text -- Using cultural or historical information provided in a text to develop perspectives on other situations Official -- 1992 Eighth-grade students performing at the Advanced level should be able to describe the more abstract themes and ideas of the overall text. When reading text appropriate to eighth grade, they should be able to analyze both meaning and form and support their analyses explicitly with examples from the text; they should be able to extend text information by relating it to their experiences and to world events.
|
From page 182... ...
When reading informational texts such as exposition and argumentation, eighthgrade students performing at the Advanced level should be able to infer and explain a variety of connections that are intratextual (such as the relation between specific information and the main idea) or intertextual (such as the relation of ideas across expository and argument texts)
|
From page 183... ...
For example, when reading literary text, 12th-grade students should be able to explain the theme, support their conclusions with information from the text, and make connections between aspects of the text and their own experiences. When reading informational text, Basic-level 12th graders should be able to explain the main idea or purpose of a selection and use text information to support a conclusion or make a point.
|
From page 184... ...
They should be able to interpret the meaning of a word as it is used in the text. When reading literary texts, such as fiction, literary nonfiction, and poetry, twelfth-grade students performing at the Basic level should be able to describe essential literary elements such as character, narration, setting, and theme; provide examples to illustrate how an author uses a story element for a specific effect; and provide interpretations of figurative language.
|
From page 185... ...
These students should be able to analyze the author's use of literary devices. When reading literary text, Proficient-level 12th graders should be able to integrate their personal experiences with ideas in the text to draw and support conclusions.
|
From page 186... ...
Students performing at this level should be able to make and compare generalizations about different characters' perspectives within and across texts. When reading informational texts including exposition, argumentation, and documents, 12th-grade students performing at the Proficient level should be able to integrate and interpret texts to provide main ideas with general support from the text.
|
From page 187... ...
For example, when reading literary text, Advanced-level 12th graders should be able to produce complex, abstract summaries and theme statements. They should be able to use cultural, historical, and personal information to develop and explain text perspectives and conclusions.
|
From page 188... ...
When reading literary texts such as fiction, poetry, and literary nonfiction, 12thgrade students performing at the Advanced level should be able to analyze and evaluate how an author uses literary devices, such as sarcasm or irony, to enhance and convey meaning. They should be able to determine themes and explain thematic connections across texts.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.