National Academies Press: OpenBook

Grading the Nation's Report Card: Research from the Evaluation of NAEP (2000)

Chapter: 2 Families of Items in the NAEP Mathematics Assessment

« Previous: 1 Introduction
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 5
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 6
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 7
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 8
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 9
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 10
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 11
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 12
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 13
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 14
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 15
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 16
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 17
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 18
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 19
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 20
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 21
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 22
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 23
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 24
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 25
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 26
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 27
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 28
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 29
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 30
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 31
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 32
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 33
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 34
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 35
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 36
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 37
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 38
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 39
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 40
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 41
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 42
Suggested Citation:"2 Families of Items in the NAEP Mathematics Assessment." National Research Council. 2000. Grading the Nation's Report Card: Research from the Evaluation of NAEP. Washington, DC: The National Academies Press. doi: 10.17226/9751.
×
Page 43

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Families of Items in the NAEP Mathematics Assessment Patricia Ann Kenney This paper discusses families of items in the National Assessment of Educa- tional Progress (NAEP) mathematics assessment and presents a sample family of items for grade 4. Item families can serve as an illustration of how to more fully understand and describe levels of students' understanding by examining students' responses across a set of related items. The paper presents a brief overview of the NAEP mathematics framework developed for the 1996 assessment, the idea of families of items and how they have appeared in previous NAEP assessments, a rationale for the development of a sample family of items around the topic of number patterns at the fourth-grade level, and the family of items itself based on released NAEP pattern items and other items developed for this paper. OVERVIEW OF THE 1996 NAEP MATHEMATICS FRAMEWORK The 1996 NAEP mathematics assessment used a framework (National Assessment Governing Board, 1994) that was influenced by ideas presented in Curriculum and Evaluation Standards for School Mathematics of the National Council of Teachers of Mathematics (NCTM, 1989~. The framework sampled the content domain of mathematics using five strands: Number Sense, Proper- ties, and Operations; Measurement; Geometry and Spatial Sense; Data Analysis, Statistics, and Probability; and Algebra and Functions. Additional dimensions of the framework included mathematical abilities (Conceptual Understanding, Procedural Knowledge, Problem Solving) and mathematical power (Reasoning, Connections, Communication). In addition to defining the content, ability, and power dimensions along which to assess students' knowledge and understanding

6 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT of mathematics, the NAEP framework document included recommendations con- cerning other aspects of the assessment, such as the distribution of items across the content strands for grades 4, 8, and 12; the use of calculators and manipulatives; and design considerations for special sets of items to appear on the test. One of these special sets of items, called "families of items," is described in more detail below. Families of Items in NAEP The 1996 NAEP mathematics framework document recommends that the assessment include sets of related tasks, called families of items, to "measure the breadth and depth of student knowledge in mathematics" (National Assessment Governing Board, 1994:5~. The framework describes two types of item families: a vertical family and a horizontal family. A vertical family includes items or tasks that measure students' understanding of a single important mathematics concept in a content strand (e.g., numerical patterns in algebra) but at different levels, such as giving a definition, applying the concept in both familiar and novel settings, and generalizing knowledge about the concept to represent a new level of understanding. A horizontal family of items involves assessment of students' understanding of a concept or principle across the various content strands in NAEP within a grade level or across grade levels. For example, the concept of proportionality can be assessed in a variety of content strands, such as number properties and operations, measurement, geometry, probability, and alge- bra. The framework also suggests that a family of items could be related through a common mathematical or real-world context that serves as a rich problem setting for the items. Although the notion of item families was first articulated in the 1996 math- ematics framework document, sets of related items have appeared in prior NAEP assessments. However, these sets exhibited few characteristics of either horizon- tal or vertical item families. Instead, the relationships between items involved such features as a common stimulus (e.g., a table, chart, or graph), a rudimentary form of scaffolding in which one item draws on information from the preceding iambs), or a common context. Each of these relationships is discussed next. The item set in Box 2-1 is an example of the first kind of relatedness in that the items share a common stimulus. These three items were administered to students in the grade 4 sample in 1992 and were classified by NAEP assessment developers in the content strand Numbers and Operations (as the strand was called then). While working on these items, students were permitted to use simple four-function calculators. A table that displayed the number of points earned from two school events for each of three classes served as the common stimulus for the items. The first item asked for a comparison based on adding the points earned by each class, comparing the total points per class, and selecting the class that earned the most points; the second item required students to obtain the

PATRICIA ANN KENNEY 7

8 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT correct total number of points from the Mathathon column; and the third item involved a within-class comparison concerning the number of points earned for each event. Because the items in Box 2-1 were related in only a cursory way (i.e., a table was used as a common stimulus), looking at performance across them reveals little about students' ability to select appropriate data from a chart and their use of arithmetic operations on those data. Performance results were not very different across items, and the results appeared to be related only to the item type; that is, students' performance was higher on the multiple-choice item (66 percent cor- rect) than on either of the short constructed-response items (52 and 49 percent, respectively). It is well documented that, on the NAEP mathematics assessment, student performance is higher on multiple-choice items than on constructed- response items (Dossey et al.,l993; Silver et al., in press). With respect to the item set in Box 2-1, it is reasonable to suspect that guessing could be contributing to the higher performance on the first question (a multiple-choice item), but because the items are not related in other ways it is difficult to interpret perfor- mance among them on the basis of important mathematical concepts and . . pnnclp es. Another way that sets of mathematics items in NAEP were related involved an attempt to scaffold items; that is, a particular item is based on important and purposeful ways on one or more items that precede it. For example, the answer from the first item is used again in later items, or the first item presents a simple concept that is elaborated on in the items that follow (e.g., the "superitems" as described in Collis et al., 1986, and Wilson and Chavarria,1993~. An example of a scaffolded item pair from the 1992 NAEP mathematics assessment appears in Box 2-2. This item pair, administered to students in the grade 4 sample and classified by NAEP assessment developers in the content strand Algebra and Functions, assessed students' understanding of a number relationship involving an arithmetic operation. In the first item, students were asked to identify the rule used to transform the numbers in column A to those in column B. and its com- panion item required them to use that same rule to generate another number in the pattern. Thus, the pair illustrates a simple form of scaffolding in which the first item provides important information to be used in the second item. NAEP results show that 42 percent of the fourth-grade students chose the correct rule (divide the number in column A by 4) in the first item. However, as expected for constructed-response items, performance was lower on item 2: only 24 percent of the fourth-grade students found the correct value of 30. Given that the items were related, it is reasonable to think that students would use the rule they chose in the first item in order to answer the second item, but NAEP results show that this was not the case. For example, of the students who selected the correct rule in item 1, only about 50 percent also answered item 2 correctly, with about another 40 percent giving numerical values that were incorrect and the

PATRICIA ANN KENNEY 9

10 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT remaining 10 percent leaving the item blank. Unfortunately, NAEP does not provide additional information on the kinds of incorrect answers students pro- duced for item 2. Thus, for example, there is no information on ways in which students could have implemented the correct rule from item 1 to get an incorrect answer in item 2. We can only speculate on some of these ways. For example, did students make an error in dividing 120 by 4 and give an answer of 3 instead of 30? Did students arbitrarily change the operation from division to multiplica- tion, giving an answer of 480 (120 x 4~? Did students completely ignore their correct choice of the rule in item 1 and instead perform an arbitrary operation using an arbitrary divisor, such as dividing 120 by 10 for an answer of 12? Based on the way NAEP results were reported for the item pair, we have no answers to these and other questions about the kinds of errors students made. The two kinds of related items just discussed, however, by no means match the definition of families of items in NAEP. And perhaps there is no reason to expect that they would, given that those related items appeared on the 1992 NAEP mathematics assessment and given that the directive about the assessment including item families was first prescribed in the 1996 version of the mathematics framework. Yet even in the 1996 NAEP there were few, if any, true families of items that match the horizontal or vertical description in the mathematics frame- work document. As was the case in 1992, the 1996 assessment included related sets of items, but the majority of these sets included only two or three items that were related by the common stimulus such as a graph or table or that were scaffolded in limited ways. The exception to this was the sets of items related by context and referred to as "theme blocks" (Hawkins et al., 1999~. In the 1996 mathematics assessment the theme blocks were the operationali- zation of the framework recommendation concerning contextually related sets of tasks. According to a recent NAEP report, the questions in each theme block "related to some aspect of a rich problem setting that served as a unifying theme for the entire block" (Reese et al., 1997:79~. The theme blocks were designed as a special study at the national level in NAEP and as such the results were reported separately from the main NAEP assessment. For the 1996 assessment there were five different theme blocks, two at each grade level with one block common to grades 8 and 12. Each block, containing 6 to 10 items, was administered to a special sample of students at each grade level. The item formats included multiple-choice questions and short and extended constructed-response questions developed around important mathematical ideas set in a real-world context. Each student was allotted 30 minutes to complete the questions in the theme block. (To complete the 45-minute testing time allocated to the cognitive items, a student took another block of items consisting entirely of multiple-choice questions that had a 15-minute time limit.) While working on the items in the theme block,

PATRICIA ANN KENNEY 11 students were permitted to use calculators and were provided with other materials such as rulers, protractors, and models. A released grade 4 theme block built around the context of a science fair project on butterflies appears in Appendix 2A.1 While working on the six constructed-response items in this block, students had access to calculators and other materials (butterfly information sheet, butterfly cutouts, ruler). A close look at the six constructed-response items that make up the block reveals that, although all items involved the assessment of important mathematical concepts and all were set in the context of the butterfly display, the items themselves do not satisfy the definition of either a vertical or a horizontal family of items. Instead, the items individually assess mathematical topics that ranged from geometry to measurement to number concepts to proportions to patterns, with few if any connections between the items. The only obvious connection between items was that students had to use the wingspan measurements from the second item in order to answer the sixth item in the block. The performance results for the six theme block items, shown in Table 2-1, cannot be interpreted in any connected way that has to do with the mathematics in the items. Moreover, the difficulty in making connections between performance on the items is exacerbated by differences in the number of score levels. With respect to the only two items that were connected, the results suggest that, although students could measure wingspan (40 percent correct on item 2), they had difficulty using these measurements as part of a complex problem (1 percent correct on item 6~. However, these results were not presented so such direct comparisons can be made between students' performance on those two items. That is, we do not know the percentage of students who, having obtained the correct wingspan measurement in item 2, provided correct answers accompanied by reasonable work in item 6. In recent NAEP mathematics assessments, then, there has been little imple- mentation of the notion of families of items, although the 1996 framework docu- ment makes a case for their inclusion. What might a family of items look like? What kind of information would likely be generated about students' understanding of a particular concept presented at varying levels of complexity in a mathematical content area or their understanding of a concept presented across mathematical content areas? The next section of this paper investigates these questions in light of a suggested family of items structured around number patterns, an informal algebra topic for grade 4. 1A discussion of the grade 4 released theme block (as well as the released theme blocks from the grade 8 and grade 12 NAEP mathematics assessment in 1996) appears in Kenney and Lindquist (in press).

2 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT TABLE 2-1 Performance on Items in the Butterflies Theme Block: Grade 4 Item Description % Scoring at Highest Levela 1. Draw four missing markings on pictures of two butterflies to make each butterfly symmetrical. 2. Obtain two correct measurements in centimeters for the wingspans of two butterfly models. 28 40 3. Determine the greatest number of butterflies that can be stored in a case 5 and the number of cases needed to hold 28 butterflies; show how your answer was obtained. 4. Determine the maximum number of butterfly models that can be made from 3 a given number of parts (wings, bodies, antennae); show or explain how your answer was obtained. 5. Given that two caterpillars eat five leaves per day, determine the number of leaves needed each day to feed 12 caterpillars; show how your answer was obtained. 6. Find the number of each type of butterfly needed to create a repeating pattern on a banner that is 130 centimeters long; show how your answer was obtained. (Note: measurements obtained in item 2 are needed in this problem.) 1 Note: All items in this block were either short or extended constructed-response items. aBecause of differences in the number of levels in scoring guides, the highest score level varied among the items from three to five levels. The highest score levels are as follows: "satisfactory" for items 1, 4, and 6 (based on four score levels); "extended" for items 2 and 3 (based on five score levels); and "complete" for item 5 (based on three score levels). A Sample Family of Items Based on Number Patterns at Grade 4 This section of the paper is divided into three parts. The first part contains a discussion of the importance of number patterns in the elementary mathematics curriculum and of the appropriateness of this topic as the basis for an item family in NAEP. The purpose of the next part is to justify the use of released NAEP mathematics items as the basis for the family of items and the limitations inherent in this method. The concluding part contains the sample item family, an explana- tion of its structure, and the kinds of information that might be obtained from looking at students' performance across the items.

PATRICIA ANN KENNEY Number Patterns in Elementary Mathematics and in NAEP 13 The topic of patterns and relationships, particularly number patterns in elementary school mathematics, is an appropriate and important content topic around which to create a family of items. Exploring patterns helps students in the early grades develop the ability to think algebraically (Armstrong, 1995; National Council of Teachers of Mathematics [NCTMi, 1989; Reys et al., 1995~. In fact, the NCTM Algebra Working Group realized that "children can develop algebraic concepts at an early age" (NCTM, 1994:5) and suggested that working with patterns of shapes and numbers helps to build the foundation for algebraic think- ing needed in later grades. In addition to information about the importance of patterns and relationships in the elementary mathematics curriculum, there exists a body of research that examines elementary students' performance on various types of numerical series and pattern items (e.g., Holzman et al., 1982, 1983; Pellegrino and Glaser,1982~. Included in these studies is important information on the nature of pattern items and possible factors that affect their ease or difficulty. For example, Holzman et al. (1982) report that the degree of difficulty of a numerical pattern item depends on such factors as the operations used to generate the pattern (i.e., incrementing operations [addition, multiplication] are easier than decrementing operations [sub- traction and divisional; the number of operations used to generate subsequent numbers in the pattern (i.e., patterns based on one operation [e.g., add 41 are easier than those based on two operations [e.g., first multiply by 2 and then add 11~; and the magnitude of the numbers used (i.e., patterns based on increments or decrements of small numbers [5 or less] are easier than those based on larger numbers [11 or greater]~. The importance of patterns and relationships in the elementary mathematics curriculum is further supported by the fact that the NAEP mathematics frame- work included this topic at grade 4. There is evidence that patterns, particularly numerical patterns, were a topic assessed in NAEP. Within the Algebra and Functions content strand at grade 4, recent NAEP mathematics assessments have included items that assess informal algebraic thinking through patterns and rela- tionships. In 1992 about 10 percent of the items on the grade 4 assessment dealt with informal algebra, and most of those items involved patterns of figures, symbols, or numbers (Kenney and Silver, 1997~. The 1996 assessment had about the same percentage of items at grade 4, and most of them also involved a variety of patterns, including number patterns. Thus, given the importance of patterns in both the elementary mathematics curriculum and recent NAEP mathematics assessments as well as a research base that speaks to characteristics of pattern items that can affect performance, selecting this as the topic for a family of items was both reasonable and appropriate.

4 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT Released NAEP Pattern Items as the Basis for an Item Family Not only did numerical pattern items appear on recent NAEP mathematics assessments, but also some of those items were released to the public. These released pattern items were not part of an item family in the assessment; instead, they appeared as single items in various parts of the assessment. However, because released pattern items and related performance data on those items were available from NAEP, it seemed reasonable to use these single items along with appropriate supplemental items to form a sample family of items. The advantage of this method of constructing a sample family of items is that the sample family uses items that have already appeared on a NAEP assessment, and we know how students performed on them. In addition, using existing items enabled the item family to be created with minimal time devoted to the development of original items. Finally, the existing items could be evaluated not only according to student performance but also using findings from published research studies (e.g., Holzman et al., 1982) and then organized into a family in a somewhat hierarchical fashion reflecting the level of cognitive demand. However, this method of creating a sample item family using existing NAEP items carries with it potentially serious limitations. In particular, taking items developed individually and putting them together as a set post hoc has a degree of artificiality. The ideal method that should be used to create an item family is to begin with a particular topic and information based on research about students' understanding of that topic, build the family of items, and then validate the structure of the set by administering it to students and examining performance results. Obviously, this method was not used in this paper. As a consequence, the family of items presented herein should be considered as an illustrative but very modest example of what such a family might look like, with the understand- ing that better families of items will be created for future NAEP assessments. It is hoped that the simple example in this paper will be used as the basis for further thought about and discussion of important features of families of items in NAEP. A Proposed Item Family Based on Numerical Patterns The six items in Appendix 2B constitute a proposed family of items built around the topic of numerical patterns. The set was developed according to the following guidelines: · Each item in the set involves an increasing pattern of numbers based on a particular rule that governs the growth. In the elementary mathematics curricu- lum, these kinds of patterns are often referred to as "growing patterns" (Reys et al., 1995; NCTM, 1992~. In some items the pattern is based on constant increases between consecutive terms, and in others the pattern is based on nonconstant Increases.

PATRICIA ANN KENNEY 15 · The set represents an attempt to organize the items from the easiest to the most difficult. In the case of released NAEP items, performance data were used to determine the level of difficulty (e.g., an item for which performance was 75 percent correct was "easier" than an item for which performance was 53 percent correct). For items created especially for the set, the degree of difficulty was based either on a rational analysis of mathematical concepts or on factors found to influence the difficulty of number pattern item, such as kind of operation, number of operations, and magnitude of numbers, as presented in published research articles (e.g., Holzman et al., 1982~. . In some cases the items are presented in two formats: multiple choice and constructed response. Given that the NAEP has always advocated a judicious blend of multiple-choice and constructed-response items, presenting an alterna- tive format for items (especially those developed specifically for the sample item family) seemed to be appropriate. However, because of the performance differ- ences in NAEP concerning lower percent-correct results on constructed-response items, this could affect the hierarchy of items (easiest to more difficult) in the sample set. The source is given for each item in Appendix 2B (e.g., a released NAEP item; an item created for the set). Following each item is a rationale for why the item was included in the item family and the kind of information that could be obtained from performance results. Figure 2-1 summarizes the concepts and progression of items in the sample family. Performance on these related items could provide insights into students' understanding of numerical patterns and where that understanding falters. For example, performance results could show that most fourth-grade students can work with patterns involving constant increases between the terms (items 1, 2, and 3), but performance levels could be lower for items involving patterns based on nonconstant increases (items 4 and 5), especially for complex problems (item 6~. Performance results could also provide information on misunderstandings that students have about number pat- terns, with the same misunderstandings possibly occurring across items in the family. For example, some students may expect a number pattern always to have a constant increase between contiguous numbers. In this case, when faced with a pattern containing nonconstant increases such as the number pattern in item 4 (i.e., 8, 9, 12, 17, 24, 33, 44, . . . [increase based on the set of odd numbers]), those students could reason that, because the increase between the last two numbers shown in the pattern is 11, 55 (44 + 11) is the next number in the pattern. Because the next two items in the family (items 5 and 6) also involve nonconstant increases, results from those items can provide additional evidence about this misunderstanding. Some might argue that such information about students' understanding and misunderstandings of numerical patterns is already available from the NAEP mathematics assessment results. All that would be needed would be to analyze

16 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT ITEM 1 Constant increase by a one-digit number ITEM 2 Constant increase by a two-digit number ITEM 3 Constant increase by a multiplicative setting ITEM 4 Nonconstant increase based on odd numbers (Note: represents transition to a more advanced pattern concept) ITEM 5 Nonconstant increase ("decreasing increase") ITEM 6 Nonconstant increase in a complex, problem solving setting FIGI~E 2-1 Progression of concepts in the number pattern family of items in Appendix 2B.

PATRICIA ANN KENNEY 17 the performance results from the pattern items included on the assessment. While there is some truth to this argument, the fact remains that most NAEP items (other than the item pairs or triples or the theme block items) are discrete; that is, each item is essentially unrelated to any other item in the assessment. Therefore, identifying the numerical pattern items in NAEP and then analyzing the perfor- mance data as if those items had been developed as an intact set is likely to result in information about students' understanding that is fragmented and difficult to interpret. In fact, an attempt was made by Blume and Heckman (1997) to analyze student performance on the set of seven numerical pattern items that appeared on the grade 4 1992 NAEP mathematics assessment. Of those seven items, only two were related the scaffolded item pair shown in Box 2-2. Because there were so few items and because most of them were discrete, Blume and Heckman were able to draw only limited conclusions about what students know about numerical patterns, such as: "ENumber] patterns based on non-constant rates of change [e.g., the pattern shown above based on increasing by the set of odd numbers] are more difficult for fourth-grade students than patterns based on constant rates of change" (1997:232~. If those seven number pattern items had been developed as an item family, perhaps Blume and Heckman could have told a different story about student performance. As stated earlier, the advantage of an item family is that the items are purposely developed to be related in ways that could illuminate students' understandings and misunderstanding of important mathematical concepts. Ana- lyzing the performance data from a related set of items, then, is more likely to provide results that are connected and interpretable. Also, expanding NAEP to include qualitative analyses of within-student performance across all items in a family (without identifying individual students) could provide information about patterns of performance concerning general knowledge about a particular topic and where such knowledge breaks down. This kind of information has the potential to make NAEP results more meaningful to teachers, parents, and others, thus satisfying an important recommendation of the National Research Council (1999~. CONCLUDING COMMENTS The recommendation in the 1996 framework document about including families of items represents a positive direction for future NAEP mathematics assessments. The inclusion of item families can increase NAEP's potential to provide important information about the depth of students' knowledge in a particular content strand and across strands. The example given in this paper presents only one very limited way in which items can be related to make a family and how the results can be analyzed to provide a more complete picture of students' understanding. As stated previously, the best way to develop families of items is de nova, that is, after determining in advance the desired concepts to

18 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT be assessed and examining existing research on mathematics learning in content areas. Current research does exist concerning a variety of mathematical topics such as fractions (e.g., Mack, 1990, 1995), decimals (e.g., Hiebert and Wearne, 1985), and probability (e.g., Jones et al., 1997) that could be used in the develop- ment of vertical item families, that is, those that assess students' understanding of important topics at a variety of levels. It would also be wise to consider examples from the mathematics education literature that have some of the same features as described in the NAEP math- ematics framework for horizontal item families that assess understanding of a mathematical concept (e.g., proportionality) across content strands. For example, in their framework for assessing conceptual understanding, Zawojewski and Silver (1998) propose that one way to ascertain the robustness of students' under- standing of important mathematical concepts is to vary the context (e.g., content topics such as measurement, geometry, and number) in which the concept is presented. Their examples of "constellations" or collections of tasks that assess related aspects of a target concept might be useful to future development of related items in the NAEP. By carefully choosing important mathematical topics and basing the devel- opment on prior research on the learning of mathematics, developers of the NAEP mathematics assessment can make better use of the idea of families of items. Reporting methods must also reflect the integrated nature of those item families. In particular, scoring schemes should be developed that link results on sets of items and perhaps even interpret those results in light of the concept assessed in the item faculty. It is hoped that future NAEP assessments will include item families that reflect the intentions for the special sets of items in the 1996 NAEP mathematics framework document and that NAEP reports will take full advantage of the information provided by these related item sets. REFERENCES Armstrong, B.E. 1995 Teaching patterns, relationships, and multiplication as worthwhile mathematical tasks. Teaching Children Mathematics 1:446-450. Blume, G.W., and D.S. Heckman 1997 What do students know about algebra and functions? Pp. 225-277 in Results from the Sixth Mathematics Assessment of the National Assessment of Educational Progress, P.A. Kenney and E.A Silver, eds. Reston, Va.: National Council of Teachers of Mathematics. Collis, K.F., T.A. Romberg, and M.E. Jurdak 1986 A technique for assessing mathematical problem-solving ability. Journalfor Research in Mathematics Education 17:206-221. Dossey, J.A., I.V.S. Mullis, and C.O. Jones 1993 Can Students Do Mathematical Problem Solving? Results from Constructed-Response Questions in NAEP's 1992 Mathematics Assessment. Washington, D.C.: National Center for Education Statistics.

PATRICIA ANN KENNEY 19 Hawkins, E.F., J.H. Mitchell, F.B. Stancavage, and J.A. Dossey 1999 Estimation Skills, Mathematics-in-Context, and Advanced Skills in Mathematics: Results from Three Studies of the National Assessment of Educational Progress 1996 Mathematics Assessment. National Center for Education Statistics, Washington, D.C. Hiebert, J., and D. Wearne 1985 A model of students' decimal computation procedures. Cognition and Instruction 2: 175-205. Holzman, T.G., J.W. Pellegrino, and R. Glaser 1982 Cognitive dimensions of numerical rule induction. Journal of Educational Psychology 74:360-373. 1983 Cognitive variables in series completion. Journal of Educational Psychology 75:603-618. Jones, G.A., C.A. Thornton, C.W. Langrall, T.M. Johnson, and J.E. Tarr 1997 Assessing and Using Students' Probabilistic Thinking to Inform Instruction. Paper pre- sented at the research precession to the annual meeting of the National Council of Teachers of Mathematics, April, Minneapolis, Minn. Kenney, P.A., and M.M. Lindquist in Performance of students on thematically related NAEP tasks. In Results from the Seventh press Mathematics Assessment of the National Assessment of Educational Progress, E.A. Silver and P.A. Kenney, eds. Reston, Va.: National Council of Teachers of Mathematics. Kenney, P.A., and E.A. Silver 1997 Probing the foundations of algebra: Grade 4 pattern items in NAEP. Teaching Children Mathematics 3:268-274. Mack, N.K. 1990 Learning fractions with understanding: Building on informal knowledge. Journal for Research in Mathematics Education 21:16-32. 1995 Confounding whole-number and fraction concepts when building on informal knowledge. Journalfor Research in Mathematics Education 26:422-441. National Assessment Governing Board 1994 Mathematics Framework for the 1996 and 2000 National Assessment of Educational Progress. Washington, D.C.: National Assessment Governing Board. National Council of Teachers of Mathematics (NCTM) 1989 Curriculum and Evaluation Standards for School Mathematics. Reston, Va.: NCTM. 1992 Curriculum and Evaluation Standards for School Mathematics Addenda Series, Grades K-6: Fourth-Grade Book. Reston, Va.: NCTM. 1994 A Framework for Constructing a Vision of Algebra. Reston, Va.: NCTM. National Research Council 1999 Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress, J.W. Pellegrino, L.R. Jones, and K.J. Mitchell, eds. Committee on the Evaluation of National and State Assessments of Educational Progress, Board on Testing and Assessment. Washington, D.C.: National Academy Press. Pellegrino, J.W., and R. Glaser 1982 Analyzing aptitudes for learning: Inductive reasoning. Pp. 269-345 in Advances in Instructional Psychology, Vol. 2, R. Glaser, ed. Hillsdale, N.J.: Lawrence Erlbaum Associates. Reese, C.M., K.E. Miller, J. Mazzeo, and J.A. Dossey 1997 NAEP 1996 Mathematics Report Card for the Nation and the States: Findings from the National Assessment of Educational Progress. Washington, D.C.: National Center for Education Statistics. Reys, R.E., M.N. Suydam, and M.M. Lindquist 1995 Helping Children Learn Mathematics, Fourth Edition. Boston: Allyn and Bacon.

20 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT Silver, E.A., C. Alacaci, and D. Stylianou in Students' performance on extended constructed response tasks. In Results from the press Seventh Mathematics Assessment of the National Assessment of Educational Progress, E.A. Silver and P.A. Kenney, eds. Reston, Va.: National Council of Teachers of Math- ematics. Wilson, L.D., and S. Chavarria 1993 Superitem tests as a classroom assessment tool. Pp. 135-142 in Assessment in the Math- ematics Classroom: 1993 Yearbook of the National Council of Teachers of Mathematics, N.L. Webb and A.F. Coxford, eds. Reston, Va.: National Council of Teachers of Math- ematics. Zawojewski, J.A., and E.A. Silver 1998 Assessing conceptual understanding. Pp. 287-295 in Classroom Assessment in Math- ematics: Views from a National Science Foundation Working Conference, G.W. Bright and J.M. Joyner, eds. Lanham, Md.: University Press of America.

PATRICIA ANN KENNEY APPENDIX 2A: RELEASED GRADE 4 THEME BLOCK: 1996 NAEP MATHEMATICS ASSESSMENT Materials Used in the Grade 4 Theme Block 21 11~17'1'1;.RPt Y INE0RM.A'I {~N SIlE}-T I ]~) r~ Loot A.... ~ 1:<C31ES IN I~ COMMON BLUE BUTTERFLY CUT-OUTS RULER SOURCE: Hawkins et al. (1999~.

22 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT This part has 6 questions. Mark your answers in your booklet. You will have to fill in an oval or write your answer as directed. In those questions where you must mite Art answer, it is impotent that your answer be clear and complete and that you show ~ of your work since partial credit may be awarded. Some questions may each require 5 minutes or more to think about and answer. After each question, fill in the oval to indicate whether you used the calculator. Use the packet you have been given to help you answer the questions in this section. = - Each class ~ Oakville School will have a boom at the Science Fair. Your class is planing to have a Butterfly Booth. Your class has a lot to do to get ready far the Science Fair. You need to make decorations for the booth, plan activities, and order matenals.

PATRICIA ANN KENNEY 1. The butterfly booth will be decorated with butterfly drawings. Draw only the missing markings on each picture to make each butterfly symmetncal. Did you use the calculator on this questions O Yes O No a. Take the ButterBy Information Sheet hom your packet. On the Butterfly Information Sheet the wingspan of the Monarch butterfly is shown. Use your ruler to measure the wingspans of the other two butterflies on the sheet, the Black Swallowtail butterfly and the Common Blue butterfly, to the nearest centimeter. Black Swallowtail Wingspan: centimeters Common Blue Wingspan: centuneters Did you use the calculator on this questions O Yes O No ~3 - l':

24 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT 3. Take the butterfly cutouts from your packet. What is the greatest number of Common Blue butterflies that can be stored in the case below? (When you put butterflies in the case, you can't stack ~em. The butterflies con touch, but they can't overlap at At-) Answer: _ Show how the butterflies fit in the case. Storage Case ~- -- 6 centimeters 12 centimeters

Hov may stole cases Would you need to store 28 Common Blue butters Isle: Use ~g~ wordy or Ibex to ~1^ boa you got your asker. Did you use Me cunctator ~ Is Despond O As 23 No

26 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT 4. The children who visit your booth are going to build models of butterflies. For each model' they will need the following: 4 wing pieces When the model is put together it looks like this: Cad l body 2 antennae fl If the class has a supply of 29 wigs, 8 bodies, and 13 antennae, how many complete butterfly models can be made2 Answer: Use drawings, words, or numbers to explain how you got your answer. Did you use the calculator on this questions O Yes O No

PATRICIA ANN KENNEY 5. A fourth-~ade class needs ~ leaves each day to feed its 2 caterpillars. How mostly leaves would they need each day for 12 caterpillars? AIlswer: Use drawings, words, or numbers to show how you got your answer. Did you use He calculator on this question? O Yes O No 27

28 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT 6. Use the Butterfly Information Sheet and your answer from question 2 to solve this question. Your class has decided to have a banner that will be 130 centuneters long. This banner will have a repeating pattern of one Monarch butterfly followed by two Black Swallowtail butterflies, as shown here. en This part keeps repeating across the banner. The butterflies will just touch but will not overlap. Monarch ~ _. L ~J Black Swallowtails How many of each type of butterfly are needed for The banner? Monarch Black Swallowing Show how you got your answers.

PATRICIA ANN KENNEY If you need more room for your work, use the space below. Did you use the calculator on this question? O Yes O No 29 5

30 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT APPENDIX 2B: SAMPLE FAMILY OF ITEMS BASED ON NUMBER PATTERNS: GRADE 4

PATRICIA ANN KENNEY 3 Rationale for Item 1 This item would appear first in the family because nearly all the fourth-grade students should be able to produce a correct answer based on the constant in- crease of 6 between the numbers in the pattern. This conjecture concerning the ease of the item is supported by characteristics suggested in Holzman et al. (1982~; that is, the pattern is based on a single incrementing operation (addition) and the use of a relative small number (6) as the increment between terms in the pattern. The first version (multiple choice, no context) would best serve the purpose of determining the floor effect. The other versions are presented here as addi- tional examples of simple pattern items based on a single-digit, constant increase between consecutive numbers. The last version set within a context could possi- bly be too difficult to appear as the first item in the set, but its multiple-choice format could make it more accessible to fourth-grade students.

32 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT

PATRICIA ANN KENNEY 33 Rationale for Item 2 A version of the 1992 NAEP item would appear next in the item family. Although the pattern of numbers is still constantly increasing and although the operation is still addition, the increase itself is a double-digit number greater than 11, which adds to the difficulty of the item for elementary school students (Holzman et al., 1982~. Despite the fact that the increase is a multiple of both 5 and 10 and that the increase is given in the problem, this item is considered as a "step up" from the first problem because of its constructed-response format and the need to work with a pattern involving a two-digit number increase. The NAEP version, however, should be modified so that more information can be obtained from student responses. In particular, the original NAEP item asked for three numbers in the pattern based on a given constant increase of 50 students. The results showed that just over half the fourth-grade students gave completely correct responses. However, the results did not reveal which of the three numbers was the most difficult to obtain. The version proposed for the item family could remedy this situation by providing information on whether the students understood that the enrollment increases in the first year by 50 students, and then by that same number in each of the next two years.

34 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT

PATRICIA ANN KENNEY 35

36 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT Rationale for Item 3 The third item in the family represents a transition from patterns based on addition of a constant to patterns based on multiplicative models. This item would reveal whether students understand that patterns can be based on arith- metic operations other than addition, for example, multiplication or division. The original NAEP item was discussed in an earlier section of this paper, and it was noted there that the results did not completely reveal the degree of consis- tency between the rule selected by students and whether they used that rule to answer the second question. Using one of the revised versions, both of which are constructed-response questions, perhaps we can better relate the students' de- scription of the rule in part 1 and their use (or misuse) of that rule in part 2. For example, in version 1 for students who answered "Divide the number in Column A by 4," but who wrote "3" in Column B in the second part of the problem, we could more accurately attribute this incorrect answer to a place-value error or perhaps to carelessness. For other students who wrote the correct rule, but who answered "480" in the second part, it is likely that their error involved multiply- ing instead of dividing. With respect to the two versions suggested for the family, one version might be preferable over the other depending on whether the multiplicative model (Ver- sion 2) or the division model (Version 1) is more easily recognized by students. Perhaps both versions could be pilot-tested to answer this question, with only one version included in the item family.

PATRICIA ANN KENNEY Rationale for Item 4 The fourth item, presented in two versions (multiple choice and constructed response), serves as a transition between numerical patterns based on constant increases to those based on nonconstant increases. In an important way, nonconstant increases are in themselves a pattern within a pattern. For example, the pattern in the item (8, 9, 12, 17, 24, 33, 44, . . . ~ also has a pattern of increases (1, 3, 5, 7, 9, . . . ~ the set of odd numbers. Because the notion of nonconstant increases is likely to be difficulty for some fourth-grade students, basing the nonconstant increases on the set of odd numbers could make the item more accessible. Also, the operation used to create the pattern is again simple addition. As noted earlier in the paper, this item and the ones that follow could provide evidence about an important misunderstanding about patterns; that is, the notion that all patterns (even those that are based on nonconstant differences) contain pairs of numbers that have a constant difference. For item 4 in the family, it is likely that some students could choose B (55) for the multiple-choice version or write 31 and 38 as the next two numbers in the pattern for the constructed- response version. In both cases, such responses show evidence of changing the nonconstant increase to a constant increase based on the difference between the last two numbers shown in the pattern.

38 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT

PATRICIA ANN KENNEY 39

40 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT Rationale for Item 5 This item within the family has the potential to be the most difficult question to this point. Results from the original NAEP version of the item showed that about the same percent of students selected choice C (25 pounds) as selected the correct choice D (24 pounds). This error pattern shows that some students may expect a number pattern to have a constant difference between some contiguous numbers: that is, in the puppy problem, students retained the 3-pound weight gain between the third and fourth months and used it as a constant to calculate the weight at 5 months (22 + 3 = 25~. Also, the high omitted and not-reached rates suggest that some fourth-grade students thought that this problem was so difficult that they did not even try to answer it. The version proposed for the item family attempts to make the question more accessible to students. It is scaffolded so that students must identify the first two nonconstant differences between the weights, in the hope that students will more easily recognize that the weight gains are decreasing between consecutive months. The final question involves a transition from the nonconstant differences to the actual weight of the puppy. As for Item 4 in the family, this item has the potential to provide additional evidence of the misunderstanding about nonconstant increases. Despite the at- tempt at scaffolding, students could still change to a constant increase and answer 25 pounds or some other number based on a constant increase in weight.

PATRICIA ANN KENNEY 4

42 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT

PATRICIA ANN KENNEY 43 Rationale for Item 6 The original NAEP item, called Marcy's Dot Pattern in NAEP reports (e.g., Dossey et al., 1993), was administered to students in the 1992 eighth-grade sample as an extended constructed-response question in the Algebra and Func- tions content strand. As shown by the performance results, this question was difficult for the eighth-grade students: only 6 percent produced a response that was scored as satisfactory or extended. However, the fact that the item was last in an item block with previous items having little or no connection to number patterns could have affected performance levels. How would students have performed if this question, or an appropriate version thereof, appeared in a family of items devoted to number patterns? Given the structure of the family of items describe thus far, it seemed reason- able to think about including an adaptation of the Marcy's Dot Pattern as the culminating item in the family. As the culminating item, it has characteristics based on work done on the previous items. For example, one way to view the pattern in this task is that the pattern involves nonconstant increases between the number of dots in each step. Solving the problem requires students to identify the rule that underlies the pattern of nonconstant increases. The version for the item family begins with an introductory question about the number of dots in the fourth step as a way to introduce students to the problem. Here, it would be reasonable for students to draw the fourth figure so that they can better under- stand the pattern. The next part of the problem is similar to that given to students in the eighth-grade sample, but the steps are reduced from the 20th step to the 10th step. This last decision needs careful thought, however, because drawing 7 more sets of dots is more accessible than drawing 17 more sets.

Next: 3 Student Thinking and Related Assessment: Creating a Facet-Based Learning Environment »
Grading the Nation's Report Card: Research from the Evaluation of NAEP Get This Book
×
Buy Paperback | $55.00 Buy Ebook | $43.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The National Assessment of Educational Progress (NAEP), known as the nation's report card, has chronicled students' academic achievement in America for over a quarter of a century. It has been a valued source of information about students' performance, providing the best available trend data on the academic achievement of elementary, middle, and secondary school students in key subject areas. NAEP's prominence and the important need for stable and accurate measures of academic achievement call for evaluation of the program and an analysis of the extent to which its results are reasonable, valid, and informative to the public.

This volume of papers considers the use and application of NAEP. It provides technical background to the recently published book, Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress (NRC, 1999), with papers on four key topics: NAEP's assessment development, content validity, design and use, and more broadly, the design of education indicator systems.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!