Click for next page ( 6


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 5
Families of Items in the NAEP Mathematics Assessment Patricia Ann Kenney This paper discusses families of items in the National Assessment of Educa- tional Progress (NAEP) mathematics assessment and presents a sample family of items for grade 4. Item families can serve as an illustration of how to more fully understand and describe levels of students' understanding by examining students' responses across a set of related items. The paper presents a brief overview of the NAEP mathematics framework developed for the 1996 assessment, the idea of families of items and how they have appeared in previous NAEP assessments, a rationale for the development of a sample family of items around the topic of number patterns at the fourth-grade level, and the family of items itself based on released NAEP pattern items and other items developed for this paper. OVERVIEW OF THE 1996 NAEP MATHEMATICS FRAMEWORK The 1996 NAEP mathematics assessment used a framework (National Assessment Governing Board, 1994) that was influenced by ideas presented in Curriculum and Evaluation Standards for School Mathematics of the National Council of Teachers of Mathematics (NCTM, 1989~. The framework sampled the content domain of mathematics using five strands: Number Sense, Proper- ties, and Operations; Measurement; Geometry and Spatial Sense; Data Analysis, Statistics, and Probability; and Algebra and Functions. Additional dimensions of the framework included mathematical abilities (Conceptual Understanding, Procedural Knowledge, Problem Solving) and mathematical power (Reasoning, Connections, Communication). In addition to defining the content, ability, and power dimensions along which to assess students' knowledge and understanding

OCR for page 5
6 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT of mathematics, the NAEP framework document included recommendations con- cerning other aspects of the assessment, such as the distribution of items across the content strands for grades 4, 8, and 12; the use of calculators and manipulatives; and design considerations for special sets of items to appear on the test. One of these special sets of items, called "families of items," is described in more detail below. Families of Items in NAEP The 1996 NAEP mathematics framework document recommends that the assessment include sets of related tasks, called families of items, to "measure the breadth and depth of student knowledge in mathematics" (National Assessment Governing Board, 1994:5~. The framework describes two types of item families: a vertical family and a horizontal family. A vertical family includes items or tasks that measure students' understanding of a single important mathematics concept in a content strand (e.g., numerical patterns in algebra) but at different levels, such as giving a definition, applying the concept in both familiar and novel settings, and generalizing knowledge about the concept to represent a new level of understanding. A horizontal family of items involves assessment of students' understanding of a concept or principle across the various content strands in NAEP within a grade level or across grade levels. For example, the concept of proportionality can be assessed in a variety of content strands, such as number properties and operations, measurement, geometry, probability, and alge- bra. The framework also suggests that a family of items could be related through a common mathematical or real-world context that serves as a rich problem setting for the items. Although the notion of item families was first articulated in the 1996 math- ematics framework document, sets of related items have appeared in prior NAEP assessments. However, these sets exhibited few characteristics of either horizon- tal or vertical item families. Instead, the relationships between items involved such features as a common stimulus (e.g., a table, chart, or graph), a rudimentary form of scaffolding in which one item draws on information from the preceding iambs), or a common context. Each of these relationships is discussed next. The item set in Box 2-1 is an example of the first kind of relatedness in that the items share a common stimulus. These three items were administered to students in the grade 4 sample in 1992 and were classified by NAEP assessment developers in the content strand Numbers and Operations (as the strand was called then). While working on these items, students were permitted to use simple four-function calculators. A table that displayed the number of points earned from two school events for each of three classes served as the common stimulus for the items. The first item asked for a comparison based on adding the points earned by each class, comparing the total points per class, and selecting the class that earned the most points; the second item required students to obtain the

OCR for page 5
PATRICIA ANN KENNEY 7

OCR for page 5
8 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT correct total number of points from the Mathathon column; and the third item involved a within-class comparison concerning the number of points earned for each event. Because the items in Box 2-1 were related in only a cursory way (i.e., a table was used as a common stimulus), looking at performance across them reveals little about students' ability to select appropriate data from a chart and their use of arithmetic operations on those data. Performance results were not very different across items, and the results appeared to be related only to the item type; that is, students' performance was higher on the multiple-choice item (66 percent cor- rect) than on either of the short constructed-response items (52 and 49 percent, respectively). It is well documented that, on the NAEP mathematics assessment, student performance is higher on multiple-choice items than on constructed- response items (Dossey et al.,l993; Silver et al., in press). With respect to the item set in Box 2-1, it is reasonable to suspect that guessing could be contributing to the higher performance on the first question (a multiple-choice item), but because the items are not related in other ways it is difficult to interpret perfor- mance among them on the basis of important mathematical concepts and . . pnnclp es. Another way that sets of mathematics items in NAEP were related involved an attempt to scaffold items; that is, a particular item is based on important and purposeful ways on one or more items that precede it. For example, the answer from the first item is used again in later items, or the first item presents a simple concept that is elaborated on in the items that follow (e.g., the "superitems" as described in Collis et al., 1986, and Wilson and Chavarria,1993~. An example of a scaffolded item pair from the 1992 NAEP mathematics assessment appears in Box 2-2. This item pair, administered to students in the grade 4 sample and classified by NAEP assessment developers in the content strand Algebra and Functions, assessed students' understanding of a number relationship involving an arithmetic operation. In the first item, students were asked to identify the rule used to transform the numbers in column A to those in column B. and its com- panion item required them to use that same rule to generate another number in the pattern. Thus, the pair illustrates a simple form of scaffolding in which the first item provides important information to be used in the second item. NAEP results show that 42 percent of the fourth-grade students chose the correct rule (divide the number in column A by 4) in the first item. However, as expected for constructed-response items, performance was lower on item 2: only 24 percent of the fourth-grade students found the correct value of 30. Given that the items were related, it is reasonable to think that students would use the rule they chose in the first item in order to answer the second item, but NAEP results show that this was not the case. For example, of the students who selected the correct rule in item 1, only about 50 percent also answered item 2 correctly, with about another 40 percent giving numerical values that were incorrect and the

OCR for page 5
PATRICIA ANN KENNEY 9

OCR for page 5
10 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT remaining 10 percent leaving the item blank. Unfortunately, NAEP does not provide additional information on the kinds of incorrect answers students pro- duced for item 2. Thus, for example, there is no information on ways in which students could have implemented the correct rule from item 1 to get an incorrect answer in item 2. We can only speculate on some of these ways. For example, did students make an error in dividing 120 by 4 and give an answer of 3 instead of 30? Did students arbitrarily change the operation from division to multiplica- tion, giving an answer of 480 (120 x 4~? Did students completely ignore their correct choice of the rule in item 1 and instead perform an arbitrary operation using an arbitrary divisor, such as dividing 120 by 10 for an answer of 12? Based on the way NAEP results were reported for the item pair, we have no answers to these and other questions about the kinds of errors students made. The two kinds of related items just discussed, however, by no means match the definition of families of items in NAEP. And perhaps there is no reason to expect that they would, given that those related items appeared on the 1992 NAEP mathematics assessment and given that the directive about the assessment including item families was first prescribed in the 1996 version of the mathematics framework. Yet even in the 1996 NAEP there were few, if any, true families of items that match the horizontal or vertical description in the mathematics frame- work document. As was the case in 1992, the 1996 assessment included related sets of items, but the majority of these sets included only two or three items that were related by the common stimulus such as a graph or table or that were scaffolded in limited ways. The exception to this was the sets of items related by context and referred to as "theme blocks" (Hawkins et al., 1999~. In the 1996 mathematics assessment the theme blocks were the operationali- zation of the framework recommendation concerning contextually related sets of tasks. According to a recent NAEP report, the questions in each theme block "related to some aspect of a rich problem setting that served as a unifying theme for the entire block" (Reese et al., 1997:79~. The theme blocks were designed as a special study at the national level in NAEP and as such the results were reported separately from the main NAEP assessment. For the 1996 assessment there were five different theme blocks, two at each grade level with one block common to grades 8 and 12. Each block, containing 6 to 10 items, was administered to a special sample of students at each grade level. The item formats included multiple-choice questions and short and extended constructed-response questions developed around important mathematical ideas set in a real-world context. Each student was allotted 30 minutes to complete the questions in the theme block. (To complete the 45-minute testing time allocated to the cognitive items, a student took another block of items consisting entirely of multiple-choice questions that had a 15-minute time limit.) While working on the items in the theme block,

OCR for page 5
PATRICIA ANN KENNEY 11 students were permitted to use calculators and were provided with other materials such as rulers, protractors, and models. A released grade 4 theme block built around the context of a science fair project on butterflies appears in Appendix 2A.1 While working on the six constructed-response items in this block, students had access to calculators and other materials (butterfly information sheet, butterfly cutouts, ruler). A close look at the six constructed-response items that make up the block reveals that, although all items involved the assessment of important mathematical concepts and all were set in the context of the butterfly display, the items themselves do not satisfy the definition of either a vertical or a horizontal family of items. Instead, the items individually assess mathematical topics that ranged from geometry to measurement to number concepts to proportions to patterns, with few if any connections between the items. The only obvious connection between items was that students had to use the wingspan measurements from the second item in order to answer the sixth item in the block. The performance results for the six theme block items, shown in Table 2-1, cannot be interpreted in any connected way that has to do with the mathematics in the items. Moreover, the difficulty in making connections between performance on the items is exacerbated by differences in the number of score levels. With respect to the only two items that were connected, the results suggest that, although students could measure wingspan (40 percent correct on item 2), they had difficulty using these measurements as part of a complex problem (1 percent correct on item 6~. However, these results were not presented so such direct comparisons can be made between students' performance on those two items. That is, we do not know the percentage of students who, having obtained the correct wingspan measurement in item 2, provided correct answers accompanied by reasonable work in item 6. In recent NAEP mathematics assessments, then, there has been little imple- mentation of the notion of families of items, although the 1996 framework docu- ment makes a case for their inclusion. What might a family of items look like? What kind of information would likely be generated about students' understanding of a particular concept presented at varying levels of complexity in a mathematical content area or their understanding of a concept presented across mathematical content areas? The next section of this paper investigates these questions in light of a suggested family of items structured around number patterns, an informal algebra topic for grade 4. 1A discussion of the grade 4 released theme block (as well as the released theme blocks from the grade 8 and grade 12 NAEP mathematics assessment in 1996) appears in Kenney and Lindquist (in press).

OCR for page 5
2 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT TABLE 2-1 Performance on Items in the Butterflies Theme Block: Grade 4 Item Description % Scoring at Highest Levela 1. Draw four missing markings on pictures of two butterflies to make each butterfly symmetrical. 2. Obtain two correct measurements in centimeters for the wingspans of two butterfly models. 28 40 3. Determine the greatest number of butterflies that can be stored in a case 5 and the number of cases needed to hold 28 butterflies; show how your answer was obtained. 4. Determine the maximum number of butterfly models that can be made from 3 a given number of parts (wings, bodies, antennae); show or explain how your answer was obtained. 5. Given that two caterpillars eat five leaves per day, determine the number of leaves needed each day to feed 12 caterpillars; show how your answer was obtained. 6. Find the number of each type of butterfly needed to create a repeating pattern on a banner that is 130 centimeters long; show how your answer was obtained. (Note: measurements obtained in item 2 are needed in this problem.) 1 Note: All items in this block were either short or extended constructed-response items. aBecause of differences in the number of levels in scoring guides, the highest score level varied among the items from three to five levels. The highest score levels are as follows: "satisfactory" for items 1, 4, and 6 (based on four score levels); "extended" for items 2 and 3 (based on five score levels); and "complete" for item 5 (based on three score levels). A Sample Family of Items Based on Number Patterns at Grade 4 This section of the paper is divided into three parts. The first part contains a discussion of the importance of number patterns in the elementary mathematics curriculum and of the appropriateness of this topic as the basis for an item family in NAEP. The purpose of the next part is to justify the use of released NAEP mathematics items as the basis for the family of items and the limitations inherent in this method. The concluding part contains the sample item family, an explana- tion of its structure, and the kinds of information that might be obtained from looking at students' performance across the items.

OCR for page 5
PATRICIA ANN KENNEY Number Patterns in Elementary Mathematics and in NAEP 13 The topic of patterns and relationships, particularly number patterns in elementary school mathematics, is an appropriate and important content topic around which to create a family of items. Exploring patterns helps students in the early grades develop the ability to think algebraically (Armstrong, 1995; National Council of Teachers of Mathematics [NCTMi, 1989; Reys et al., 1995~. In fact, the NCTM Algebra Working Group realized that "children can develop algebraic concepts at an early age" (NCTM, 1994:5) and suggested that working with patterns of shapes and numbers helps to build the foundation for algebraic think- ing needed in later grades. In addition to information about the importance of patterns and relationships in the elementary mathematics curriculum, there exists a body of research that examines elementary students' performance on various types of numerical series and pattern items (e.g., Holzman et al., 1982, 1983; Pellegrino and Glaser,1982~. Included in these studies is important information on the nature of pattern items and possible factors that affect their ease or difficulty. For example, Holzman et al. (1982) report that the degree of difficulty of a numerical pattern item depends on such factors as the operations used to generate the pattern (i.e., incrementing operations [addition, multiplication] are easier than decrementing operations [sub- traction and divisional; the number of operations used to generate subsequent numbers in the pattern (i.e., patterns based on one operation [e.g., add 41 are easier than those based on two operations [e.g., first multiply by 2 and then add 11~; and the magnitude of the numbers used (i.e., patterns based on increments or decrements of small numbers [5 or less] are easier than those based on larger numbers [11 or greater]~. The importance of patterns and relationships in the elementary mathematics curriculum is further supported by the fact that the NAEP mathematics frame- work included this topic at grade 4. There is evidence that patterns, particularly numerical patterns, were a topic assessed in NAEP. Within the Algebra and Functions content strand at grade 4, recent NAEP mathematics assessments have included items that assess informal algebraic thinking through patterns and rela- tionships. In 1992 about 10 percent of the items on the grade 4 assessment dealt with informal algebra, and most of those items involved patterns of figures, symbols, or numbers (Kenney and Silver, 1997~. The 1996 assessment had about the same percentage of items at grade 4, and most of them also involved a variety of patterns, including number patterns. Thus, given the importance of patterns in both the elementary mathematics curriculum and recent NAEP mathematics assessments as well as a research base that speaks to characteristics of pattern items that can affect performance, selecting this as the topic for a family of items was both reasonable and appropriate.

OCR for page 5
4 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT Released NAEP Pattern Items as the Basis for an Item Family Not only did numerical pattern items appear on recent NAEP mathematics assessments, but also some of those items were released to the public. These released pattern items were not part of an item family in the assessment; instead, they appeared as single items in various parts of the assessment. However, because released pattern items and related performance data on those items were available from NAEP, it seemed reasonable to use these single items along with appropriate supplemental items to form a sample family of items. The advantage of this method of constructing a sample family of items is that the sample family uses items that have already appeared on a NAEP assessment, and we know how students performed on them. In addition, using existing items enabled the item family to be created with minimal time devoted to the development of original items. Finally, the existing items could be evaluated not only according to student performance but also using findings from published research studies (e.g., Holzman et al., 1982) and then organized into a family in a somewhat hierarchical fashion reflecting the level of cognitive demand. However, this method of creating a sample item family using existing NAEP items carries with it potentially serious limitations. In particular, taking items developed individually and putting them together as a set post hoc has a degree of artificiality. The ideal method that should be used to create an item family is to begin with a particular topic and information based on research about students' understanding of that topic, build the family of items, and then validate the structure of the set by administering it to students and examining performance results. Obviously, this method was not used in this paper. As a consequence, the family of items presented herein should be considered as an illustrative but very modest example of what such a family might look like, with the understand- ing that better families of items will be created for future NAEP assessments. It is hoped that the simple example in this paper will be used as the basis for further thought about and discussion of important features of families of items in NAEP. A Proposed Item Family Based on Numerical Patterns The six items in Appendix 2B constitute a proposed family of items built around the topic of numerical patterns. The set was developed according to the following guidelines: Each item in the set involves an increasing pattern of numbers based on a particular rule that governs the growth. In the elementary mathematics curricu- lum, these kinds of patterns are often referred to as "growing patterns" (Reys et al., 1995; NCTM, 1992~. In some items the pattern is based on constant increases between consecutive terms, and in others the pattern is based on nonconstant Increases.

OCR for page 5
PATRICIA ANN KENNEY 15 The set represents an attempt to organize the items from the easiest to the most difficult. In the case of released NAEP items, performance data were used to determine the level of difficulty (e.g., an item for which performance was 75 percent correct was "easier" than an item for which performance was 53 percent correct). For items created especially for the set, the degree of difficulty was based either on a rational analysis of mathematical concepts or on factors found to influence the difficulty of number pattern item, such as kind of operation, number of operations, and magnitude of numbers, as presented in published research articles (e.g., Holzman et al., 1982~. . In some cases the items are presented in two formats: multiple choice and constructed response. Given that the NAEP has always advocated a judicious blend of multiple-choice and constructed-response items, presenting an alterna- tive format for items (especially those developed specifically for the sample item family) seemed to be appropriate. However, because of the performance differ- ences in NAEP concerning lower percent-correct results on constructed-response items, this could affect the hierarchy of items (easiest to more difficult) in the sample set. The source is given for each item in Appendix 2B (e.g., a released NAEP item; an item created for the set). Following each item is a rationale for why the item was included in the item family and the kind of information that could be obtained from performance results. Figure 2-1 summarizes the concepts and progression of items in the sample family. Performance on these related items could provide insights into students' understanding of numerical patterns and where that understanding falters. For example, performance results could show that most fourth-grade students can work with patterns involving constant increases between the terms (items 1, 2, and 3), but performance levels could be lower for items involving patterns based on nonconstant increases (items 4 and 5), especially for complex problems (item 6~. Performance results could also provide information on misunderstandings that students have about number pat- terns, with the same misunderstandings possibly occurring across items in the family. For example, some students may expect a number pattern always to have a constant increase between contiguous numbers. In this case, when faced with a pattern containing nonconstant increases such as the number pattern in item 4 (i.e., 8, 9, 12, 17, 24, 33, 44, . . . [increase based on the set of odd numbers]), those students could reason that, because the increase between the last two numbers shown in the pattern is 11, 55 (44 + 11) is the next number in the pattern. Because the next two items in the family (items 5 and 6) also involve nonconstant increases, results from those items can provide additional evidence about this misunderstanding. Some might argue that such information about students' understanding and misunderstandings of numerical patterns is already available from the NAEP mathematics assessment results. All that would be needed would be to analyze

OCR for page 5
PATRICIA ANN KENNEY 33 Rationale for Item 2 A version of the 1992 NAEP item would appear next in the item family. Although the pattern of numbers is still constantly increasing and although the operation is still addition, the increase itself is a double-digit number greater than 11, which adds to the difficulty of the item for elementary school students (Holzman et al., 1982~. Despite the fact that the increase is a multiple of both 5 and 10 and that the increase is given in the problem, this item is considered as a "step up" from the first problem because of its constructed-response format and the need to work with a pattern involving a two-digit number increase. The NAEP version, however, should be modified so that more information can be obtained from student responses. In particular, the original NAEP item asked for three numbers in the pattern based on a given constant increase of 50 students. The results showed that just over half the fourth-grade students gave completely correct responses. However, the results did not reveal which of the three numbers was the most difficult to obtain. The version proposed for the item family could remedy this situation by providing information on whether the students understood that the enrollment increases in the first year by 50 students, and then by that same number in each of the next two years.

OCR for page 5
34 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT

OCR for page 5
PATRICIA ANN KENNEY 35

OCR for page 5
36 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT Rationale for Item 3 The third item in the family represents a transition from patterns based on addition of a constant to patterns based on multiplicative models. This item would reveal whether students understand that patterns can be based on arith- metic operations other than addition, for example, multiplication or division. The original NAEP item was discussed in an earlier section of this paper, and it was noted there that the results did not completely reveal the degree of consis- tency between the rule selected by students and whether they used that rule to answer the second question. Using one of the revised versions, both of which are constructed-response questions, perhaps we can better relate the students' de- scription of the rule in part 1 and their use (or misuse) of that rule in part 2. For example, in version 1 for students who answered "Divide the number in Column A by 4," but who wrote "3" in Column B in the second part of the problem, we could more accurately attribute this incorrect answer to a place-value error or perhaps to carelessness. For other students who wrote the correct rule, but who answered "480" in the second part, it is likely that their error involved multiply- ing instead of dividing. With respect to the two versions suggested for the family, one version might be preferable over the other depending on whether the multiplicative model (Ver- sion 2) or the division model (Version 1) is more easily recognized by students. Perhaps both versions could be pilot-tested to answer this question, with only one version included in the item family.

OCR for page 5
PATRICIA ANN KENNEY Rationale for Item 4 The fourth item, presented in two versions (multiple choice and constructed response), serves as a transition between numerical patterns based on constant increases to those based on nonconstant increases. In an important way, nonconstant increases are in themselves a pattern within a pattern. For example, the pattern in the item (8, 9, 12, 17, 24, 33, 44, . . . ~ also has a pattern of increases (1, 3, 5, 7, 9, . . . ~ the set of odd numbers. Because the notion of nonconstant increases is likely to be difficulty for some fourth-grade students, basing the nonconstant increases on the set of odd numbers could make the item more accessible. Also, the operation used to create the pattern is again simple addition. As noted earlier in the paper, this item and the ones that follow could provide evidence about an important misunderstanding about patterns; that is, the notion that all patterns (even those that are based on nonconstant differences) contain pairs of numbers that have a constant difference. For item 4 in the family, it is likely that some students could choose B (55) for the multiple-choice version or write 31 and 38 as the next two numbers in the pattern for the constructed- response version. In both cases, such responses show evidence of changing the nonconstant increase to a constant increase based on the difference between the last two numbers shown in the pattern.

OCR for page 5
38 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT

OCR for page 5
PATRICIA ANN KENNEY 39

OCR for page 5
40 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT Rationale for Item 5 This item within the family has the potential to be the most difficult question to this point. Results from the original NAEP version of the item showed that about the same percent of students selected choice C (25 pounds) as selected the correct choice D (24 pounds). This error pattern shows that some students may expect a number pattern to have a constant difference between some contiguous numbers: that is, in the puppy problem, students retained the 3-pound weight gain between the third and fourth months and used it as a constant to calculate the weight at 5 months (22 + 3 = 25~. Also, the high omitted and not-reached rates suggest that some fourth-grade students thought that this problem was so difficult that they did not even try to answer it. The version proposed for the item family attempts to make the question more accessible to students. It is scaffolded so that students must identify the first two nonconstant differences between the weights, in the hope that students will more easily recognize that the weight gains are decreasing between consecutive months. The final question involves a transition from the nonconstant differences to the actual weight of the puppy. As for Item 4 in the family, this item has the potential to provide additional evidence of the misunderstanding about nonconstant increases. Despite the at- tempt at scaffolding, students could still change to a constant increase and answer 25 pounds or some other number based on a constant increase in weight.

OCR for page 5
PATRICIA ANN KENNEY 4

OCR for page 5
42 FAMILIES OF ITEMS IN THE NAEP MATHEMATICS ASSESSMENT

OCR for page 5
PATRICIA ANN KENNEY 43 Rationale for Item 6 The original NAEP item, called Marcy's Dot Pattern in NAEP reports (e.g., Dossey et al., 1993), was administered to students in the 1992 eighth-grade sample as an extended constructed-response question in the Algebra and Func- tions content strand. As shown by the performance results, this question was difficult for the eighth-grade students: only 6 percent produced a response that was scored as satisfactory or extended. However, the fact that the item was last in an item block with previous items having little or no connection to number patterns could have affected performance levels. How would students have performed if this question, or an appropriate version thereof, appeared in a family of items devoted to number patterns? Given the structure of the family of items describe thus far, it seemed reason- able to think about including an adaptation of the Marcy's Dot Pattern as the culminating item in the family. As the culminating item, it has characteristics based on work done on the previous items. For example, one way to view the pattern in this task is that the pattern involves nonconstant increases between the number of dots in each step. Solving the problem requires students to identify the rule that underlies the pattern of nonconstant increases. The version for the item family begins with an introductory question about the number of dots in the fourth step as a way to introduce students to the problem. Here, it would be reasonable for students to draw the fourth figure so that they can better under- stand the pattern. The next part of the problem is similar to that given to students in the eighth-grade sample, but the steps are reduced from the 20th step to the 10th step. This last decision needs careful thought, however, because drawing 7 more sets of dots is more accessible than drawing 17 more sets.