In this appendix, an example of a family of items in grade 4 mathematics is presented. This information is drawn from a research paper in a volume that accompanies this report (Kenney, 1999).

The topic of patterns and relationships, and in particular numerical patterns in elementary school mathematics, is an appropriate content topic around which to create a family of items. Exploring patterns helps students in the early grades develop the ability to think algebraically (Armstrong, 1995; National Council of Teachers of Mathematics [NCTM], 1989; Reys et al., 1995). In fact, the NCTM Algebra Working Group realized that children can develop algebraic concepts at an early age and suggested that working with patterns of shapes and numbers helps to build the foundation for algebraic thinking needed in the later grades.

Not only did numerical pattern items appear on recent NAEP mathematics assessments, but also some of those items were released to the public. These released pattern items were not part of an item family in the assessment; instead, they appeared as single items in various parts of the assessment. However, because released pattern items and related performance data on those items were

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.

Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 237

GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress
APPENDIX C
A Sample Family of Items Based on Number Patterns at Grade 4
In this appendix, an example of a family of items in grade 4 mathematics is presented. This information is drawn from a research paper in a volume that accompanies this report (Kenney, 1999).
NUMERICAL PATTERNS IN ELEMENTARY MATHEMATICS AND IN NAEP
The topic of patterns and relationships, and in particular numerical patterns in elementary school mathematics, is an appropriate content topic around which to create a family of items. Exploring patterns helps students in the early grades develop the ability to think algebraically (Armstrong, 1995; National Council of Teachers of Mathematics [NCTM], 1989; Reys et al., 1995). In fact, the NCTM Algebra Working Group realized that children can develop algebraic concepts at an early age and suggested that working with patterns of shapes and numbers helps to build the foundation for algebraic thinking needed in the later grades.
RELEASED NAEP PATTERN ITEMS AS THE BASIS FOR AN ITEM FAMILY
Not only did numerical pattern items appear on recent NAEP mathematics assessments, but also some of those items were released to the public. These released pattern items were not part of an item family in the assessment; instead, they appeared as single items in various parts of the assessment. However, because released pattern items and related performance data on those items were

OCR for page 237

GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress
available from NAEP, it seemed reasonable to use these single items along with appropriate supplemental items to form a sample family of items.
The advantage of this method of constructing a sample family of items is that the sample family uses items that have already appeared on a NAEP assessment, and we know how students performed on them. Also, the family could be created with minimal time spent on developing original items. However, a potentially serious limitation of this method is that taking items developed individually and putting them together as a set post hoc carries with it a degree of artificiality. The ideal way to create an item family is to begin with a particular topic and information based on research about students' understanding of that topic and then build the family of items. Thus, the family of items presented here should be considered as an illustrative, but modest, example of what such a family might look like, with the understanding that better families of items should be created for future NAEP assessments. However, it is hoped that the example presented here will be used as the basis for further thought about and discussion of important features of families of items in NAEP.
NUMERICAL PATTERNS: AN ITEM FAMILY
The six items presented in this appendix constitute a proposed family of items built around the topic of numerical patterns. The set was developed according to these guidelines:
Each item within the set involves an increasing pattern of numbers based on a particular rule that governs the growth. In the elementary mathematics curriculum, these kinds of patterns are often referred to as ''growing patterns'' (e.g., Reys et al., 1995; National Council of Teachers of Mathematics, 1992). In some items, the pattern is based on constant differences between consecutive terms, and in others the pattern is based on nonconstant differences.
The set represents an attempt to organize the items from the easiest to the most difficult. In the case of released NAEP items, the performance data were used to determine the level of difficulty (e.g., an item for which performance was 75 percent correct was "easier" than an item for which performance was 53 percent correct). For items created especially for the set, the degree of difficulty was speculative and based solely on an educated guess.
In some cases, the items are presented in two formats: multiple choice and constructed response. Given that NAEP has always advocated a judicious blend of multiple-choice and constructed-response items, presenting an alternative format for items (especially those developed specifically for this paper) seemed to be appropriate. However, because of the performance differences in NAEP concerning lower percent-correct results on constructed-response items, this could affect the hierarchy of items (easiest to more difficult) within the sample set.

OCR for page 237

GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress
The source is provided for each item (e.g., a released NAEP item, an item created for the set). Following each item is a rationale about why the item was included in the item family and about the kind of information that could be obtained from performance results.
Figure C-1 summarizes the concepts and progression of the items within the sample family. Performance on these related items could provide insights into students' understanding about numerical patterns and where that understanding falters. For example, performance results could show that most fourth graders can work with patterns involving constant increases between the terms (Items 1, 2, and 3), but performance levels could be lower for items involving patterns based on nonconstant increases (Items 4 and 5) and especially for complex problems (Item 6). Performance results could also provide information on misunderstandings that students have about number patterns, with the same misunderstandings possibly occurring across items within the family. For example, some students may expect a number pattern always to have a constant difference between contiguous numbers. In this case, when faced with a pattern containing nonconstant differences, such as the number pattern in Item 4 (8, 9, 12, 17, 24, 33, 44, …), those students could reason that, because the difference between the last two numbers shown in the pattern is 11, then 55 (44 + 11) is the next number in the pattern. Because the next two items in the family also involve nonconstant differences, results from these items can provide additional evidence about this misunderstanding.
Some might argue that such information about students' understanding and misunderstandings of numerical patterns is already available from the NAEP mathematics assessment results. All one would need to do would be to analyze the performance results from the pattern items included on the assessment. There is a some truth to this argument, but the fact remains that most NAEP items (other than the item pairs or triples or the theme block items) are discrete; that is, each item is essentially unrelated to any other item in the assessment. Therefore, identifying the numerical pattern items in NAEP and then analyzing the performance data as if those items had been developed as an intact set is likely to result in information about students' understanding that is fragmented and difficult to interpret. The advantage of an item family is that the items were purposely developed to be related in ways that could illuminate students' understandings and misunderstanding of important mathematical concepts. Analyzing the performance data from a related set of items, then, is more likely to provide results that are connected and interpretable.
The recommendation in the 1996 NAEP mathematics framework to include families of items represents a positive direction for future NAEP mathematics assessments to take. The inclusion of families of items can increase NAEP's potential to provide important information about the depth of students' knowledge within a particular content strand and across content strands. The example presented here presents one fairly limited way in which items can be related to

OCR for page 237

GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress
FIGURE C-1 Progression of concepts within the number pattern family of items. SOURCE: Kenney (1999).

OCR for page 237

GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress
make a family and how the results can be analyzed to provide a more complete picture about students' understanding. The best way to develop families of items is de novo—that is, after determining in advance the desired concepts and levels of student understanding to be assessed. However, as illustrated in the example, it is possible to use existing NAEP materials as the foundation for building families. We recommend that future assessment developers build item families that better reflect the intentions for families of items described in the 1996 NAEP mathematics framework document.

OCR for page 237

GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress
AN EXAMPLE FAMILY OF ITEMS
ITEM 1
Version 1: Multiple choice
8, 14, 20, 26, 32, . . . .
If the pattern shown continues, which of the following numbers would be next in the pattern?
34
36
38
40
Version 2: Constructed response
write the next two numbers in the number pattern.
8 14 20 26 32 ____ ____
Version 3: Multiple-choice set within a context
Emily started her stamp collection with 8 stamps and added the same number of stamps to her collection each week. If she had 14 stamps after the first week, 20 stamps after the second week, and 26 stamps after the third week, how many stamps would she have after the fourth week?
28
32
38
40
Based on an example from Kenney and Silver (1997: 270).

OCR for page 237

GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress
Rationale for Item 1
This item would appear first in the family to determine the "floor" effect—that is, nearly all the fourth-grade students should be able to produce a correct answer based on the constant difference of 6 between the numbers in the pattern. The first version (multiple choice, no context) would best serve this purpose. The other versions are presented here as additional examples of simple pattern items based on single-digit, constant differences between consecutive numbers. The last version set within a context could possibly be too difficult to appear as the first item in the set, but its multiple-choice format could make it more accessible to fourth-grade students.

OCR for page 237

GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress
ITEM 2
Original NAEP item
In 1990 a school had 125 students. Each year the number of students in the school increases by 50. Fill in the table to show the number of students expected for each year.
Year
Number of Students
1990
125
1991
____
1992
____
1993
____
Source: 1992 NAEP mathematics assessment [calculator use permitted]
Performance results:
All three answers correct:
51 percent
Any two answers correct:
3 percent
Any one answer correct:
9 percent
At least one answer correct:
63 percent
Version for the item family
In 1990 a school had 125 students. Each year the number of students in the school increase by 50. Answer the questions based on the table.
Year
Number of Students
1990
125
1991
____
1992
____
1993
____
1. How many students will be school have in 1991?
Answer: ______________
2. Complete the table to show the number of students expected for 1992 and 1993.

OCR for page 237

GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress
Rationale for Item 2
A version of the 1992 NAEP item would appear next in the item family because, although the pattern of numbers is still constantly increasing, the increase itself is a double-digit number. Despite the fact that the increase is a multiple of both 5 and 10 and that the increase is given in the problem, this item is considered as a step up from the first problem because of its constructed-response format and the need to work with a pattern involving a two-digit number increase.
The NAEP version, however, should be modified so that more information can be obtained from student responses. In particular, the original NAEP item asked for three numbers in the pattern based on a given constant increase of 50 students. The results showed that just over half the fourth-grade students gave completely correct responses. However, the results did not reveal which of the three numbers was the most difficult to obtain. The version proposed for the item family could remedy this situation by providing information on whether the students understood that the enrollment increases in the first year by 50 students, and then by that same number in each of the next two years.

OCR for page 237

GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress
ITEM 3
Original NAEP item pair
Items 1 and 2 refer to the table below:
Column A
Column B
12—>
3
16—>
4
24—>
6
40—>
10
1. What is a rule used in the table to get the numbers in column B from the numbers in column A?
Divide the number in column A by 4.
Multiply the number in column A by 4.
Subtract 9 from the number in column A.
Add 9 to the number in column A.
Column A
Column B
120—>
2. Suppose 120 is a number in column A of the table. Use the same rule to fill in the number in column B.
Source: 1992 NAEP mathematics assessment [calculator use permitted]
Performance results:
Item 1: 42 percent selected correct choice (A)
Item 2: 24 percent obtained correct answer of 30

OCR for page 237

GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress
Version 1 for the item family: Division
The next questions use the following table:
Column A
Column B
12—>
3
16—>
4
24—>
6
40—>
10
1. Write the rule used to get the numbers in column B from the numbers in column A.
Rule:_______________________________________________________________
Column A
Column B
120—>
Suppose 120 is a number in column A of the table. Use the rule you wrote to fill in the number in column B.
Version 2 for the item family: Multiplication
The next questions use the following table:
Column A
Column B
12—>
3
16—>
4
24—>
6
40—>
10
Write the rule used to get the numbers in column A from the numbers in column B.
Rule: ___________________________________________________________
Column A
Column B
30
Suppose 30 is a number in column B of the table. Use the rule you wrote fo fill in the number in column A.

OCR for page 237

GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress
Rationale for Item 3
The third item in the family represents a transition from patterns based on addition of a constant to patterns based on multiplicative models. This item would reveal whether students understand that patterns of differences could be based on arithmetic operations other than addition. The original NAEP item was discussed in an earlier section of this paper, and it had some flaws from the lack of analysis of performance results on both parts of the item pair together. In particular, the results did not completely reveal the degree of consistency between the rule selected by students and whether they used that rule to answer the second question.
Using one of the revised versions, both of which are constructed-response questions, perhaps we can better relate the students' description of the rule in part 1 and their use (or misuse) of that rule in part 2. For example, in version 1 for students who answered ''Divide the number in Column A by 4,'' but who wrote "3" in Column B in the second part of the problem, we could more accurately attribute this incorrect answer to a place-value error or perhaps to carelessness. For other students who wrote the correct rule, but who answered "480" in the second part, it is likely that their error involved multiplying instead of dividing.
With respect to the two versions suggested for the family, one version might be preferable over the other depending on whether the multiplicative model or the division model is more easily recognized by students. Both versions could be pilot-tested to answer this question, but only one version would be included in the family.

OCR for page 237

GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress
ITEM 4
Version 1: Multiple choice
8, 9, 12, 17, 24, 33, 44, ...
If the pattern shown continues, which of the following numbers would be next in the pattern?
53
55
57
59
Version 2: Constructed response
Write the next two numbers in the number pattern.
8 9 12 17 24 ____ ____
Source: Created as an example for this report.
Rationale for Item 4
The fourth item, presented in two versions (multiple choice and constructed response), serves as a transition between numerical patterns based on constant increases to those based on nonconstant increases. In an important way, nonconstant increases are in themselves a pattern within a pattern. For example, the pattern in the item (8, 9, 12, 17, 24, 33, 44, …) also has a pattern of increases (1, 3, 5, 7, 9, …)—the set of odd numbers. Because the notion of nonconstant increases is likely to be difficulty for some fourth-grade students, basing the nonconstant increases on the set of odd numbers could make the item more accessible. Also, the operation used to create the pattern is again simple addition.
As noted earlier in the paper, this item and the ones that follow could provide evidence about an important misunderstanding about patterns; that is, the notion that all patterns (even those that are based on nonconstant differences) contain pairs of numbers that have a constant difference. For item 4 in the family, it is likely that some students could choose B (55) for the multiple-choice version or write 31 and 38 as the next two numbers in the pattern for the constructed-response version. In both cases, such responses show evidence of changing the nonconstant increase to a constant increase based on the difference between the last two numbers shown in the pattern.

OCR for page 237

GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress
ITEM 5
Original NAEP item
Puppy's Age
Puppy's Weight
1 month
10 lbs.
2 months
15 lbs.
3 months
19 lbs.
4 months
22 lbs.
5 months
?
John records the weight of his puppy every month in a chart like the one shown above. If the pattern of the puppy's weight gain continues, how many pounds will the puppy weight at 5 months?
A. 30
B. 27
C. 25
D. 24
Source: 1992 NAEP mathematics assessment
Performance results:
Choice A 12
percent
Choice B 24
percent
Choice C 29
percent
Choice D* 32
percent
*correct response
Note: Four percent of the students did not answer this item, and it had a 20 percent "not reached" rate (i.e., 20 percent of the students in sample left this item and all items that followed it blank).

OCR for page 237

GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress
Version for the item family
Puppy's Age
Puppy's Weight
1 month
10 lbs.
2 months
15 lbs.
3 months
19 lbs.
4 months
22 lbs.
5 months
?
John records the weight of his puppy every month in a chart like the one shown above. suppose the pattern of the puppy's weight gain continues.
1. How many pounds did the puppy gain from 1 month to 2 months?
Answer: ______________
2. How many pounds did the puppy gin from 2 months to 3 months
Answer: ______________
If the pattern of the puppy's weight gain continues, how many pounds will the puppy weigh at 5 months?
Answer: ______________

OCR for page 237

GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress
Rationale for Item 5
This item within the family has the potential to be the most difficult question to this point. Results from the original NAEP version of the item showed that about the same percent of students selected choice C (25 pounds) as selected the correct choice D (24 pounds). This error pattern shows that some students may expect a number pattern to have a constant difference between some contiguous numbers: that is, in the puppy problem, students retained the 3-pound weight gain between the third and fourth months and used it as a constant to calculate the weight at 5 months (22 + 3 = 25). Also, the high omitted and not-reached rate suggests that some fourth-grade students thought that this problem was so difficult that they did not even try to answer it.
The version proposed for the item family attempts to make the question more accessible to students. It is scaffolded so that students must identify the first two nonconstant differences between the weights, in the hope that students will more easily recognize that the weight gains are decreasing between consecutive months. The final question involves a transition from the nonconstant differences to the actual weight of the puppy.
As for Item 4 in the family, this item has the potential to provide additional evidence of the misunderstanding about nonconstant increases. Despite the attempt at scaffolding, students could still change to a constant increase and answer 25 pounds or some other number based on a constant increase in weight.

OCR for page 237

GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress
ITEM 6
Original NAEP item
A pattern of dots is shown below. At each step, more dots are added to the pattern. The number of dots added at each step is more than the number added in the previous step. The pattern continues infinitely.
(1st step)
(2nd step)
(3rd step)
• • • •
• • •
• • • •
• • •
• • •
• • • •
2 dots
6 dots
12 dots
Marcy has to determine the number of dots in 20th step, but she does not want to draw all 20 pictures and then count the dots.
Explain or show how she could do this and give the answer that Marcy should get for the number of dots.
Source: 1992 NAEP mathematics assessment—grade 8 [calculator use permitted]
Performance results:
Extended response
5 percent
Satisfactory response
1 percent
Partial response
6 percent
Minimal response
10 percent
Incorrect response
63 percent
Note: sixteen percent of the eighth-grade students did not answer this question.

OCR for page 237

GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress
Version for the item family
A pattern of dots is shown below. At each step, more dots are added to the pattern. The number of dots added at each step is more than the number added in the previous step. The pattern continues and does not stop.
(1st step)
(2nd step)
(3rd step)
• • • •
• • •
• • • •
• • •
• • •
• • • •
2 dots
6 dots
12 dots
How many dots would be in the 4th step? show how you got your answer.
Marcy has to determine the number of dots in the 10th step, but she does not want to draw all 10 pictures and then count the dots.
Explain or show how she could do this and give the answer that Marcy should get for the number of dots in the 10th step.
Rationale for Item 6
The original NAEP item, called Marcy's Dot Pattern in NAEP reports (e.g., Dossey et al., 1993), was administered to students in the 1992 eighth grade sample as an extended constructed-response question in the algebra and functions content strand. As shown by the performance results, this question was difficult for the eighth-grade students: only 6 percent produced a response that was scored as satisfactory or extended. However, the fact that the item was last in an item block with previous items having little or no connection to number patterns could have affected performance levels. How would students have performed if this question, or an appropriate version thereof, appeared in a family of items devoted to number patterns?
Given the structure of the family of items describe thus far, it seemed reasonable to think about including an adaptation of the Marcy's Dot Pattern as the culminating item in the family. As the culminating item, it has characteristics based on work done on the previous items. For example, Marcy's Dot Pattern involves a pattern of nonconstant differences between the number of dots in each step and requires students to identify the rule that underlies the pattern. The

OCR for page 237

GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress
version for the item family begins with an introductory question about the number of dots in the fourth step as a way to introduce students to the problem. Here, it would be reasonable for students to draw the fourth figure so that they can better understand the pattern. The next part of the problem is similar to that given to students in the eighth grade sample, but the steps are reduced from the 20th step to the 10th step. This last decision needs careful thought, however, because drawing 7 more sets of dots is more accessible than drawing 17 more sets. Pilot-testing could reveal the differences between working with the 10th step or a step further out in the pattern of dots.
REFERENCES
Armstrong, B. E. 1995 Teaching patterns, relationships, and multiplication as worthwhile mathematical tasks. Teaching Children Mathematics 1: 446-450.
Dossey, John A., Ina V. Mullis, and C.O. Jones 1993 Can Students Do Mathematical Problem Solving?: Results from Constructed-response Questions in NAEP's 1992 Mathematics Assessment . Washington, DC: National Center for Education Statistics.
Kenney, P.A. 1999 Families of items in the NAEP mathematics assessment, In Grading the Nation's Report Card: Research from the Evaluation of NAEP, James W. Pellegrino, Lee R. Jones, and Karen J. Mitchell, eds. Committee on the Evaluation of National and State Assessments of Educational Progress, Board on Testing and Assessment, Washington, DC: National Academy Press.
Kenney, Patricia A., and Edward A. Silver 1997 Probing the foundations of algebra: Grade 4 pattern items in NAEP . Teaching Children Mathematics 3(6):268-274.
National Council of Teachers of Mathematics 1992 Curriculum and Evaluation Standards for School Mathematics Addenda Series, Grades K-6: Fourth-Grade Book. Reston, VA: National Council of Teachers of Mathematics.
1989 Curriculum and Evaluation Standards for School Mathematics. Reston, VA: National Council of Teachers of Mathematics.
Reys, R. E., M.N. Suydam, and M.M. Lindquist 1995 Helping Children Learn Mathematics, Fourth edition. Boston: Allyn and Bacon.