National Academies Press: OpenBook
« Previous: References
Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

Appendixes

Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
This page in the original is blank.
Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

APPENDIX A

Enhancing the Assessment of Reading

In Chapter 4, we concluded that the NAEP reading framework reflects current theory and research about reading process, but that the assessment itself does not adequately reflect the goals of the framework. It does not adequately assess several important characteristics of good readers, nor does it reflect how students' reading is influenced by interactions among reader, text, and context. In this appendix we describe more specific conclusions and recommendations designed to improve the current main NAEP reading assessment, followed by examples intended to illustrate ways of implementing the recommendations.

ASSESSMENT TASKS, ITEMS, TEXTS, AND SCORING RUBRICS

The current assessment does not adequately reflect the reading document in terms of assessment tasks, items, texts, and scoring rubrics. The assessment tasks confound reading purpose with type of text. It is assumed that a particular type of text always engenders a particular purpose for reading, yet it is possible, and often desirable, to read the same text for different purposes (e.g., read an informational article to understand the specific cause/effect relationships that led to a war versus to gain a general impression of the situation that led to conflict as a strategy for understanding another historical event). Students are never explicitly given a purpose for reading, forcing them to adopt an unfocused or personally constructed purpose for reading that may or may not be aligned with the focus of the comprehension items. Ultimately, purpose and focus influence comprehension. In addition, in an effort to hold text types more or less constant across grades and test blocks, texts used at a particular grade level may not be representative

Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

of texts students read. For example, to assess reading for literary experience, NAEP uses tales across grades 4, 8, and 12 even though this genre is rare in students' reading at the high school level.

NAEP reading items are designed to align with four stances of comprehension, yet research suggests that the four types are not judged to be discrete. In general, the items are developed using a 3 × 4 grid (passage type by comprehension type) for each reading selection rather than designed to meet the demands of a particular text or a focused purpose for reading. Furthermore, extended response questions often are not put to good use. In some cases they require limited thinking or writing from students (which does not improve on multiple-choice items); in other cases, the items are good but the scoring rubrics often don't require evaluation of the quality or depth of the students' written response, but rather are vague or focus on superficial elements. This is most likely a result of NAEP using a generic rubric to guide development of passage-specific rubrics.

Students should be provided with explicit and varying purposes for reading specific texts. Items should follow up on those purposes rather than being forced to fit the 3 × 4 matrix. Text types, purposes, and item types should be systematically varied across the entire NAEP assessment. The nature of the cognitive processing of text follows from a particular text and purpose for reading. Not all types of questions or levels of processing are appropriate for every text/purpose combination.

Similarly, texts should be selected to reflect what students read at a particular grade level rather than selecting text types that are held consistent across grade levels. Item formats and scoring rubrics should be appropriate to the specific questions and the depth of understanding students should demonstrate.

ASSESSING THE CONCEPTUAL FRAMEWORK

The assessment leaves many of the characteristics of effective reading described in the document unexamined or unmanipulated. For example, background knowledge, fluency, reading strategies, purpose, habits, and attitudes are not adequately assessed. Although it is true that some of these characteristics have been examined in special studies (fluency) or background questions (habits), the results have not been used to inform the larger NAEP dataset or to inform NAEP reports.

In the same way, the assessment does not adequately reflect the complex interaction among the reader, text, and context portrayed in the reading document and influential in reading performance. That is to say, that variables such as background knowledge, reading strategies, habits, comprehension (reader), text complexity, topic, comprehension items (text), and purpose for reading and classroom instructional opportunities (context) are not systematically varied or considered in a way that reflects the framework's theoretical basis, nor do they inform reporting and interpretation of student performance.

Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

When possible, the characteristics of good readers described in the framework should be included in the large-scale portion of NAEP. When this is not feasible (because of time, money, requirements of reliable and valid assessment of these characteristics), alternative assessment materials should be used to gather data on student performance in these areas. When alternatives are used, the results should be integrated with results of the large-scale assessment, providing a more complete and more useful assessment of student performance.

To address an interactive model of reading, NAEP should develop coherent families of items to meet the demands of particular purposes for reading. Each family should include a reading selection and items that fit a particular purpose for reading the text as well as items that assess students' strategies, dispositions, and instructional experiences, as appropriate.

DATA ANALYSIS AND REPORTING

The process of data analysis treats each reading item as independent rather than analyzing the family of items around a particular reading selection as a coherent, interrelated group of items. In addition, results are reported as using a single scale, suggesting that student performance is uniform across various contexts (e.g., text types, purposes). As a result, we are not able to understand reading performance in terms of the interaction among reader, text, and context. For example, we cannot tell how well students read for different types of understanding or use various strategies when they are reading for different purposes or under different conditions. We cannot determine how these factors influence reading performance, nor can we gain insight into when students may have difficulties.

NAEP should explore alternative methods of data analysis that are based on the theoretical and empirical basis of the NAEP reading document. They should find ways to analyze and report student performance under varying conditions by treating item sets as the unit of analysis. For example, they may be able to build profiles of students' cognitive abilities as a function of various texts, tasks, and purposes for reading. At the same time, NAEP should explore ways to aggregate scores that fairly represent student performance for reporting to outside audiences.

AN EXAMPLE FROM THE 1994 NAEP READING ASSESSMENT-GRADE 8

Figure A-1 is a reading passage from the 1994 NAEP reading assessment for grade 8. In the discussion that follows, assessment material is in regular font; comments are italicized.

Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

FIGURE A-1 Reading passage from the 1994 NAEP reading assessment for grade 8. Reprinted by permission.

Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

No purpose is given for reading this informational piece. As a result, students don't have a reason to focus their reading. The following is recommended:

GLOBAL PURPOSE: This informational article is like one that you might read when doing a research report.

SPECIFIC PURPOSE: Read this informational article to understand the theories about what happened to one ancient civilization—the Anasazi.

or

You are going to write a short research report on ancient civilizations that disappeared. Read this article on the Anasazi to understand the theories scientists have for their disappearance. (This purpose would be followed-up with another short piece on the Mayan civilization and a short writing assignment comparing the two. Students could also be asked to take notes to aid in their report writing, providing data that could be analyzed as part of a target study of students' reading and study strategies.)

Both these purposes focus readers on theories and supporting evidence. They call for a close reading and reasoning about the text.

A logical line of questioning for these purposes would include:

Who were the Anasazi?

Where and when did they live?

Why is their disappearance of interest to scientists?

What are the theories and supporting evidence for their disappearance?

Existing Items

1. (Extended response) After reading this article, what do you think is the most important information about the Anasazi?

This question doesn't have a clear focus and, as a result, elicits vague responses from students. What students determine to be most important depends on their purpose for reading.

Scoring Rationale = Initial Understanding

Initial understanding requires students to provide an initial impression or unreflective understanding of what was read.

1 = Evidence of little or no comprehension—these responses contain inaccurate information from the article or inappropriate personal opinions about the

Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

article. They do not provide any valid information or appropriate interpretation about the Anasazi as they were portrayed in the article.

3 = Evidence of full comprehension—these responses provide a specific detail or a general impression from the passage that related to some aspect of the Anasazi portrayed in the article.

(The reading assessment development panel determined that it was more appropriate to score this item with a 2-point scoring guide.)

Examples of specific statements too vague for a score of 3:

how they stayed alive

interesting things they made

what they ate

the way they lived/farmed/built houses/grew food

they worked hard

they had a rough life

Examples of specific statements acceptable for a score of 3:

They left.

They moved.

The scoring rubric is vague. Any specific text-based response is given full credit. There are only 2 credit levels-full (3) or partial (1). Ironically, ''they left/they moved'' is given full credit but "the way they lived/farmed/built houses/grew food" are all given a score of 1. The nature of the full response does not seem to merit an extended constructed-response item.

2. (Extended response) The three moves made by the Anasazi are listed below. Explain the possible reasons that were suggested in the article for each move.

This question is intended to have students develop an interpretation. Although it requires students to process information across the text, it misleads them since there are no reasons given in the article for two of the moves, so answers such as "no reason given" as well as "so they could live in apartments" (which is a questionable, reader-based opinion) are given full credit. This question could be better focused to our purpose and to the interpretation of theories and evidence. The focus would be on theories for Anasazis leaving the mesa.

Scoring Rationale = Developing an Interpretation

Developing an interpretation requires students to go beyond an initial impression

Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

of a text by linking information across parts of the text or focusing on specific information.

1 = Evidence of unsatisfactory comprehension—these responses do not identify the reasons provided by the article for any of the moves that were made by the Anasazi. These responses may provide some fact about the Anasazi but do not relate them to any of the moves.

2 = Evidence of partial comprehension—these responses discuss reasons for only one or two of the moves described in the article. They receive credit for one explanation if they state that the article did not provide possible reasons for the first move.

3 = Evidence of essential comprehension—these responses identify a reason for three of the moves discussed in the article. The reasons may be brief or simple restatements of information in the article as long as they are logical and taken from the passage.

4 = Evidence of extensive comprehension—these responses identify a reason for all three moves, even though the reasons for all the moves are not explicitly discussed in the article. These responses go beyond simply restating the article to interpret some of the information provided in the article as it relates to the moves.

Unacceptable reasons for the first move:

for protection

for food (too vague)

to make better houses (not specific enough)

hard to live in a slanted house

Acceptable reasons for the first move:

article gives no reason

probably thought farming was better/easier

get more rainfall

closer to farming

to farm on top

ran out of room

so they would not have to climb up and down

hunt easier

to build houses of stone and mortar

so they could live in apartments

Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

Unacceptable reasons for the second move:

wanted to build a new place (too vague)

because water became scarce

top became too crowded

reasons are not clear

because Mese Verde life was hard

because they had more than 200 rooms

Acceptable reasons for the second move:

bad growing seasons

lack of rain

for protection

too hot on top

bad weather

woods became scarce

top became too crowded

reasons are not clear

because Mesa Verde life was hard

because they had more than 200 rooms

Unacceptable reasons for third move:

no reasons were given

the cliffs were falling apart

for new farm land

Acceptable reasons for third move:

land was not useful

for better food and water

everyone was dying

life was miserable

for better farming

living too close together

driven out by enemies

The rubric partially confounds the completion of all three sections with the quality of the responses. Rubrics should distinguish the quality of the thinking from the quantity of responses provided.

Suggested alternate item:

The article suggests several theories about why the Anasazis left the mesa. List the theories and provide evidence from the article to support each theory.

THEORY EVIDENCE

Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

3. (Extended response) If you had lived with the Anasazi at Mesa Verde, would you have preferred living on the top of the mesa or in the cliff houses built into the alcoves? Explain you preference by using information from the article.

This question directs students away from the essential information in the article. It demonstrates what happens when item writers try to develop items that fit each major category in the framework matrix. Personal response questions are probably not appropriate in the context of this particular reading passage.

4. (Extended response) If you could talk to the author of this article, what is one question you could ask her about the Anasazi that is not already answered in the article? Explain why you would want to know this information.

The intent of this question (seeking additional information) is reasonable for an informational "research" purpose but talking with the author is not. It would be more meaningful to ask:

What other information would you need for your report on the Anasazi? Where would you go to get it?

This not only gets at the information missing from the article, but also serves as a reading strategy item (metacognitive item) about sources of information.

This question is labeled personal response, although it might qualify equally well as critical stance. The rubric requires students to respond with an appropriate question and a clear explanation about how this additional information would be useful. Rationales such as "I would be interested" or "I want to know" are not acceptable even though students aren't given a clear purpose of reading.

5. (Multiple choice) Which idea from the text about the Anasazi do the photographs support?

  1. They were able to create many useful objects.

  2. Farming was probably their major source of food.

  3. Wood seems to have been their primary building material.

  4. Their life became much easier when they moved into the cliff dwellings.

There are only 2 photo sections in the reading passage—one that includes the passage title and four small photos of baskets, sandals, and pottery. These pictures are only minimally relevant to the main thrust of the text. This text is not about the culture of the Anasazi but about their unexplained disappearance. Again, this question is the result of trying to fit a particular type of question (using graphic aids) to a reading passage without regard for the passage's content or the appropriate purpose for reading it.

Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×

6. (Extended response) Imagine that you are living with the people of Mesa Verde during the 1200s when they left the mesa. Some of your friends and neighbors do not want to leave the area. Based on information in the article, what would you tell these people to convince them to leave?

This question overlaps with question 2—to get full credit, students must make an argument based on information from the text (same information needed in question 2). This item is labeled "critical stance" although it could just as easily be classified as "personal response." The two are obviously not distinct.

7. (Multiple choice) The Anasazi's life before 1200 A.D. was portrayed by the author as being

  1. dangerous and warlike

  2. busy and exciting

  3. difficult and dreary

  4. productive and peaceful

Rather than an emphasis on the date, this question should focus on the strange, unexplained change in Anasazi life—their disappearance. Before this time, their life was good, so it is difficult to explain their disappearance.

8. (Multiple choice) The title and photograph on the first page of the article are probably meant to make the disappearance of the Anasazi seem to be

  1. personal tragedy

  2. a terrible mistake

  3. an unsolved mystery

  4. an important political event

The best use of this photo is to help students anticipate the content of the piece and set purpose—what would they expect to read about.

9. (Extended response) Some people say that the Anasazi's success as a civilization may have actually caused their own decline. Using information in the article, explain why you agree or disagree with this statement.

This question does a good job of hitting the conceptual main idea and combining it with a personal response. It requires students to take a position and defend it with evidence. Although it is labeled as "personal response," it is more likely a cross between "critical stance" and "personal response."

Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 217
Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 218
Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 219
Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 220
Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 221
Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 222
Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 223
Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 224
Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 225
Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 226
Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 227
Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 228
Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 229
Suggested Citation:"Appendix A: Enhancing the Assessment of Reading." National Research Council. 1999. Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. Washington, DC: The National Academies Press. doi: 10.17226/6296.
×
Page 230
Next: Appendix B: Research About Student Learning as a Basis for Developing Assessment Materials: An Example from Science »
Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress Get This Book
×
Buy Hardback | $55.00 Buy Ebook | $43.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Since the late 1960s, the National Assessment of Educational Progress (NAEP)--the nation's report card--has been the only continuing measure of student achievement in key subject areas. Increasingly, educators and policymakers have expected NAEP to serve as a lever for education reform and many other purposes beyond its original role.

Grading the Nation's Report Card examines ways NAEP can be strengthened to provide more informative portrayals of student achievement and the school and system factors that influence it. The committee offers specific recommendations and strategies for improving NAEP's effectiveness and utility, including:

  • Linking achievement data to other education indicators.
  • Streamlining data collection and other aspects of its design.
  • Including students with disabilities and English-language learners.
  • Revamping the process by which achievement levels are set.

The book explores how to improve NAEP framework documents--which identify knowledge and skills to be assessed--with a clearer eye toward the inferences that will be drawn from the results.

What should the nation expect from NAEP? What should NAEP do to meet these expectations? This book provides a blueprint for a new paradigm, important to education policymakers, professors, and students, as well as school administrators and teachers, and education advocates.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!