Enhancing the Assessment of Reading
In Chapter 4, we concluded that the NAEP reading framework reflects current theory and research about reading process, but that the assessment itself does not adequately reflect the goals of the framework. It does not adequately assess several important characteristics of good readers, nor does it reflect how students' reading is influenced by interactions among reader, text, and context. In this appendix we describe more specific conclusions and recommendations designed to improve the current main NAEP reading assessment, followed by examples intended to illustrate ways of implementing the recommendations.
ASSESSMENT TASKS, ITEMS, TEXTS, AND SCORING RUBRICS
The current assessment does not adequately reflect the reading document in terms of assessment tasks, items, texts, and scoring rubrics. The assessment tasks confound reading purpose with type of text. It is assumed that a particular type of text always engenders a particular purpose for reading, yet it is possible, and often desirable, to read the same text for different purposes (e.g., read an informational article to understand the specific cause/effect relationships that led to a war versus to gain a general impression of the situation that led to conflict as a strategy for understanding another historical event). Students are never explicitly given a purpose for reading, forcing them to adopt an unfocused or personally constructed purpose for reading that may or may not be aligned with the focus of the comprehension items. Ultimately, purpose and focus influence comprehension. In addition, in an effort to hold text types more or less constant across grades and test blocks, texts used at a particular grade level may not be representative
of texts students read. For example, to assess reading for literary experience, NAEP uses tales across grades 4, 8, and 12 even though this genre is rare in students' reading at the high school level.
NAEP reading items are designed to align with four stances of comprehension, yet research suggests that the four types are not judged to be discrete. In general, the items are developed using a 3 × 4 grid (passage type by comprehension type) for each reading selection rather than designed to meet the demands of a particular text or a focused purpose for reading. Furthermore, extended response questions often are not put to good use. In some cases they require limited thinking or writing from students (which does not improve on multiple-choice items); in other cases, the items are good but the scoring rubrics often don't require evaluation of the quality or depth of the students' written response, but rather are vague or focus on superficial elements. This is most likely a result of NAEP using a generic rubric to guide development of passage-specific rubrics.
Students should be provided with explicit and varying purposes for reading specific texts. Items should follow up on those purposes rather than being forced to fit the 3 × 4 matrix. Text types, purposes, and item types should be systematically varied across the entire NAEP assessment. The nature of the cognitive processing of text follows from a particular text and purpose for reading. Not all types of questions or levels of processing are appropriate for every text/purpose combination.
Similarly, texts should be selected to reflect what students read at a particular grade level rather than selecting text types that are held consistent across grade levels. Item formats and scoring rubrics should be appropriate to the specific questions and the depth of understanding students should demonstrate.
ASSESSING THE CONCEPTUAL FRAMEWORK
The assessment leaves many of the characteristics of effective reading described in the document unexamined or unmanipulated. For example, background knowledge, fluency, reading strategies, purpose, habits, and attitudes are not adequately assessed. Although it is true that some of these characteristics have been examined in special studies (fluency) or background questions (habits), the results have not been used to inform the larger NAEP dataset or to inform NAEP reports.
In the same way, the assessment does not adequately reflect the complex interaction among the reader, text, and context portrayed in the reading document and influential in reading performance. That is to say, that variables such as background knowledge, reading strategies, habits, comprehension (reader), text complexity, topic, comprehension items (text), and purpose for reading and classroom instructional opportunities (context) are not systematically varied or considered in a way that reflects the framework's theoretical basis, nor do they inform reporting and interpretation of student performance.
When possible, the characteristics of good readers described in the framework should be included in the large-scale portion of NAEP. When this is not feasible (because of time, money, requirements of reliable and valid assessment of these characteristics), alternative assessment materials should be used to gather data on student performance in these areas. When alternatives are used, the results should be integrated with results of the large-scale assessment, providing a more complete and more useful assessment of student performance.
To address an interactive model of reading, NAEP should develop coherent families of items to meet the demands of particular purposes for reading. Each family should include a reading selection and items that fit a particular purpose for reading the text as well as items that assess students' strategies, dispositions, and instructional experiences, as appropriate.
DATA ANALYSIS AND REPORTING
The process of data analysis treats each reading item as independent rather than analyzing the family of items around a particular reading selection as a coherent, interrelated group of items. In addition, results are reported as using a single scale, suggesting that student performance is uniform across various contexts (e.g., text types, purposes). As a result, we are not able to understand reading performance in terms of the interaction among reader, text, and context. For example, we cannot tell how well students read for different types of understanding or use various strategies when they are reading for different purposes or under different conditions. We cannot determine how these factors influence reading performance, nor can we gain insight into when students may have difficulties.
NAEP should explore alternative methods of data analysis that are based on the theoretical and empirical basis of the NAEP reading document. They should find ways to analyze and report student performance under varying conditions by treating item sets as the unit of analysis. For example, they may be able to build profiles of students' cognitive abilities as a function of various texts, tasks, and purposes for reading. At the same time, NAEP should explore ways to aggregate scores that fairly represent student performance for reporting to outside audiences.
AN EXAMPLE FROM THE 1994 NAEP READING ASSESSMENT-GRADE 8
Figure A-1 is a reading passage from the 1994 NAEP reading assessment for grade 8. In the discussion that follows, assessment material is in regular font; comments are italicized.
No purpose is given for reading this informational piece. As a result, students don't have a reason to focus their reading. The following is recommended:
GLOBAL PURPOSE: This informational article is like one that you might read when doing a research report.
SPECIFIC PURPOSE: Read this informational article to understand the theories about what happened to one ancient civilization—the Anasazi.
You are going to write a short research report on ancient civilizations that disappeared. Read this article on the Anasazi to understand the theories scientists have for their disappearance. (This purpose would be followed-up with another short piece on the Mayan civilization and a short writing assignment comparing the two. Students could also be asked to take notes to aid in their report writing, providing data that could be analyzed as part of a target study of students' reading and study strategies.)
Both these purposes focus readers on theories and supporting evidence. They call for a close reading and reasoning about the text.
A logical line of questioning for these purposes would include:
Who were the Anasazi?
Where and when did they live?
Why is their disappearance of interest to scientists?
What are the theories and supporting evidence for their disappearance?
1. (Extended response) After reading this article, what do you think is the most important information about the Anasazi?
This question doesn't have a clear focus and, as a result, elicits vague responses from students. What students determine to be most important depends on their purpose for reading.
Scoring Rationale = Initial Understanding
Initial understanding requires students to provide an initial impression or unreflective understanding of what was read.
1 = Evidence of little or no comprehension—these responses contain inaccurate information from the article or inappropriate personal opinions about the
article. They do not provide any valid information or appropriate interpretation about the Anasazi as they were portrayed in the article.
3 = Evidence of full comprehension—these responses provide a specific detail or a general impression from the passage that related to some aspect of the Anasazi portrayed in the article.
(The reading assessment development panel determined that it was more appropriate to score this item with a 2-point scoring guide.)
Examples of specific statements too vague for a score of 3:
how they stayed alive
interesting things they made
what they ate
the way they lived/farmed/built houses/grew food
they worked hard
they had a rough life
Examples of specific statements acceptable for a score of 3:
The scoring rubric is vague. Any specific text-based response is given full credit. There are only 2 credit levels-full (3) or partial (1). Ironically, ''they left/they moved'' is given full credit but "the way they lived/farmed/built houses/grew food" are all given a score of 1. The nature of the full response does not seem to merit an extended constructed-response item.
2. (Extended response) The three moves made by the Anasazi are listed below. Explain the possible reasons that were suggested in the article for each move.
This question is intended to have students develop an interpretation. Although it requires students to process information across the text, it misleads them since there are no reasons given in the article for two of the moves, so answers such as "no reason given" as well as "so they could live in apartments" (which is a questionable, reader-based opinion) are given full credit. This question could be better focused to our purpose and to the interpretation of theories and evidence. The focus would be on theories for Anasazis leaving the mesa.
Scoring Rationale = Developing an Interpretation
Developing an interpretation requires students to go beyond an initial impression
of a text by linking information across parts of the text or focusing on specific information.
1 = Evidence of unsatisfactory comprehension—these responses do not identify the reasons provided by the article for any of the moves that were made by the Anasazi. These responses may provide some fact about the Anasazi but do not relate them to any of the moves.
2 = Evidence of partial comprehension—these responses discuss reasons for only one or two of the moves described in the article. They receive credit for one explanation if they state that the article did not provide possible reasons for the first move.
3 = Evidence of essential comprehension—these responses identify a reason for three of the moves discussed in the article. The reasons may be brief or simple restatements of information in the article as long as they are logical and taken from the passage.
4 = Evidence of extensive comprehension—these responses identify a reason for all three moves, even though the reasons for all the moves are not explicitly discussed in the article. These responses go beyond simply restating the article to interpret some of the information provided in the article as it relates to the moves.
Unacceptable reasons for the first move:
for food (too vague)
to make better houses (not specific enough)
hard to live in a slanted house
Acceptable reasons for the first move:
article gives no reason
probably thought farming was better/easier
get more rainfall
closer to farming
to farm on top
ran out of room
so they would not have to climb up and down
to build houses of stone and mortar
so they could live in apartments
Unacceptable reasons for the second move:
wanted to build a new place (too vague)
because water became scarce
top became too crowded
reasons are not clear
because Mese Verde life was hard
because they had more than 200 rooms
Acceptable reasons for the second move:
bad growing seasons
lack of rain
too hot on top
woods became scarce
top became too crowded
reasons are not clear
because Mesa Verde life was hard
because they had more than 200 rooms
Unacceptable reasons for third move:
no reasons were given
the cliffs were falling apart
for new farm land
Acceptable reasons for third move:
land was not useful
for better food and water
everyone was dying
life was miserable
for better farming
living too close together
driven out by enemies
The rubric partially confounds the completion of all three sections with the quality of the responses. Rubrics should distinguish the quality of the thinking from the quantity of responses provided.
Suggested alternate item:
The article suggests several theories about why the Anasazis left the mesa. List the theories and provide evidence from the article to support each theory.
3. (Extended response) If you had lived with the Anasazi at Mesa Verde, would you have preferred living on the top of the mesa or in the cliff houses built into the alcoves? Explain you preference by using information from the article.
This question directs students away from the essential information in the article. It demonstrates what happens when item writers try to develop items that fit each major category in the framework matrix. Personal response questions are probably not appropriate in the context of this particular reading passage.
4. (Extended response) If you could talk to the author of this article, what is one question you could ask her about the Anasazi that is not already answered in the article? Explain why you would want to know this information.
The intent of this question (seeking additional information) is reasonable for an informational "research" purpose but talking with the author is not. It would be more meaningful to ask:
What other information would you need for your report on the Anasazi? Where would you go to get it?
This not only gets at the information missing from the article, but also serves as a reading strategy item (metacognitive item) about sources of information.
This question is labeled personal response, although it might qualify equally well as critical stance. The rubric requires students to respond with an appropriate question and a clear explanation about how this additional information would be useful. Rationales such as "I would be interested" or "I want to know" are not acceptable even though students aren't given a clear purpose of reading.
5. (Multiple choice) Which idea from the text about the Anasazi do the photographs support?
They were able to create many useful objects.
Farming was probably their major source of food.
Wood seems to have been their primary building material.
Their life became much easier when they moved into the cliff dwellings.
There are only 2 photo sections in the reading passage—one that includes the passage title and four small photos of baskets, sandals, and pottery. These pictures are only minimally relevant to the main thrust of the text. This text is not about the culture of the Anasazi but about their unexplained disappearance. Again, this question is the result of trying to fit a particular type of question (using graphic aids) to a reading passage without regard for the passage's content or the appropriate purpose for reading it.
6. (Extended response) Imagine that you are living with the people of Mesa Verde during the 1200s when they left the mesa. Some of your friends and neighbors do not want to leave the area. Based on information in the article, what would you tell these people to convince them to leave?
This question overlaps with question 2—to get full credit, students must make an argument based on information from the text (same information needed in question 2). This item is labeled "critical stance" although it could just as easily be classified as "personal response." The two are obviously not distinct.
7. (Multiple choice) The Anasazi's life before 1200 A.D. was portrayed by the author as being
dangerous and warlike
busy and exciting
difficult and dreary
productive and peaceful
Rather than an emphasis on the date, this question should focus on the strange, unexplained change in Anasazi life—their disappearance. Before this time, their life was good, so it is difficult to explain their disappearance.
8. (Multiple choice) The title and photograph on the first page of the article are probably meant to make the disappearance of the Anasazi seem to be
a terrible mistake
an unsolved mystery
an important political event
The best use of this photo is to help students anticipate the content of the piece and set purpose—what would they expect to read about.
9. (Extended response) Some people say that the Anasazi's success as a civilization may have actually caused their own decline. Using information in the article, explain why you agree or disagree with this statement.
This question does a good job of hitting the conceptual main idea and combining it with a personal response. It requires students to take a position and defend it with evidence. Although it is labeled as "personal response," it is more likely a cross between "critical stance" and "personal response."