|
Items in cart [0] |
Questions? Call 888-624-8373 |
| ||||||||||||
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 71
APPENDIX A
BACKGROUND PAPERS
This appendix contains three papers prepared specifically for the CASH
project. The first two--nCognitive Science and Survey Methods by Roger
Tourangeau and Potential Contributions of. Cognitive Sciences to Survey
Questionnaire Designs by Norman M. Bradburn and Catalina Danin--were Rent
to participants in advance of the St. Michael's seminar. The third
paper, "Record Checks for Sample Surveyed by Kent Marquis, is based on a
presentation by the author at St. Michaels.
The first two papers focus on the basic questions addressed by the
CASH project: What are the potential links between the cognitive
sciences and survey research and how can each discipline contribute to
progress in the other? However, the authors examine the questions from
different perspectives.
Tourangeau identifies four kinds of cognitive operations performed by
respondents in survey interviews: understanding questions, retrieving
relevant information from memory, making judgments (if called for), and
selecting responses. He reviews research in the cognitive sciences on
each of the four topics and discusses the possible relevance of research
findings to survey interviews. In discussing Judgment and response, he
Lives special attention to the implications of rinds ngs from the
cognitive sciences for attitude questions in surveys. Tourangeau also
discusses the possible uses of surveys as a Laboratory for cognitive
research and explains how using surveys in this way might help to
overcome some of the limitations of small-scale laboratory experiments.
He describes one instance in which laboratory-based generalizations have
been tested in surveys and briefly summarizes the results.
Bradburn and Danis approach the came general subject from the point
of view of the survey researcher. The authors first present a conceptual
model of response effects, i.e., sources of variation in the quality of
data obtained in surveys, and a general model for human information
processing. They then contrast the research methods used in the
cognitive sciences and in survey research methodological studies. The
main part of their paper reviews potential contributions of findings from
cognitive research to four of the issues most frequently studied by
_ .
. . .
71
OCR for page 72
72
survey researchers: question wording, response categories, contextual
meaning' and response to survey items that require information on time or
frequency of specific events or transactions. For each of these issues,
Rome of the findings from the field of survey research are described,
followed by a discussion of possible application of pertinent theories
and findings from cognitive research. In the concluding section, the
authors caution that the application of findings from cognitive research
Into events outside of the laboratory is not direct and requires
additional research. ~
The paper by Marquis addresses a topic that is of critical importance
in attempting to use surveys as a vehicle [or cognitive research.
Laboratory experiments in the cognitive sciences are generally designed
in a manner that permits direct, objective evaluation of the success of
subjects in performing specified cognitive tacks. For example, subjects
may be exposed to verbal material or other stimuli developed by the
experimenter and then asked to recall, recognize, or make Judgments about
these known stimuli. In survey research, however, the stimuli (in survey
research terms, the "truths) are not in general known to the survey
researcher. This makes it difficult, in comparing alternative interview
procedures, to determine which ones are most successful in minimizing
response effects. Attempts to overcome this difficulty come under the
general heading of validity checks, the subject of Marquis 'a paper. In
validity checks, data are sought from external sources, such an official
records, that are deemed to be less subject to response effects than are
the survey results. These data, either in aggregate or individual form,
are compared with data on the same subjects from surveys.
Marquises paper reviews alternative designs for validity checks and
discusses their strengths and weaknesses. ~~ ~~ ~ ~ ~ ~ ~
He argues that certain designs
lead to wrong conclusions about the direction and size of survey
reporting bias and gives some examples from record checks associated with
health surveys. He concludes that simple forgetting in not necessarily
the dominant problem in health surveys.
OCR for page 73
COGNITIVE SCIENCES AND SURVEY METHODS
Roger Tourangeau
In the horse of cognitive science there are many mansions. In one are
the carefully controlled studies of the laboratory; in another, elaborate
simulations or cognitive processes Dy computer sczentze~. Its subject
matter is no less varied than its methods, including topics as diverse as
the understanding of language and the inferring of causes, remembering
and forgetting, perception, and Judgment. It would take a kind of
archit eat of ideas to characterize thin rambling structure.
In this review, I shall risk far more undercoverage than any survey
researcher. I shall be purposive in my -sampling, fitting the selection
to my twin aims: my review will focus on the areas of cognitive science
that have the most direct bearing on how surveys are conducted, arid it
will focus on one or two arbitrarily selected areas where the cognitive
sciences could benefit most directly from the use of surveys. Much that
is relevant will be overlooked. The territory is Just too large and too
varied for a single foray to include more than a flew salient landmarks.
~ shall attempt to take two points of view in this survey. In
attempting to bring the cognitive sciences to bear on survey methods,
present a cognitive analysis of the tally of the respondent. In
attempting to bring survey methods to bear on cognitive problem, I shell
present the case for adding another mansion to the house of cognitive
science. ~ examine two particular topics--forgetting and
"optimism~--that could, I think, benefit from the use of survey samples.
My hope is that the reader will fall prey to the amply documented
tendency to overgeneralize from a few concrete canes.
The Respondent's Task
In the usual interview situation, the interviewer reads the respondent a
question and a set of response options. The respondent is supposed to
attend to and understand the question, recall whatever facts are
relevant, make a Judgment if the question calls for one, and select a
response. This list of cognitive operations is by no means exhaustive:
for example, with open-ended questions the respondent must formulate an
answer rather than selecting one from the pre-existing options.
Sometimes a respondent will short-circuit the process, deciding to refuse
to answer rather than to retrieve the relevant facts from memory. Nor is
the canonical order--cosIprehension, retrieval, Judgment, and
response--in~rariably followed. The respondent may select the answer
before the interviewer even finishes the question. Still, the respondent
must usually go through each of these four stops, typically in the
prescribed order.
As should be obvious from this description of the respondent's task,
there is considerable room for error. Respondents may misunderstand the
question or the response categories; they may forget or misremember the
crucial information; they may misjudge the information they do recall;
73
OCR for page 74
74
and they may misreport their answer. Consider respondents who are asked
whether they have seen a doctor in the past two weeks. First of all,
they may misunderstand the reference period: come may think that the
question covers the current calendar week plus the preceding one; others
may interpret it to mean the period begining two weeks ago to the day.
Even if the respondent does understand which period is being referred to,
he or she may find it difficult to determine the exact date bounding the
reference period. A study by Elizabeth Loftus (Loftus and Marburger,
1983) indicates that Rome people may even misreport whether they have had
a birthday "within the last six months. It seems likely that the source
of their error is their inability to translate that phrase into a
concrete date to which their birthday can be compared.
Even when they understand the question, respondents may forget or
misremember the relevant events. Two similar visits to the doctor can be
remembered as a single event; Linton's (1982) massive study of her own
memory for events in her daily life indicates that the inability to
distinguish similar or repeated events is a major source of forgetting.
The respondent who understands the question and who recalls the
relevant events may still slip up at the judgment stage. A respondent
who recalls a visit to the doctor but who can't quite date it has a
judgment to make: Did the visit fall within the reference period or
outside of it? Such judgments of recency are affected by a number of
factors aside from the actual timing of the event. For example, we tend
to judge emotionally salient events an more recent. Even when the
respondent can correctly retrieve the relevant facts and correctly judge
them, he or she may not report them accurately. We may omit some things
to avoid embarrassing ourselves, invent others to avoid disappointing the
interviewer. We also resolve ambiguities by recourse to what the
situation seems to demand. We ask ourselves whether a telephone call
constitutes a consultation with a doctor and our answer may depend on how
we perceive the relative demands of completeness versus relevance.
In the next four sections I examine each of these processes--
comprehension, recall, judgment, and response--in more detail; I consider
the main theoretical approaches used in studying them and attempt to draw
out the implications of the theories for the practical problems of survey
research.
Comprehension
Research on comprehension has, for reasons of methodological convenience,
concentrated on the comprehension of written material. The same
principles, however, are assumed to apply to spoken material as well; I
refer to materials of both types simply an "text. n
There are two main approaches to the study of comprehension. One
emphasizes the large cognitive structures the reader or listener
brings to bear on the text; the other places somewhat more emphasis on
the demands of the text itself. The two views are complementary rather
than contradictory. I refer to the first as the top-down approach and to
the second as the bottom-up approach. Although I try to sharpen the
differences between them, the two approaches share several important
OCR for page 75
75
notionn--the profound impact of context, the use of prior general
knowledge by the reader or listener, and the influence of inferential
processes during comprehension. The difference in the approaches lies
mainly in their views on the nature of the information we use in
interpreting a text. The top-down approach emphasizes large pre-existing
structures that can organize an entire text; the bottom-up approach
emphasi Zen lower-level structures that can be used piecemeal.
Top-Down Processing
According to the top-down view, we understand a t ext by imposing a
pre-existing organization on it . Until we impose such a structure, we
have considerable difficulty in stitching together successive ideas into
a coherent fabric (Bransford and Johnson, 1 972 ):
First you arrange items into groups. Of course one pile may be
sufficient depending on how much there is to do. If you have to go
somewhere else due to a lack of facilities that is the next step;
otherwise you are pretty well set. It in important not to overdo
things. That is, it is better to do too few things at once than too
many. . . . After the procedure ze completed, one arranges the
materials, into different groups again. They then can be put into
their appropriate places.
Although each sentence in this passage presents no special
difficulty, the passage as a whole is nearly impossible to
interpret--until, that is, we learn that the passage in about doing the
laundry. After that, the sentences fall neatly together into a tidy
conceptual package. With most passages we are able to rind some
conceptual package early on in the text . ~ The Branstord and Johnson
passage is deliberately constructed to prevent this from happening; it
uses general terms--nprocedure, n Biters , n Indifferent group, etc.--where
particular terms would have tipped us off.) Once the relevant structure
is recognized, each succeeding idea finds its niche in the larger
edifice. Comprehension is seen by the top-down approach as a process of
recognition: first we recognize the general pattern and then we
recognize how each piece taken its foreordained place in the pattern.
The patterns or structures are of two types. Some theorists (Mandler and
Johnson, 1977; R''melhart, 1975) stress the importance of our knowledge of
the general form of texts. We know, for example, that stories consist of
settings, beginnings, middles, and endings. We know, further, how each
constituent of a story breaks down into smaller constituents (e.g., a
setting includes the time, place, and cast of characters) and we know how
the constituents relate to each other ( e . g ., the beginning of the story
triggers some goal in the protagonist which he or she then attempts to
satisfy in the middle part of the story ~ . Our knowledge of the grammar
and logic of stories allows us to fit the events of a story into a
coherent structure. Other theorists (Abelson, 1 981; Bower et al ., 1 979 ;
Schank and Abelson, 1977) emphasize more particular knowledge of the
reader, knowledge about stereotypical situations. "Scripted is the label
OCR for page 76
76
usually applied to our knowledge about such everyday matters as doing the
laundry, going to the doctor, eating at a restaurant. We comprehend a
text by finding the pertinent script; then we match the ideas in the text
with the prototypical events and roles of the -script. The meaning of
Bransford and Johnson's passage remains elusive precisely because it
evokes no script.
Bottom-Up Processing
Not everything we read or hear falls neatly into some pre-existing mental
cubbyhole ~ Thoreau , cited in Miller , ~ 979 ):
Near the end of March, ~ 845, I borrowed an axe and went down to the
woods by Walden Pond, nearest to where I intended to build my house,
and began to cut down come tall, arrowy white pines, still in their
youth, for timber . . . . It was a pleasant hillside where I worked,
covered with pine woods, through which I looked out on the pond
. . . . The ice in the pond was not yet dissolved, though there were
some open spaces, and it was all dark-colored and maturated with
water.
Miller, taking the bottom-up approach, suggests we interpret this
passage by constructing a Memory image" for it (Miller, 1979:204~ :
My memory image grew piecemeal in roughly the following way. First
I read that the time was late March ; I formed no image at this point,
but filed it away for possible use later.... Next, I saw an
indistinct Thoreau borrow an axe from an even less distinct somebody
and walk, axe in hand, to some woods near a pond .
Most of us have never built a log cabin and have only the sketchiest
notion of what it would entail. Despite the absence of anything so well-
formed as a script, and despite the discursive, unstorylike nature of the
text, we have little difficulty in following the passage. We form some
sort of picture of what's going on (Miller's memory image) and the
details of the passage fit into that picture. All this is not to deny
the importance of prior knowledge or of higher-level cognitive structures
in the interpretation process: we may not know much about building log
cabins but we understand that Thoreau needs the axe for that purpose.
Although Miller does not stress the point, we inter a structure of goals
and subgoals: Thoreau borrows the axe to cut down the trees to build his
house.
The bottom-up approach (Kintsch and van Dick, 1978, on
Microstructure Foulest Miller, 1979; Miller and Johnson-Laird, 1976)
emphasizes the range of the prior knowledge used in interpreting texts.
In interpreting Thoreau, we draw upon our knowledge of early spring in
New England, of pine trees and axes , of the conditions of 1 845, of log
cabins , of whatever, in fact, is needed for us to form a coherent mental
image for the passage. It is mainly this data-driven character that
distinguishes the bottom-up approach. The story grammar and script
OCR for page 77
77
theorists tend to focus on a relatively small number of prior structures;
when discussing bottom-up processes by contract, theorists tend to be
quite catholic in their tastes, pointing out how the reader uses whatever
background knowledge may be relevant to the text at hand.
The Importance of Prior Knowledge and Context
Both approaches to comprehension emphasize our use of prior
knowledge--knowledge about the form of texts, knowledge about
stereotypical situations, knowledge about concrete details--in
interpretation. They also share the notion that context allows us to
activate and use the relevant pieces from our vast fund of background
information . Without context, we are unable to determine what
information is relevant to the passage at hand . With changing contexts,
we draw on radically different pieces of information in the
comprehension process leading to radically different readings. If
Raskolnikov, rather than Thoreau, had borrowed the axe, we would draw
rather different conclusions about his purpose.
Prior knowledge is used to connect the ideas in a passage. In
Thoreau's account, we saw how different actions are imbedded as goals and
subgoals. We fill in the linking connections and the omitted details.
Millers image of Thoreau's passage may be hazy in some respects, but it
includes more detail than the passage strictly warrants--he infers a man,
for example, perhaps snow on the ground. His reading honors both the
claims of the text and the claims of his own knowledge of the world.
Implications for Survey Methods
There are several main themes that both approaches share. We go far
beyond the information given in interpreting a text; we fill in gaps and
add details, making inferences that our background knowledge and the text
at hand seem to call for. The inferences we make depend on the prior
knowledge that is activated by the passage and its context; context
guides the selection of the relevant prior knowledge.
The work on comprehension has focused on connected text rather than
discrete items, on exposition rather than interrogation, on written
rather than spoken prone. The relatively little work that has been done
on answering questions has also tended to point up the importance of
prior structures--such as scripts and story gr~mmars--in guiding the
processes by which we seek the information that the question requests
(see Bower et al., 1979, and Mandler and Johnson, 1977~.
Despite the differences between an interview and a prose passage,
there are a few generalizations that probably apply in both situations.
First, the inference process can go awry--we may incorrectly infer what
is only possible or at best probable. The problem is compounded because
we do not sharply distinguish probable inferences (e.g., Thoreau's goal
in borrowing an axe) from necessary ones (e.g., that going to the site
required the narrator to move). Second, the context of a sentence (or
question) will to a large extent determine the nature of the inferences
OCR for page 78
78
drawn, the scripts invoked, and the background knowledge brought to bear
in interpreting it.
In more practical terms, the literature on comprehension suggests
that, other things being equal, related questions that are grouped may
have the advantage over questions that are scattered throughout an
instrument, because the grouped questions provide a helpful context for
interpretation; longer questions may have the advantage over shorter
ones, because longer questions create more context; explicit questions
may have the advantage over questions that rely on tacit knowledge,
because they leave less room for erroneous inferences; questions tailored
for different subgroups may have the advantage over more uniformly
standardized questions, because different subgroups have different stores
of background information that can lead to different interpretations.
Each of these generalizations in an oversimplification; they are intended
as guidelines rather than hard-and-fast rules. The advantages of long
questions, for example, may be offset by the increase in syntactic
complexity that usually accompanies increased length.
Survey researchers are hardly unaware of many of these points.
Bradburn (1982) describes several methodological studies on context and
question order effects. One of the findings apparently illustrates how
respondents may draw erroneous inferences about the meaning of a
question. In some cases, the answer to a general question (such as
Taken all together, how would you say things are these days--would you
say you are very happy, pretty happy, or not too happy? may change,
depending on whether it comes before or after specific questions (e.g.,
about one's marriage). Bradburn interprets this result to mean that when
the general question comes lash people infer that it covers only what has
been left out of the particular questions. Their inference may stem from
a general conversational rule that tells us to avoid unnecessary
redundancy in our answers (Haviland and Clark, 1974~.
Other studies (Belson, 1968; Fee, 1979) show how differences in
background can lead to differences in how a question is interpreted.
Even terms in common use (such an Energy crisis") have a wide range of
meanings; different kinds of people tend to favor different
interpretations of them.
Oral Presentation
Cognitive scientists have concentrated their energies on the
comprehension of written materials; it is natural to wonder how far we
can apply their conclusions to spoken ones. Much of the work that has
been done on the differences between reading and listening has come under
the banner of research on attitude change, where the concern has been to
compare the effectiveness of oral and written arguments. There is no
clear winner. Studies suggest, for example, that oral presentation may
be better for simple arguments but worse for complicated ones (Chaiken
and Eagly, 1976~. Other processes besides comprehension tend to be
implicated in these ~tudie~--oral presentations may be more effective
simply because they are more likely to commend our attention, but it may
be easier to understand written presentations if we bother to read them .
OCR for page 79
79
Mode of presentation may interact with other variables besides
complexity. Oral presentation can produce gains in short-term memory
performance, but the increaser seem to be limited to the last few items
in a series. Some phenomena are likelier to arise when material is
presented orally--homonyms present little difficulty when read, and
channel discrepancies between the verbal and nonverbal messages are
unlikely to occur with material presented in writing. On the whole,
however, there is little research on how the mode of presentation affects
comprehension.
There Is a fair amount of evidence that memory for spoken language i-Q
generally poor (see, for example, Reenan et al., 1977), although not
necessarily worse than memory for written language. (Eeenan et al. also
suggest that we may remember the irrelevancies, the asides, and the Joker
better than the gist of the presentation.) Given the typical length of
survey questions, forgetting the question is probably a leas common
source of respondent error than misunderstanding it.
Retrieval
Having interpreted--or misinterpreted--the question, the respondent now
faces the problem of answering it. Almost all questions require us to
rummage about our memories in search of an answer. The fallibility of
memory is, of course, more widely appreciated than the fallibility of the
comprehension process. This section takes up four issues: What do
cognitive scientists say about the structure of memory and the processes
used to search for information? What are the sources of forgetting?
What can be done about them? When can memory be trusted?
Memory is arguably academic psychology's oldest topic, systematic
work having begun more than a century ago with Ebbinghaus. Like all
topics, memory has had its ups and downs, but it has never fallen
completely out of favor, not even during the behavioristic reign of
terror that sent so many mentalistic topics into banishment. It would be
pointless to pretend that the vast literature on memory can be summarized
in a few broad strokes. It cannot. The best I can hope to do is to
highlight a few key ideas that relate most directly to the problems faced
by survey researchers.
Episodic Versus Semantic Memory
Hemory researchers distinguish between different stores for information,
ranging from the very brief persistence of visual information in iconic
storage to the more or less permanent retention of information in
long-term memory. Generally, three stores are distinguished: sensory
memory (which is thought to be very short lived); short-term memory
(which corresponds, in some accounts, roughly to the active contents of
consciousness); and long-term memory. Tulving (1972) has introduced a
further distinction between semantic and episodic memory. Episodic
memory contains our memories for experience, for events that are defined
by a particular time and place; semantic memory contains more general
OCR for page 80
80
knowledge, such as our knowledge of the meanings of words, that is
independent of the setting in which it was learned. Our memory for a
particular text generally resides in episodic memory; the background
knowledge used in interpreting the text generally resides in semantic
memory.
Tulving ( 1968; Tulving and Thomson, 1973) has argued that the
context in which an event is experienced or a word is encountered
determines how it in interpreted, stored, and retrieved from episodic
memory. His encoding specificity principle states that sometimes we [ail
even to recognize a previously learned item because at recognition the
item [ails to reinstate the exact encoding it was given during learning.
For example, when one puts eggs on a shopping list, he or she may fail to
purchase the right item if candy Easter eggs were intended and hen's eggs
were recalled.
Although come psychologists (J. Anderson, 1976) question Tulving's
distinction between semantic and episodic memory--arguing that both sorts
of memory are retained in a single store and follow the same principle~--
few question the importance of context in encoding and recall.
Memory as an Associative Network
Our long-term memories are not long lists of all the facts we have
learned and the events we have experienced. Memories are connected to
each other and the connections can be long and complicated or short and
direct. Some theorists (J. Anderson, 1976; Collins and Quillian, 1972)
view ~ong-term memory as an associative network. They view individual
ideas as nodes or points of intersection in the network; the ideas are
connected by links, which correspond to the relations between ideas .
The network analogy is u Ireful for a number of reasons . First, it in
compatible with most models of language comprehension; individual
sentences or passages of text can be represented using the same network
formalism. Second, it allows us to understand how items can be retrieved
and forgotten from memory. According to Anderson, for example, we
remember by searching relevant portions of the memory network. We forget
items that are relatively isolated--tew paths in the network lead to the
item sought--or whose connections to other items are poorly learned.
This perspective implies that good cues for remembering something are
those that lead us to search the right part of the network; in most
cases, the best cue is the item itself . It comes as no surprise that
recognition--recalling whether an item has been presented before--is
almost always better than other forms of recall.
Retrieval as Reconstruction
We noted earlier that when we read a passage, we make inferences; we
supply connections that are implicit in the text and we add plausible
details. There is an abundance or evidence that we encode and store our
interpretation of a text rather than the text itself ; our memory for gist
is far better than our verbatim memory (Bransford et al., 1972; Sachs,
OCR for page 81
81
1967). Not only do we lose verbatim details, we also seem to add (and
recall ) plausible details that fit our interpretation (Bartlett, 1932;
Bower et al., 1979; Bransford et al., 1972; Cantor and Mischel, 1977;
Handler and Johnson, 1977 ~ . It is not always clear whether we make the
inferences when we first encounter the material or when we remember it
later. Some theorists argue that we may do much of the "filling in" at
the time of recall . What we recall, of course, is likely to be
incomplete or spotty; we may round out what we can remember with what we
can infer. Once we draw an inference--during our initial encoding or
later during an attempt at retrieval--it can be added to our
representation of the event. LofLus and her colleagues have conducted
several demonstrations on thin point. In one (Loftus and Palmer, 1974,
Experiment II), the subjects watched a film of a traffic accident.
Afterwards, some of them were asked the question, "About how fast were
the cars going when they smashed into each other?n lathe rest were asked
a similar question with the word "hits replacing "smashed into." The
~smashed" subjects were more than twice as likely later on to report
incorrectly that they had seen broken glass in the film. The subjects
drew such inferences as "smashed into" seemed to warrant; later they
could not disentangle what they had seen from what they had inferred.
Sources of Forgetting
As with research on comprehension, so with research on memory: it is
more than a little hazardous to extrapolate from the sort of research
conducted by cognitive psychologists to the practical concerns of the
survey researcher. The vast bulk of the research on memory has concerned
the memory over short periods of time (typically a few minutes) of
arbitrary lists generally consisting of nonsense syllables. Although my
review has mainly covered research on more meaningful materials, such as
text or filmed events, it is still a long way from laboratory studies to
everyday memory. Fortunately, Linton's (1982) study of her own everyday
memory suggests that the memory problems encountered in the research
laboratory are similar to the ones encountered elsewhere.
The research I have attempted to summarize pinpoints several
d' {ferent reasons for forgetting. One reason for forgetting is that we
may not have transferred the relevant information into lon8-term memory.
We do not attend to everything and, even when we do, we often attend in
a rather mindless fashion. To find out whether one has already purchased
the morning newspaper, it may be better to search one' s briefcase than
one's memory. Another reason that we forget things ze that we simply
cannot retrieve them. Linton ~ ~ 982 ~ refers to this type of forgetting as
Simple failure to recall. ~ Some representation of the event made it
into long-term memory--in Linton's case, she could often recall such
items for months or years--but gradually it becomes inaccessible, lost
somewhere in the associative network. Linton describes another type of
retrieval failure: "During the fourth and subsequent years tof the
study] I began to encounter a few old items that simply di d not 'make
sense. ' ~ These items--brief descriptions of events that she had written
shortly after the events had taken place--no longer functioned as
OCR for page 138
138
of the response bias is as predicted: the idea of a dominant forgetting
bias appears to be a methodological artifact.
~ show estimates of hospital stay reporting errors first and then
estimates of physician visit reporting errors.
Hospital Stay Reporting Biases
The hospital stay bias esteem ten from the three kinds of record-check
designs (Table 6) show the expected effects with respect to the direction
of the response errors . The AC designs imply a net omission ~ i . e .,
forgetting) bias and the AB designs imply a net positive (i.e.,
telescoping ~ response bias. The two ABC design studies come close to
full design procedures; both find approximately equal rates of
underreporting and overreporting; one estimates a slightly higher
relative underreporting rate while the other estimates a slightly higher
overreporting rate.
Although a full design estimate inn 'A possible (neither study
provides enough information to estimate the D-cell ), net hospital stay
response bias is probably very small across purveys, and most of the
incomplete designs miss this conclusion entirely by concluding the
existence of a large directional bias.
Some of the studies that used incomplete designs obtained information
about the Missing. cell; for example, Campbell et al . ~ ~ 967 ~ found some
apparent overreports in their AC design and Ander-nen and Anderson found
some underreports in their AB design. This can happen if the person
reports (or the record contains) more than one hospital stay for the
person and the two sources disagree about the extra staying. But
multiple stays are atypical events and not a good basis for interring a
value for the missing cell. Occasionally researchers have pointed out
the low frequency in the Ming cell of incomplete designs and offered
this as empirical evidence for assuming that those kinds of' errors are
infrequent enough to ignore. But readers who have stayed with me this
far can see the potential fallacy in that argument.
The modified AB designs, which were used in the studies at the bottom
of Table 6, unsuccessfully tried to turn an AB design Onto a full
design. The modification was to check the records of all hospitals that
each family reported using for all members of. the faintly. For example,
it Mr. Jokes said he stayed at Metropolitan Hospital, the researchers
3The hall design estimates suggest the absence of. subst~tia' net reporting
biases. This is not something that can be derived from the record check
design and estimation characteristics. It is ~new" information that
Speculate about in the final section.
A reviewer suggests that ~ caution readers not to generalize this result to
response errors in reporting details of hospital admissions. Methods and
estimates of the latter kind are given in Marquis (1980, Section IIT). For
example, using ~A-cell" cares for which both survey and record information
was available, out-of-pocket cost net bias was close to zero while reports
of length-of-stay and month of admission showed a small net positive bins.
OCR for page 139
139
Cot
a.
a,
o
:c
o
ho
o
Cot
0
~ be
0 ~
o,
0
.=
~ C)
C. ~
t
~
Cat
o
c)
at;
w
o
~ o
o
0
O EN
or
a: or
:
>
CQ
:
D
Cat
O
S
or
:5
o
~
~3
J
m
¢
ED
be
~ _
0
0 ~ ~
to_
0 _
I:
0 ~ 0
C,
~ I:
at: ~ _
o
0
I l ° ~m'~ JO
Or ~ ~ ~ - -
_ _' _ _ _ _
1 1 1 1 1 1
c:
bo o
0 ~
0 ~
~ m
~n
;~
0
0
e ~
C,
5:
C,
6o
-
d—
~;
1 —
1 —
-
_
~o
_
0
:,
0
C,
o ~
1 0
1 0
_
~ 1
`0 1
~ _
_ C~
to
C~
_
0 _
~o
.
0
`: ~
~ ~ ~ ~ ~:
00 a
c:
0 m c~ c~ ~:
o
03
03
:,
0
0
~:
o
c)
, 1 ' ' O 1 , ~
~:
C)
0
C'
o
c)
0
4)
C~ ~
~: 53
C) ~ C,
y C~
e, 0
0 CI:
s 0 C, ~
C~ O ~ 0
~ ~ 0.
C, ~ 0 ~
~ O C)
0 a. `4 0
~ 0 C~
- ~ I 0
m =; 1 0
1 1 —
1 1 m
__10 ~
00 ~ 1
\0 ~o _ _
~ 01_
_ _ \0
_ _ · 0`
-
_ 0
`0 ~
— C)
- 0
~ O!)
O O
o,
0 1
1
C—
- :—
· -
C~
_
O O
~ ~ _.
0
m m m
~:
0 0
£:
bO 0
03 C
a'
0
~:
:~:
0
Co
-
-
0
0
~ 03
0 0
J 0
e
_
0
0 _
-
~:
0
0
a~
~ -
C, _
S 0
_ ~
D o
:~:
e~ _
C~
- ~
- ~
bO
~ 0
; 0
o
0
0
~ 0
0
0 o,
~ o
,
~ 0
o,
0 ~
cs ~
o ~
~ 0
~ s
o ~
~:
o
·e
0 S
C)
O td
Z ~
OCR for page 140
140
also asked Metropolitan Hospital to check its records for stays of
Mrs. Jones and for each of the children. While the modified procedure
does yield more C-cell entries, the C-cel' estimate in still not an
unbiased estimate of this population parameter. The resulting estimate
of the net response bias remains unsatisfactory.
Physician Visit Reporting Biases
Record-check estimates of errors in reporting physician visits also
illustrate the design biases of the incomplete approaches. There data
illustrate two other things: (~) the effect of a bias in the record
information (it can lead to an interpretation of a reporting bias with
the opposite sign) and (2) the length of the survey reference (recall)
period apparently does not have the systematic effect on omissions that
a memory decay theory would predict (the omission bias does not seem to
increase as the length of the average recall period gets larger ~ . These
three points are discussed next using the estimates in Table 7.
The incomplete designs yield physician visit reporting bias estimates
with the predicted signs. The three studies using AC designs (asking
about doctor visits known from records) produce large underreporting bins
estimates and suggest that either about a quarter, a third, or a half of
known visits are not reported in surveys. On the other hand, the two AB
design studies yield positive (overreporting) estimates of the response
bins . Within each of the ABCD design studies, the estimates of over- and
underreporting rates are approximately equal, raising the possibility5
that the net reporting bias is not significantly different from zero.
The ABC design estimates from Feather (1972) show how record bins can
distort estimates of the survey reporting bins, even when using something
close to a full design. Her overreport estimate (46 percent) is
substantially larger than her underreport estimate (14 percent),
suggesting that, on the average, her respondents reported a lot of
physician visits that actually did not occur within the 2-week reference
period of the survey. Feather's records of doctor visits were summary
"tee ~ubmissions,~ which are bills for complete outpatient services, each
of which may represent more than one office visit (e.g., a visit for the
complaint, a visit for diagnostic tenting, several visits for treatment,
and possibly one or two follow-up visits to monitor the treatment's
effectiveness). Thus the records ~underreport" visits. When they are
crons-cla-~sified with the survey reports, there are a large number of
B-cell entries (respondent reports of visits that cannot be matched to
record reports of visits ~ . The effect is to int1 ate the survey
overrepor~cing estimate, B/(A + B). Had Feather calculated a full design
estimate, (B - C)/(A ~ B + C + D), it would have had a large positive
value also. This is an illustration of the general principle that record
biases can show up as response biases with the opposite sign.
51 red, Marquis et al. (1979) used the information in these studies to
pr de full design estimates of the net response bias. These estimates
ar- .ot significantly different from zero.
OCR for page 141
141
o ~
A:
o ~ ~
~ C'
a: o _
ho
· -
o
ho
A:
o
o
Cat
o
of
lo;
· - be
of
I:
of C)
· - aC
of Cat
,
o
C)
at;
·.
o
C,
o
u3
~ En
a:
¢
En
D
~0
O
O
~ c:
to: 3 ~—
C,
A:
~ O
C"
a: ~
0
of
P At:
P.
be _
0 C~ L) :' o'
P. m - 0
—
U~ ~ ~ ~o
~ ~o
_
_ ~
C~ ~ ~o
C~
1 1 1
a 3 ~ ~ ~ 3 ~ 3 a ~i
~ ~ c~ cu ~ c~
C C C ~ ~ C ~ C ~ C
~ o,
· R ' ~ —
_ 3 3 ti30 11 3 ~ ,. ~
C~
0
~o ~
~o
0
-
o
o,
o'
o'
o,
p
:3
al
oq
0
~; ~
¢
o
C,
o:
o
c)
0
C~
OCR for page 142
142
The last point about Table 7 concerns memory deca, forgetting bias.
The studies use different reference period lengths, ranging from 2 weeks
deco 12 months. (They also use different crisis measures but the visit
measure and the reference period length are not completely confounded.)
If the memory decay hypothesis is correct, we should observe higher rates
of omission (underreports) in studies that use longer recall (reference)
periods. But the underreporting rates are not always larger for the
longer reference periods. For example, restricting our attention to the
"any visits measure, the 36 percent underreport estimate for 2 weeks in
the Bal?~uth et al. (1965) study is larger than the 11 percent
underreport estimate for 12 months in the Loewenstein (t969) study.6 I
return to the memory decay issue in the next section.
Reporting Biases for Other Variables
The principles illustrated above also apply to record checks for other
objective subject matter variables. Of particular interest is a recent
summary of Insensitive topics record checks (Marquis et al., 1981~. The
topics include receipt Or welfare, wage or salary income, illegal drug
use, alcohol consumption, arrests ~d convictions for crimes, and
embarrassing chronic conditions (e.g., h~orrho$~s and manta] illness).
Not surprisingly, there have been incompletely designs record checks in
these areas, and the conclusions drawn (~.g., people won't report
socially undesirable information) are a function of the type of design
used. Surprisingly (at least to me), the distribution of bias estimates
from the full design record checks center on zero or possibly a small
positive value for most of the sensitive topics, suggesting that the
errors respondents (or records, or match procedures) make are largely
Offsetting on the average (rather than being mostly in one direction)
as would be the case for forgetting or lying about sensitive information.
Implications for Future Research
In this section I speculate about the implications for research of a net
reporting bias close to zero, mention an example of a model that might
fit the data, and suggest that perhaps a different conceptualization of
offsetting errors is needed for health service use reports.
The empirical findings from Holly design" record checks of hospital
slay and doctor visit reports suggest that the net response bias is
approximately zero, at leant for surveys designed an those involved in
the cited record checks. ~ see three implications of these findings.
One implication in that forgetting (and its possible underlying
causes, such as [allure to acquire the information, failure to retrieve
6However, within a single reference period length, record checks usually show
a positive correlation of underreport probability and elapsed time between
the event and the interview. See Cannell et al. (1977) or Table 3 in this
paper for examples.
OCR for page 143
143
it, or decisions not to report it) in not necessarily the dominant
response problem in health surveys as we normally design them. Thus,
cognitive research that focuses only on forgetting hospital stays and
doctor visits may not have much applied value for contemporary health
surveys.
A second implication for cogn' Live research on health surveys in that
it should contain criterion validity features such as fully designed
record checks or carefully thought-out strategies of construct validity.
Over the past 10-15 years, survey methodologists have sometimes
substituted the assumption that "more is better" for empirical validity
studies, inferring that a survey procedure that produces more reports of
something yields better estimates than a lens productive procedure. But
the more-the-better assumption is unwarranted when fully designed record
checks show that the net response bias for current procedures is
approximately zero.
A third implication is that survey methodologi~t~ may want to
reexamine the effects of recent changes in health survey designs that
were incorporated to reduce forgetting biases (see, e.g., Cannell et al.,
1 977, Appendix ~ . Such changes may be producing a net positive response
bias.
A zero net response bias does not mean that survey responses are
given without error, only that the errors are offsetting when calculating
the survey mean (proportion, or other first-moment statistic).
Offsetting errors are of concern to survey designers and analysts because
they place limits on the precision of some population estimates and cause
biased estimates of other population parameters (e.g., coefficients of
associations. Cognitive science can make an extremely important
contribution to survey design by describing (e.g., modeling) these errors
and discovering what causes them (especially if the causes can be
influenced by features of the survey design such as the construction of
question sequencer , the length of recall periods, or the behavior of the
interviewer). The important point here is that the errors to be
described, understood, and controlled can be offsetting or possibly
random. They are not the product of a single cognitive process that
produces only omissions. Whatever cognitive processes are operating can
produce Just as many false positive reports as false negative reports, so
our explanations need to expand to take this phenomenon into account.
Sudman and 8radburn ( 1973) provide an example of a relevant response
error model and show that it can make useful predictions in household
expenditure surveys. They assume that forgetting and telescoping are two
separate cognitive processes that are affected by the length of the
recall interval. The proportion of forgetting response errors increases
as the recall interval increases while the proportion of telescoping
errors decreases with elapsed time. lithe effects of the two types of
errors will be approximately equal, then, for one particular reference
period length . So far, the full design record-check results do not
contradict this formulation if one assumes that the omission and
telescoping tendencies have approximately balanced out to create a net
reporting bias close to zero in all of the full design studies examined.
In addition, Cannell et al. (1977) cite several AC design studies that
show an apparent increase in omissions with an increase in e, apsed time.
OCR for page 144
144
But is this another methodological artifact or are we, indeed, observing
half of the compensating phenomenon suggested by Sudman and Bradburn?
If, for example, we could find AB design studies that show o~rerreporting
rates inversely proportional to elapsed time, the compensating forgetting
and telescoping explanation would receive support in the health survey
context. Unfortunately, none of the AB design record-check studies have
published elapsed time data.
Feather ' ~ ABC design study (Feather, ~ 972 ~ does not provide elapsed
time data for both hospital stay underreporting and overreporting. I
have adapted there data in Table 8, using the record version of date of
admission (where possible) to make the elapsed time classification.
Since date of admission was asked only for the most recent (or only)
hospital stay, we must confine the analysis to these stays (somewhere
between 75 and 80 percent of all reported and of all recorded stays).
The overreporting and underreporting trends in Table ~ do not support
the Sudman and Bradburn model. Both overreporting and underreporting
increase with elapsed time. Although not shown, the underreporting
trends are similar to those cited by Cannell et al. ~ ~ 977), suggesting
that the reporting dynamics in Feather's research are similar to those
operating in the other record-check surveys that ask about hospital
stays. Thus, a different error model may be needed for reporting of
hospital stays and physician visits.
We do observe, in the Feather data, an increase in both kinds of
response errors with greater elapsed times between the survey and the
event occurrence. It in thin phenomenon (that Rome survey methodologists
label random response error or simple response variance) that cognitive
science might address. For example, is this apparent increase in
"carelessness" with elapsed time characteristic of reporting of other
kinds of events? Is it due to problems recalling the relevant event
attributes correctly? Are the reporting problems due to faulty demigods
of the respondent about whether to report a recalled event or in the
retrieval of incorrect information about the event followed by a
logically correct decision about reporting? Understanding these kinds of
phenomena is not going to be gained by traditional survey evaluation
methods such as record checks. They need the kind of creative hypothesis
formulation and carefully controlled laboratory testing that cognitive
science can provide.
1
OCR for page 145
145
A:
C,
s
C'
1
o
c)
A:
· -
3
·rl
~0
o
lo;
U)
00
o
:r:
a)
a:
C)
A:
o
C)
Cal
m
e
En V)
0
o
~ a)
c)
~ 0
o
0
c) ~
a:
:D
c)
· -
o, D4
O a, ~
0—
:: _
o
3
Z
00
S
:^ 00
O O
a.
~ 0\ TO
_ a' up
:^
:3
o
C'
~ - ~ - -
s
b~
0 ~
~s ~
) oq
, ~:
-
~l
~:
~ 0
0
m
C~
C)
E~
:^
3
U.
a,
SC~
a'
_
1 1
— a'
CM
S
:~
D
0
~:
a
a,
o
C,
C~
o'
C)
S
C' C)
:~
0
`: ~
D
V
o
C)
o
C)
a,
3
0 0
S C}
E~
o
a.
0
:~
:,
a,
S.
o
a.
0
o
0
S
S
O o~
a)
:5
a
a.
c~ ~
~ ct
- :
·- ;4
~ o
)
o ~
z ~
o
o.
OCR for page 146
146
References
Andersen, R., end Anderson, 0.
967 I. Chicago: University of Chicago
Press.
Andersen, R., Gravity, J., and Anderson 0., ens.
1975 ~_~
Poliov. Cambridge, Mass.: Ballinger.
Andersen, R., Rasper, J., and Franicel, M.
1979 I. San Francisco: ~Jossey-Bass.
Balamuth, E., Shapiro, S., and Densen, P.~.
1961 Health interview responses compared with medical records.
_, Publication No. 584-D5, Ser. D, No. 5.
Washington, D.C.: 0.S. Public Health Service.
Barlow, R., Morgan, J., and Wirick, G.
1960 A study of validity in reporting medical care in Michigan.
Pp . 54-65 in ~ _
Washington, D.C.: American Statistical Association.
Cartwright,
1963
Feather, J.
1972
Belloc, N. B .
1954 Validation of morbidity survey data by comparison with
hospital records. ~ _
~ 49: 832-846.
Cannell, C. F., and Fowler, F. J.
1963 Comparison of hospitalization reporting in three survey
procedures. ~ Ser. D., No. 8. Washington,
D. C.: O. S. Public Health Service. Reprinted in Vi tat am
~ M I, Ser. 2, No. 8, July 1965.
Cannell, C,F., Fisher, G., and Bakker, T.
1961 Reporting of hospitalization in the health interview survey.
I, Ser. D., No. 4. Washington, D. C.:
U.S. Public Health Service. Reprinted in
Statl~i~, Ser. 2, No. 6, July. 1965.
Cannell, C., ~guis, R., and Laurent, A.
1977 A sugary of studies of interviewing methodology. Vital and
I, DH~1 Publication No. (ERA) 77-134B,
Ser. 2, No. 69. Washington, D. C.: U. S. Government Printing
Of.tice.
A.
Memory errors in morbidity survey. =
0~ 41:5-24.
A Response/Record Discrepancy Study. 0ni~rersi~cy of
Saskatchewan, Saskatoon, November 1972. Available from the
National Technical Information Service, Springfield,
Virginia.
Kirchner, C., Lerner, R. C., and Cla~rery, 0.
1969 The reported use of. medical care sources by low-in come
inpatients and outpatients. I_
84: 107-1 17.
OCR for page 147
147
Lowenstein, R.
1969
Two Approaches to Health Interview Surveys. School of Public
Health and Administrative Medicine, Columbia University.
Madow, W. G.
973 Net differences in interview data on chronic conditions and
information derived from medical records. ~ Row
Stating as, DHE~ Publication No. (BRA) 75-1331, Ser. 2,
No. 57. Washington, D. C.: U. S. Department of Health,
Education and Welfare.
Marquis, R . H.
1 97Ba Inferring health interview response bias from imperfect
record checks. Pp. 265-270 in _9
on Survey Research Methods. Washington, D. C.: American
Statistical Association.
1978b RYecord Check Y~iditY_of Rev Responses
,,,, , __ , _~.- _ _ A Reassessment
1980
_, R-23 1 9-HEN. Santa
Monica, Calit.: The Rand Corporation.
t~ m~ 6~ ~~
i, R-2555-HE-W. Santa
Monica, Calit.: The Rand Corporation.
_.
Marquis, R., et al.
1979 ~
~_, N-1 152-HhW. Santa
Monica, Calif.:
The Rand Corporation.
Marquis, R.R., et al.
1981 ~
_' R-2710/2-HHS.
Calit.: The Rand Corporation.
Santa Monica,
Net er, J., Maynes, F. S., and Ramanathan, R.
1965 The effect of mismatching on the measurement of response
errors.
60: 1005-1027.
Sudman, S., and Bradburn, N.M.
1973 Effects of. time and memory factors on responses in surveys.
~ _ 68:805-~15.
Sudman, S., Wallace, W., and Ferber, R.
1974 The Cost-Effectiveness of Using the Diary an an Instrument
for Collecting Health Data in Household Surveys. Survey
Research Laboratory, University of Illinois.
OCR for page 148
Representative terms from entire chapter:
record checks
