Click for next page ( 72


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 71
APPENDIX A BACKGROUND PAPERS This appendix contains three papers prepared specifically for the CASH project. The first two--nCognitive Science and Survey Methods by Roger Tourangeau and Potential Contributions of. Cognitive Sciences to Survey Questionnaire Designs by Norman M. Bradburn and Catalina Danin--were Rent to participants in advance of the St. Michael's seminar. The third paper, "Record Checks for Sample Surveyed by Kent Marquis, is based on a presentation by the author at St. Michaels. The first two papers focus on the basic questions addressed by the CASH project: What are the potential links between the cognitive sciences and survey research and how can each discipline contribute to progress in the other? However, the authors examine the questions from different perspectives. Tourangeau identifies four kinds of cognitive operations performed by respondents in survey interviews: understanding questions, retrieving relevant information from memory, making judgments (if called for), and selecting responses. He reviews research in the cognitive sciences on each of the four topics and discusses the possible relevance of research findings to survey interviews. In discussing Judgment and response, he Lives special attention to the implications of rinds ngs from the cognitive sciences for attitude questions in surveys. Tourangeau also discusses the possible uses of surveys as a Laboratory for cognitive research and explains how using surveys in this way might help to overcome some of the limitations of small-scale laboratory experiments. He describes one instance in which laboratory-based generalizations have been tested in surveys and briefly summarizes the results. Bradburn and Danis approach the came general subject from the point of view of the survey researcher. The authors first present a conceptual model of response effects, i.e., sources of variation in the quality of data obtained in surveys, and a general model for human information processing. They then contrast the research methods used in the cognitive sciences and in survey research methodological studies. The main part of their paper reviews potential contributions of findings from cognitive research to four of the issues most frequently studied by _ . . . . 71

OCR for page 71
72 survey researchers: question wording, response categories, contextual meaning' and response to survey items that require information on time or frequency of specific events or transactions. For each of these issues, Rome of the findings from the field of survey research are described, followed by a discussion of possible application of pertinent theories and findings from cognitive research. In the concluding section, the authors caution that the application of findings from cognitive research Into events outside of the laboratory is not direct and requires additional research. ~ The paper by Marquis addresses a topic that is of critical importance in attempting to use surveys as a vehicle [or cognitive research. Laboratory experiments in the cognitive sciences are generally designed in a manner that permits direct, objective evaluation of the success of subjects in performing specified cognitive tacks. For example, subjects may be exposed to verbal material or other stimuli developed by the experimenter and then asked to recall, recognize, or make Judgments about these known stimuli. In survey research, however, the stimuli (in survey research terms, the "truths) are not in general known to the survey researcher. This makes it difficult, in comparing alternative interview procedures, to determine which ones are most successful in minimizing response effects. Attempts to overcome this difficulty come under the general heading of validity checks, the subject of Marquis 'a paper. In validity checks, data are sought from external sources, such an official records, that are deemed to be less subject to response effects than are the survey results. These data, either in aggregate or individual form, are compared with data on the same subjects from surveys. Marquises paper reviews alternative designs for validity checks and discusses their strengths and weaknesses. ~~ ~~ ~ ~ ~ ~ ~ He argues that certain designs lead to wrong conclusions about the direction and size of survey reporting bias and gives some examples from record checks associated with health surveys. He concludes that simple forgetting in not necessarily the dominant problem in health surveys.

OCR for page 71
COGNITIVE SCIENCES AND SURVEY METHODS Roger Tourangeau In the horse of cognitive science there are many mansions. In one are the carefully controlled studies of the laboratory; in another, elaborate simulations or cognitive processes Dy computer sczentze~. Its subject matter is no less varied than its methods, including topics as diverse as the understanding of language and the inferring of causes, remembering and forgetting, perception, and Judgment. It would take a kind of archit eat of ideas to characterize thin rambling structure. In this review, I shall risk far more undercoverage than any survey researcher. I shall be purposive in my -sampling, fitting the selection to my twin aims: my review will focus on the areas of cognitive science that have the most direct bearing on how surveys are conducted, arid it will focus on one or two arbitrarily selected areas where the cognitive sciences could benefit most directly from the use of surveys. Much that is relevant will be overlooked. The territory is Just too large and too varied for a single foray to include more than a flew salient landmarks. ~ shall attempt to take two points of view in this survey. In attempting to bring the cognitive sciences to bear on survey methods, present a cognitive analysis of the tally of the respondent. In attempting to bring survey methods to bear on cognitive problem, I shell present the case for adding another mansion to the house of cognitive science. ~ examine two particular topics--forgetting and "optimism~--that could, I think, benefit from the use of survey samples. My hope is that the reader will fall prey to the amply documented tendency to overgeneralize from a few concrete canes. The Respondent's Task In the usual interview situation, the interviewer reads the respondent a question and a set of response options. The respondent is supposed to attend to and understand the question, recall whatever facts are relevant, make a Judgment if the question calls for one, and select a response. This list of cognitive operations is by no means exhaustive: for example, with open-ended questions the respondent must formulate an answer rather than selecting one from the pre-existing options. Sometimes a respondent will short-circuit the process, deciding to refuse to answer rather than to retrieve the relevant facts from memory. Nor is the canonical order--cosIprehension, retrieval, Judgment, and response--in~rariably followed. The respondent may select the answer before the interviewer even finishes the question. Still, the respondent must usually go through each of these four stops, typically in the prescribed order. As should be obvious from this description of the respondent's task, there is considerable room for error. Respondents may misunderstand the question or the response categories; they may forget or misremember the crucial information; they may misjudge the information they do recall; 73

OCR for page 71
74 and they may misreport their answer. Consider respondents who are asked whether they have seen a doctor in the past two weeks. First of all, they may misunderstand the reference period: come may think that the question covers the current calendar week plus the preceding one; others may interpret it to mean the period begining two weeks ago to the day. Even if the respondent does understand which period is being referred to, he or she may find it difficult to determine the exact date bounding the reference period. A study by Elizabeth Loftus (Loftus and Marburger, 1983) indicates that Rome people may even misreport whether they have had a birthday "within the last six months. It seems likely that the source of their error is their inability to translate that phrase into a concrete date to which their birthday can be compared. Even when they understand the question, respondents may forget or misremember the relevant events. Two similar visits to the doctor can be remembered as a single event; Linton's (1982) massive study of her own memory for events in her daily life indicates that the inability to distinguish similar or repeated events is a major source of forgetting. The respondent who understands the question and who recalls the relevant events may still slip up at the judgment stage. A respondent who recalls a visit to the doctor but who can't quite date it has a judgment to make: Did the visit fall within the reference period or outside of it? Such judgments of recency are affected by a number of factors aside from the actual timing of the event. For example, we tend to judge emotionally salient events an more recent. Even when the respondent can correctly retrieve the relevant facts and correctly judge them, he or she may not report them accurately. We may omit some things to avoid embarrassing ourselves, invent others to avoid disappointing the interviewer. We also resolve ambiguities by recourse to what the situation seems to demand. We ask ourselves whether a telephone call constitutes a consultation with a doctor and our answer may depend on how we perceive the relative demands of completeness versus relevance. In the next four sections I examine each of these processes-- comprehension, recall, judgment, and response--in more detail; I consider the main theoretical approaches used in studying them and attempt to draw out the implications of the theories for the practical problems of survey research. Comprehension Research on comprehension has, for reasons of methodological convenience, concentrated on the comprehension of written material. The same principles, however, are assumed to apply to spoken material as well; I refer to materials of both types simply an "text. n There are two main approaches to the study of comprehension. One emphasizes the large cognitive structures the reader or listener brings to bear on the text; the other places somewhat more emphasis on the demands of the text itself. The two views are complementary rather than contradictory. I refer to the first as the top-down approach and to the second as the bottom-up approach. Although I try to sharpen the differences between them, the two approaches share several important

OCR for page 71
75 notionn--the profound impact of context, the use of prior general knowledge by the reader or listener, and the influence of inferential processes during comprehension. The difference in the approaches lies mainly in their views on the nature of the information we use in interpreting a text. The top-down approach emphasizes large pre-existing structures that can organize an entire text; the bottom-up approach emphasi Zen lower-level structures that can be used piecemeal. Top-Down Processing According to the top-down view, we understand a t ext by imposing a pre-existing organization on it . Until we impose such a structure, we have considerable difficulty in stitching together successive ideas into a coherent fabric (Bransford and Johnson, 1 972 ): First you arrange items into groups. Of course one pile may be sufficient depending on how much there is to do. If you have to go somewhere else due to a lack of facilities that is the next step; otherwise you are pretty well set. It in important not to overdo things. That is, it is better to do too few things at once than too many. . . . After the procedure ze completed, one arranges the materials, into different groups again. They then can be put into their appropriate places. Although each sentence in this passage presents no special difficulty, the passage as a whole is nearly impossible to interpret--until, that is, we learn that the passage in about doing the laundry. After that, the sentences fall neatly together into a tidy conceptual package. With most passages we are able to rind some conceptual package early on in the text . ~ The Branstord and Johnson passage is deliberately constructed to prevent this from happening; it uses general terms--nprocedure, n Biters , n Indifferent group, etc.--where particular terms would have tipped us off.) Once the relevant structure is recognized, each succeeding idea finds its niche in the larger edifice. Comprehension is seen by the top-down approach as a process of recognition: first we recognize the general pattern and then we recognize how each piece taken its foreordained place in the pattern. The patterns or structures are of two types. Some theorists (Mandler and Johnson, 1977; R''melhart, 1975) stress the importance of our knowledge of the general form of texts. We know, for example, that stories consist of settings, beginnings, middles, and endings. We know, further, how each constituent of a story breaks down into smaller constituents (e.g., a setting includes the time, place, and cast of characters) and we know how the constituents relate to each other ( e . g ., the beginning of the story triggers some goal in the protagonist which he or she then attempts to satisfy in the middle part of the story ~ . Our knowledge of the grammar and logic of stories allows us to fit the events of a story into a coherent structure. Other theorists (Abelson, 1 981; Bower et al ., 1 979 ; Schank and Abelson, 1977) emphasize more particular knowledge of the reader, knowledge about stereotypical situations. "Scripted is the label

OCR for page 71
76 usually applied to our knowledge about such everyday matters as doing the laundry, going to the doctor, eating at a restaurant. We comprehend a text by finding the pertinent script; then we match the ideas in the text with the prototypical events and roles of the -script. The meaning of Bransford and Johnson's passage remains elusive precisely because it evokes no script. Bottom-Up Processing Not everything we read or hear falls neatly into some pre-existing mental cubbyhole ~ Thoreau , cited in Miller , ~ 979 ): Near the end of March, ~ 845, I borrowed an axe and went down to the woods by Walden Pond, nearest to where I intended to build my house, and began to cut down come tall, arrowy white pines, still in their youth, for timber . . . . It was a pleasant hillside where I worked, covered with pine woods, through which I looked out on the pond . . . . The ice in the pond was not yet dissolved, though there were some open spaces, and it was all dark-colored and maturated with water. Miller, taking the bottom-up approach, suggests we interpret this passage by constructing a Memory image" for it (Miller, 1979:204~ : My memory image grew piecemeal in roughly the following way. First I read that the time was late March ; I formed no image at this point, but filed it away for possible use later.... Next, I saw an indistinct Thoreau borrow an axe from an even less distinct somebody and walk, axe in hand, to some woods near a pond . Most of us have never built a log cabin and have only the sketchiest notion of what it would entail. Despite the absence of anything so well- formed as a script, and despite the discursive, unstorylike nature of the text, we have little difficulty in following the passage. We form some sort of picture of what's going on (Miller's memory image) and the details of the passage fit into that picture. All this is not to deny the importance of prior knowledge or of higher-level cognitive structures in the interpretation process: we may not know much about building log cabins but we understand that Thoreau needs the axe for that purpose. Although Miller does not stress the point, we inter a structure of goals and subgoals: Thoreau borrows the axe to cut down the trees to build his house. The bottom-up approach (Kintsch and van Dick, 1978, on Microstructure Foulest Miller, 1979; Miller and Johnson-Laird, 1976) emphasizes the range of the prior knowledge used in interpreting texts. In interpreting Thoreau, we draw upon our knowledge of early spring in New England, of pine trees and axes , of the conditions of 1 845, of log cabins , of whatever, in fact, is needed for us to form a coherent mental image for the passage. It is mainly this data-driven character that distinguishes the bottom-up approach. The story grammar and script

OCR for page 71
77 theorists tend to focus on a relatively small number of prior structures; when discussing bottom-up processes by contract, theorists tend to be quite catholic in their tastes, pointing out how the reader uses whatever background knowledge may be relevant to the text at hand. The Importance of Prior Knowledge and Context Both approaches to comprehension emphasize our use of prior knowledge--knowledge about the form of texts, knowledge about stereotypical situations, knowledge about concrete details--in interpretation. They also share the notion that context allows us to activate and use the relevant pieces from our vast fund of background information . Without context, we are unable to determine what information is relevant to the passage at hand . With changing contexts, we draw on radically different pieces of information in the comprehension process leading to radically different readings. If Raskolnikov, rather than Thoreau, had borrowed the axe, we would draw rather different conclusions about his purpose. Prior knowledge is used to connect the ideas in a passage. In Thoreau's account, we saw how different actions are imbedded as goals and subgoals. We fill in the linking connections and the omitted details. Millers image of Thoreau's passage may be hazy in some respects, but it includes more detail than the passage strictly warrants--he infers a man, for example, perhaps snow on the ground. His reading honors both the claims of the text and the claims of his own knowledge of the world. Implications for Survey Methods There are several main themes that both approaches share. We go far beyond the information given in interpreting a text; we fill in gaps and add details, making inferences that our background knowledge and the text at hand seem to call for. The inferences we make depend on the prior knowledge that is activated by the passage and its context; context guides the selection of the relevant prior knowledge. The work on comprehension has focused on connected text rather than discrete items, on exposition rather than interrogation, on written rather than spoken prone. The relatively little work that has been done on answering questions has also tended to point up the importance of prior structures--such as scripts and story gr~mmars--in guiding the processes by which we seek the information that the question requests (see Bower et al., 1979, and Mandler and Johnson, 1977~. Despite the differences between an interview and a prose passage, there are a few generalizations that probably apply in both situations. First, the inference process can go awry--we may incorrectly infer what is only possible or at best probable. The problem is compounded because we do not sharply distinguish probable inferences (e.g., Thoreau's goal in borrowing an axe) from necessary ones (e.g., that going to the site required the narrator to move). Second, the context of a sentence (or question) will to a large extent determine the nature of the inferences

OCR for page 71
78 drawn, the scripts invoked, and the background knowledge brought to bear in interpreting it. In more practical terms, the literature on comprehension suggests that, other things being equal, related questions that are grouped may have the advantage over questions that are scattered throughout an instrument, because the grouped questions provide a helpful context for interpretation; longer questions may have the advantage over shorter ones, because longer questions create more context; explicit questions may have the advantage over questions that rely on tacit knowledge, because they leave less room for erroneous inferences; questions tailored for different subgroups may have the advantage over more uniformly standardized questions, because different subgroups have different stores of background information that can lead to different interpretations. Each of these generalizations in an oversimplification; they are intended as guidelines rather than hard-and-fast rules. The advantages of long questions, for example, may be offset by the increase in syntactic complexity that usually accompanies increased length. Survey researchers are hardly unaware of many of these points. Bradburn (1982) describes several methodological studies on context and question order effects. One of the findings apparently illustrates how respondents may draw erroneous inferences about the meaning of a question. In some cases, the answer to a general question (such as Taken all together, how would you say things are these days--would you say you are very happy, pretty happy, or not too happy? may change, depending on whether it comes before or after specific questions (e.g., about one's marriage). Bradburn interprets this result to mean that when the general question comes lash people infer that it covers only what has been left out of the particular questions. Their inference may stem from a general conversational rule that tells us to avoid unnecessary redundancy in our answers (Haviland and Clark, 1974~. Other studies (Belson, 1968; Fee, 1979) show how differences in background can lead to differences in how a question is interpreted. Even terms in common use (such an Energy crisis") have a wide range of meanings; different kinds of people tend to favor different interpretations of them. Oral Presentation Cognitive scientists have concentrated their energies on the comprehension of written materials; it is natural to wonder how far we can apply their conclusions to spoken ones. Much of the work that has been done on the differences between reading and listening has come under the banner of research on attitude change, where the concern has been to compare the effectiveness of oral and written arguments. There is no clear winner. Studies suggest, for example, that oral presentation may be better for simple arguments but worse for complicated ones (Chaiken and Eagly, 1976~. Other processes besides comprehension tend to be implicated in these ~tudie~--oral presentations may be more effective simply because they are more likely to commend our attention, but it may be easier to understand written presentations if we bother to read them .

OCR for page 71
79 Mode of presentation may interact with other variables besides complexity. Oral presentation can produce gains in short-term memory performance, but the increaser seem to be limited to the last few items in a series. Some phenomena are likelier to arise when material is presented orally--homonyms present little difficulty when read, and channel discrepancies between the verbal and nonverbal messages are unlikely to occur with material presented in writing. On the whole, however, there is little research on how the mode of presentation affects comprehension. There Is a fair amount of evidence that memory for spoken language i-Q generally poor (see, for example, Reenan et al., 1977), although not necessarily worse than memory for written language. (Eeenan et al. also suggest that we may remember the irrelevancies, the asides, and the Joker better than the gist of the presentation.) Given the typical length of survey questions, forgetting the question is probably a leas common source of respondent error than misunderstanding it. Retrieval Having interpreted--or misinterpreted--the question, the respondent now faces the problem of answering it. Almost all questions require us to rummage about our memories in search of an answer. The fallibility of memory is, of course, more widely appreciated than the fallibility of the comprehension process. This section takes up four issues: What do cognitive scientists say about the structure of memory and the processes used to search for information? What are the sources of forgetting? What can be done about them? When can memory be trusted? Memory is arguably academic psychology's oldest topic, systematic work having begun more than a century ago with Ebbinghaus. Like all topics, memory has had its ups and downs, but it has never fallen completely out of favor, not even during the behavioristic reign of terror that sent so many mentalistic topics into banishment. It would be pointless to pretend that the vast literature on memory can be summarized in a few broad strokes. It cannot. The best I can hope to do is to highlight a few key ideas that relate most directly to the problems faced by survey researchers. Episodic Versus Semantic Memory Hemory researchers distinguish between different stores for information, ranging from the very brief persistence of visual information in iconic storage to the more or less permanent retention of information in long-term memory. Generally, three stores are distinguished: sensory memory (which is thought to be very short lived); short-term memory (which corresponds, in some accounts, roughly to the active contents of consciousness); and long-term memory. Tulving (1972) has introduced a further distinction between semantic and episodic memory. Episodic memory contains our memories for experience, for events that are defined by a particular time and place; semantic memory contains more general

OCR for page 71
80 knowledge, such as our knowledge of the meanings of words, that is independent of the setting in which it was learned. Our memory for a particular text generally resides in episodic memory; the background knowledge used in interpreting the text generally resides in semantic memory. Tulving ( 1968; Tulving and Thomson, 1973) has argued that the context in which an event is experienced or a word is encountered determines how it in interpreted, stored, and retrieved from episodic memory. His encoding specificity principle states that sometimes we [ail even to recognize a previously learned item because at recognition the item [ails to reinstate the exact encoding it was given during learning. For example, when one puts eggs on a shopping list, he or she may fail to purchase the right item if candy Easter eggs were intended and hen's eggs were recalled. Although come psychologists (J. Anderson, 1976) question Tulving's distinction between semantic and episodic memory--arguing that both sorts of memory are retained in a single store and follow the same principle~-- few question the importance of context in encoding and recall. Memory as an Associative Network Our long-term memories are not long lists of all the facts we have learned and the events we have experienced. Memories are connected to each other and the connections can be long and complicated or short and direct. Some theorists (J. Anderson, 1976; Collins and Quillian, 1972) view ~ong-term memory as an associative network. They view individual ideas as nodes or points of intersection in the network; the ideas are connected by links, which correspond to the relations between ideas . The network analogy is u Ireful for a number of reasons . First, it in compatible with most models of language comprehension; individual sentences or passages of text can be represented using the same network formalism. Second, it allows us to understand how items can be retrieved and forgotten from memory. According to Anderson, for example, we remember by searching relevant portions of the memory network. We forget items that are relatively isolated--tew paths in the network lead to the item sought--or whose connections to other items are poorly learned. This perspective implies that good cues for remembering something are those that lead us to search the right part of the network; in most cases, the best cue is the item itself . It comes as no surprise that recognition--recalling whether an item has been presented before--is almost always better than other forms of recall. Retrieval as Reconstruction We noted earlier that when we read a passage, we make inferences; we supply connections that are implicit in the text and we add plausible details. There is an abundance or evidence that we encode and store our interpretation of a text rather than the text itself ; our memory for gist is far better than our verbatim memory (Bransford et al., 1972; Sachs,

OCR for page 71
81 1967). Not only do we lose verbatim details, we also seem to add (and recall ) plausible details that fit our interpretation (Bartlett, 1932; Bower et al., 1979; Bransford et al., 1972; Cantor and Mischel, 1977; Handler and Johnson, 1977 ~ . It is not always clear whether we make the inferences when we first encounter the material or when we remember it later. Some theorists argue that we may do much of the "filling in" at the time of recall . What we recall, of course, is likely to be incomplete or spotty; we may round out what we can remember with what we can infer. Once we draw an inference--during our initial encoding or later during an attempt at retrieval--it can be added to our representation of the event. LofLus and her colleagues have conducted several demonstrations on thin point. In one (Loftus and Palmer, 1974, Experiment II), the subjects watched a film of a traffic accident. Afterwards, some of them were asked the question, "About how fast were the cars going when they smashed into each other?n lathe rest were asked a similar question with the word "hits replacing "smashed into." The ~smashed" subjects were more than twice as likely later on to report incorrectly that they had seen broken glass in the film. The subjects drew such inferences as "smashed into" seemed to warrant; later they could not disentangle what they had seen from what they had inferred. Sources of Forgetting As with research on comprehension, so with research on memory: it is more than a little hazardous to extrapolate from the sort of research conducted by cognitive psychologists to the practical concerns of the survey researcher. The vast bulk of the research on memory has concerned the memory over short periods of time (typically a few minutes) of arbitrary lists generally consisting of nonsense syllables. Although my review has mainly covered research on more meaningful materials, such as text or filmed events, it is still a long way from laboratory studies to everyday memory. Fortunately, Linton's (1982) study of her own everyday memory suggests that the memory problems encountered in the research laboratory are similar to the ones encountered elsewhere. The research I have attempted to summarize pinpoints several d' {ferent reasons for forgetting. One reason for forgetting is that we may not have transferred the relevant information into lon8-term memory. We do not attend to everything and, even when we do, we often attend in a rather mindless fashion. To find out whether one has already purchased the morning newspaper, it may be better to search one' s briefcase than one's memory. Another reason that we forget things ze that we simply cannot retrieve them. Linton ~ ~ 982 ~ refers to this type of forgetting as Simple failure to recall. ~ Some representation of the event made it into long-term memory--in Linton's case, she could often recall such items for months or years--but gradually it becomes inaccessible, lost somewhere in the associative network. Linton describes another type of retrieval failure: "During the fourth and subsequent years tof the study] I began to encounter a few old items that simply di d not 'make sense. ' ~ These items--brief descriptions of events that she had written shortly after the events had taken place--no longer functioned as

OCR for page 71
138 of the response bias is as predicted: the idea of a dominant forgetting bias appears to be a methodological artifact. ~ show estimates of hospital stay reporting errors first and then estimates of physician visit reporting errors. Hospital Stay Reporting Biases The hospital stay bias esteem ten from the three kinds of record-check designs (Table 6) show the expected effects with respect to the direction of the response errors . The AC designs imply a net omission ~ i . e ., forgetting) bias and the AB designs imply a net positive (i.e., telescoping ~ response bias. The two ABC design studies come close to full design procedures; both find approximately equal rates of underreporting and overreporting; one estimates a slightly higher relative underreporting rate while the other estimates a slightly higher overreporting rate. Although a full design estimate inn 'A possible (neither study provides enough information to estimate the D-cell ), net hospital stay response bias is probably very small across purveys, and most of the incomplete designs miss this conclusion entirely by concluding the existence of a large directional bias. Some of the studies that used incomplete designs obtained information about the Missing. cell; for example, Campbell et al . ~ ~ 967 ~ found some apparent overreports in their AC design and Ander-nen and Anderson found some underreports in their AB design. This can happen if the person reports (or the record contains) more than one hospital stay for the person and the two sources disagree about the extra staying. But multiple stays are atypical events and not a good basis for interring a value for the missing cell. Occasionally researchers have pointed out the low frequency in the Ming cell of incomplete designs and offered this as empirical evidence for assuming that those kinds of' errors are infrequent enough to ignore. But readers who have stayed with me this far can see the potential fallacy in that argument. The modified AB designs, which were used in the studies at the bottom of Table 6, unsuccessfully tried to turn an AB design Onto a full design. The modification was to check the records of all hospitals that each family reported using for all members of. the faintly. For example, it Mr. Jokes said he stayed at Metropolitan Hospital, the researchers 3The hall design estimates suggest the absence of. subst~tia' net reporting biases. This is not something that can be derived from the record check design and estimation characteristics. It is ~new" information that Speculate about in the final section. A reviewer suggests that ~ caution readers not to generalize this result to response errors in reporting details of hospital admissions. Methods and estimates of the latter kind are given in Marquis (1980, Section IIT). For example, using ~A-cell" cares for which both survey and record information was available, out-of-pocket cost net bias was close to zero while reports of length-of-stay and month of admission showed a small net positive bins.

OCR for page 71
139 Cot a. a, o :c o ho o Cot 0 ~ be 0 ~ o, 0 .= ~ C) C. ~ t ~ Cat o c) at; w o ~ o o 0 O EN or a: or : > CQ : D Cat O S or :5 o ~ ~3 J m ¢ ED be ~ _ 0 0 ~ ~ to_ 0 _ I: 0 ~ 0 C, ~ I: at: ~ _ o 0 I l ° ~m'~ JO Or ~ ~ ~ - - _ _' _ _ _ _ 1 1 1 1 1 1 c: bo o 0 ~ 0 ~ ~ m ~n ;~ 0 0 e ~ C, 5: C, 6o - d— ~; 1 — 1 — - _ ~o _ 0 :, 0 C, o ~ 1 0 1 0 _ ~ 1 `0 1 ~ _ _ C~ to C~ _ 0 _ ~o . 0 `: ~ ~ ~ ~ ~ ~: 00 a c: 0 m c~ c~ ~: o 03 03 :, 0 0 ~: o c) , 1 ' ' O 1 , ~ ~: C) 0 C' o c) 0 4) C~ ~ ~: 53 C) ~ C, y C~ e, 0 0 CI: s 0 C, ~ C~ O ~ 0 ~ ~ 0. C, ~ 0 ~ ~ O C) 0 a. `4 0 ~ 0 C~ - ~ I 0 m =; 1 0 1 1 — 1 1 m __10 ~ 00 ~ 1 \0 ~o _ _ ~ 01_ _ _ \0 _ _ · 0` - _ 0 `0 ~ — C) - 0 ~ O!) O O o, 0 1 1 C— - :— · - C~ _ O O ~ ~ _. 0 m m m ~: 0 0 £: bO 0 03 C a' 0 ~: :~: 0 Co - - 0 0 ~ 03 0 0 J 0 e _ 0 0 _ - ~: 0 0 a~ ~ - C, _ S 0 _ ~ D o :~: e~ _ C~ - ~ - ~ bO ~ 0 ; 0 o 0 0 ~ 0 0 0 o, ~ o , ~ 0 o, 0 ~ cs ~ o ~ ~ 0 ~ s o ~ ~: o ·e 0 S C) O td Z ~

OCR for page 71
140 also asked Metropolitan Hospital to check its records for stays of Mrs. Jones and for each of the children. While the modified procedure does yield more C-cell entries, the C-cel' estimate in still not an unbiased estimate of this population parameter. The resulting estimate of the net response bias remains unsatisfactory. Physician Visit Reporting Biases Record-check estimates of errors in reporting physician visits also illustrate the design biases of the incomplete approaches. There data illustrate two other things: (~) the effect of a bias in the record information (it can lead to an interpretation of a reporting bias with the opposite sign) and (2) the length of the survey reference (recall) period apparently does not have the systematic effect on omissions that a memory decay theory would predict (the omission bias does not seem to increase as the length of the average recall period gets larger ~ . These three points are discussed next using the estimates in Table 7. The incomplete designs yield physician visit reporting bias estimates with the predicted signs. The three studies using AC designs (asking about doctor visits known from records) produce large underreporting bins estimates and suggest that either about a quarter, a third, or a half of known visits are not reported in surveys. On the other hand, the two AB design studies yield positive (overreporting) estimates of the response bins . Within each of the ABCD design studies, the estimates of over- and underreporting rates are approximately equal, raising the possibility5 that the net reporting bias is not significantly different from zero. The ABC design estimates from Feather (1972) show how record bins can distort estimates of the survey reporting bins, even when using something close to a full design. Her overreport estimate (46 percent) is substantially larger than her underreport estimate (14 percent), suggesting that, on the average, her respondents reported a lot of physician visits that actually did not occur within the 2-week reference period of the survey. Feather's records of doctor visits were summary "tee ~ubmissions,~ which are bills for complete outpatient services, each of which may represent more than one office visit (e.g., a visit for the complaint, a visit for diagnostic tenting, several visits for treatment, and possibly one or two follow-up visits to monitor the treatment's effectiveness). Thus the records ~underreport" visits. When they are crons-cla-~sified with the survey reports, there are a large number of B-cell entries (respondent reports of visits that cannot be matched to record reports of visits ~ . The effect is to int1 ate the survey overrepor~cing estimate, B/(A + B). Had Feather calculated a full design estimate, (B - C)/(A ~ B + C + D), it would have had a large positive value also. This is an illustration of the general principle that record biases can show up as response biases with the opposite sign. 51 red, Marquis et al. (1979) used the information in these studies to pr de full design estimates of the net response bias. These estimates ar- .ot significantly different from zero.

OCR for page 71
141 o ~ A: o ~ ~ ~ C' a: o _ ho · - o ho A: o o Cat o of lo; · - be of I: of C) · - aC of Cat , o C) at; ·. o C, o u3 ~ En a: ¢ En D ~0 O O ~ c: to: 3 ~— C, A: ~ O C" a: ~ 0 of P At: P. be _ 0 C~ L) :' o' P. m - 0 — U~ ~ ~ ~o ~ ~o _ _ ~ C~ ~ ~o C~ 1 1 1 a 3 ~ ~ ~ 3 ~ 3 a ~i ~ ~ c~ cu ~ c~ C C C ~ ~ C ~ C ~ C ~ o, · R ' ~ — _ 3 3 ti30 11 3 ~ ,. ~ C~ 0 ~o ~ ~o 0 - o o, o' o' o, p :3 al oq 0 ~; ~ ¢ o C, o: o c) 0 C~

OCR for page 71
142 The last point about Table 7 concerns memory deca, forgetting bias. The studies use different reference period lengths, ranging from 2 weeks deco 12 months. (They also use different crisis measures but the visit measure and the reference period length are not completely confounded.) If the memory decay hypothesis is correct, we should observe higher rates of omission (underreports) in studies that use longer recall (reference) periods. But the underreporting rates are not always larger for the longer reference periods. For example, restricting our attention to the "any visits measure, the 36 percent underreport estimate for 2 weeks in the Bal?~uth et al. (1965) study is larger than the 11 percent underreport estimate for 12 months in the Loewenstein (t969) study.6 I return to the memory decay issue in the next section. Reporting Biases for Other Variables The principles illustrated above also apply to record checks for other objective subject matter variables. Of particular interest is a recent summary of Insensitive topics record checks (Marquis et al., 1981~. The topics include receipt Or welfare, wage or salary income, illegal drug use, alcohol consumption, arrests ~d convictions for crimes, and embarrassing chronic conditions (e.g., h~orrho$~s and manta] illness). Not surprisingly, there have been incompletely designs record checks in these areas, and the conclusions drawn (~.g., people won't report socially undesirable information) are a function of the type of design used. Surprisingly (at least to me), the distribution of bias estimates from the full design record checks center on zero or possibly a small positive value for most of the sensitive topics, suggesting that the errors respondents (or records, or match procedures) make are largely Offsetting on the average (rather than being mostly in one direction) as would be the case for forgetting or lying about sensitive information. Implications for Future Research In this section I speculate about the implications for research of a net reporting bias close to zero, mention an example of a model that might fit the data, and suggest that perhaps a different conceptualization of offsetting errors is needed for health service use reports. The empirical findings from Holly design" record checks of hospital slay and doctor visit reports suggest that the net response bias is approximately zero, at leant for surveys designed an those involved in the cited record checks. ~ see three implications of these findings. One implication in that forgetting (and its possible underlying causes, such as [allure to acquire the information, failure to retrieve 6However, within a single reference period length, record checks usually show a positive correlation of underreport probability and elapsed time between the event and the interview. See Cannell et al. (1977) or Table 3 in this paper for examples.

OCR for page 71
143 it, or decisions not to report it) in not necessarily the dominant response problem in health surveys as we normally design them. Thus, cognitive research that focuses only on forgetting hospital stays and doctor visits may not have much applied value for contemporary health surveys. A second implication for cogn' Live research on health surveys in that it should contain criterion validity features such as fully designed record checks or carefully thought-out strategies of construct validity. Over the past 10-15 years, survey methodologists have sometimes substituted the assumption that "more is better" for empirical validity studies, inferring that a survey procedure that produces more reports of something yields better estimates than a lens productive procedure. But the more-the-better assumption is unwarranted when fully designed record checks show that the net response bias for current procedures is approximately zero. A third implication is that survey methodologi~t~ may want to reexamine the effects of recent changes in health survey designs that were incorporated to reduce forgetting biases (see, e.g., Cannell et al., 1 977, Appendix ~ . Such changes may be producing a net positive response bias. A zero net response bias does not mean that survey responses are given without error, only that the errors are offsetting when calculating the survey mean (proportion, or other first-moment statistic). Offsetting errors are of concern to survey designers and analysts because they place limits on the precision of some population estimates and cause biased estimates of other population parameters (e.g., coefficients of associations. Cognitive science can make an extremely important contribution to survey design by describing (e.g., modeling) these errors and discovering what causes them (especially if the causes can be influenced by features of the survey design such as the construction of question sequencer , the length of recall periods, or the behavior of the interviewer). The important point here is that the errors to be described, understood, and controlled can be offsetting or possibly random. They are not the product of a single cognitive process that produces only omissions. Whatever cognitive processes are operating can produce Just as many false positive reports as false negative reports, so our explanations need to expand to take this phenomenon into account. Sudman and 8radburn ( 1973) provide an example of a relevant response error model and show that it can make useful predictions in household expenditure surveys. They assume that forgetting and telescoping are two separate cognitive processes that are affected by the length of the recall interval. The proportion of forgetting response errors increases as the recall interval increases while the proportion of telescoping errors decreases with elapsed time. lithe effects of the two types of errors will be approximately equal, then, for one particular reference period length . So far, the full design record-check results do not contradict this formulation if one assumes that the omission and telescoping tendencies have approximately balanced out to create a net reporting bias close to zero in all of the full design studies examined. In addition, Cannell et al. (1977) cite several AC design studies that show an apparent increase in omissions with an increase in e, apsed time.

OCR for page 71
144 But is this another methodological artifact or are we, indeed, observing half of the compensating phenomenon suggested by Sudman and Bradburn? If, for example, we could find AB design studies that show o~rerreporting rates inversely proportional to elapsed time, the compensating forgetting and telescoping explanation would receive support in the health survey context. Unfortunately, none of the AB design record-check studies have published elapsed time data. Feather ' ~ ABC design study (Feather, ~ 972 ~ does not provide elapsed time data for both hospital stay underreporting and overreporting. I have adapted there data in Table 8, using the record version of date of admission (where possible) to make the elapsed time classification. Since date of admission was asked only for the most recent (or only) hospital stay, we must confine the analysis to these stays (somewhere between 75 and 80 percent of all reported and of all recorded stays). The overreporting and underreporting trends in Table ~ do not support the Sudman and Bradburn model. Both overreporting and underreporting increase with elapsed time. Although not shown, the underreporting trends are similar to those cited by Cannell et al. ~ ~ 977), suggesting that the reporting dynamics in Feather's research are similar to those operating in the other record-check surveys that ask about hospital stays. Thus, a different error model may be needed for reporting of hospital stays and physician visits. We do observe, in the Feather data, an increase in both kinds of response errors with greater elapsed times between the survey and the event occurrence. It in thin phenomenon (that Rome survey methodologists label random response error or simple response variance) that cognitive science might address. For example, is this apparent increase in "carelessness" with elapsed time characteristic of reporting of other kinds of events? Is it due to problems recalling the relevant event attributes correctly? Are the reporting problems due to faulty demigods of the respondent about whether to report a recalled event or in the retrieval of incorrect information about the event followed by a logically correct decision about reporting? Understanding these kinds of phenomena is not going to be gained by traditional survey evaluation methods such as record checks. They need the kind of creative hypothesis formulation and carefully controlled laboratory testing that cognitive science can provide. 1

OCR for page 71
145 A: C, s C' 1 o c) A: · - 3 ·rl ~0 o lo; U) 00 o :r: a) a: C) A: o C) Cal m e En V) 0 o ~ a) c) ~ 0 o 0 c) ~ a: :D c) · - o, D4 O a, ~ 0— :: _ o 3 Z 00 S :^ 00 O O a. ~ 0\ TO _ a' up :^ :3 o C' ~ - ~ - - s b~ 0 ~ ~s ~ ) oq , ~: - ~l ~: ~ 0 0 m C~ C) E~ :^ 3 U. a, SC~ a' _ 1 1 — a' CM S :~ D 0 ~: a a, o C, C~ o' C) S C' C) :~ 0 `: ~ D V o C) o C) a, 3 0 0 S C} E~ o a. 0 :~ :, a, S. o a. 0 o 0 S S O o~ a) :5 a a. c~ ~ ~ ct - : ·- ;4 ~ o ) o ~ z ~ o o.

OCR for page 71
146 References Andersen, R., end Anderson, 0. 967 I. Chicago: University of Chicago Press. Andersen, R., Gravity, J., and Anderson 0., ens. 1975 ~_~ Poliov. Cambridge, Mass.: Ballinger. Andersen, R., Rasper, J., and Franicel, M. 1979 I. San Francisco: ~Jossey-Bass. Balamuth, E., Shapiro, S., and Densen, P.~. 1961 Health interview responses compared with medical records. _, Publication No. 584-D5, Ser. D, No. 5. Washington, D.C.: 0.S. Public Health Service. Barlow, R., Morgan, J., and Wirick, G. 1960 A study of validity in reporting medical care in Michigan. Pp . 54-65 in ~ _ Washington, D.C.: American Statistical Association. Cartwright, 1963 Feather, J. 1972 Belloc, N. B . 1954 Validation of morbidity survey data by comparison with hospital records. ~ _ ~ 49: 832-846. Cannell, C. F., and Fowler, F. J. 1963 Comparison of hospitalization reporting in three survey procedures. ~ Ser. D., No. 8. Washington, D. C.: O. S. Public Health Service. Reprinted in Vi tat am ~ M I, Ser. 2, No. 8, July 1965. Cannell, C,F., Fisher, G., and Bakker, T. 1961 Reporting of hospitalization in the health interview survey. I, Ser. D., No. 4. Washington, D. C.: U.S. Public Health Service. Reprinted in Statl~i~, Ser. 2, No. 6, July. 1965. Cannell, C., ~guis, R., and Laurent, A. 1977 A sugary of studies of interviewing methodology. Vital and I, DH~1 Publication No. (ERA) 77-134B, Ser. 2, No. 69. Washington, D. C.: U. S. Government Printing Of.tice. A. Memory errors in morbidity survey. = 0~ 41:5-24. A Response/Record Discrepancy Study. 0ni~rersi~cy of Saskatchewan, Saskatoon, November 1972. Available from the National Technical Information Service, Springfield, Virginia. Kirchner, C., Lerner, R. C., and Cla~rery, 0. 1969 The reported use of. medical care sources by low-in come inpatients and outpatients. I_ 84: 107-1 17.

OCR for page 71
147 Lowenstein, R. 1969 Two Approaches to Health Interview Surveys. School of Public Health and Administrative Medicine, Columbia University. Madow, W. G. 973 Net differences in interview data on chronic conditions and information derived from medical records. ~ Row Stating as, DHE~ Publication No. (BRA) 75-1331, Ser. 2, No. 57. Washington, D. C.: U. S. Department of Health, Education and Welfare. Marquis, R . H. 1 97Ba Inferring health interview response bias from imperfect record checks. Pp. 265-270 in _9 on Survey Research Methods. Washington, D. C.: American Statistical Association. 1978b RYecord Check Y~iditY_of Rev Responses ,,,, , __ , _~.- _ _ A Reassessment 1980 _, R-23 1 9-HEN. Santa Monica, Calit.: The Rand Corporation. t~ m~ 6~ ~~ i, R-2555-HE-W. Santa Monica, Calit.: The Rand Corporation. _. Marquis, R., et al. 1979 ~ ~_, N-1 152-HhW. Santa Monica, Calif.: The Rand Corporation. Marquis, R.R., et al. 1981 ~ _' R-2710/2-HHS. Calit.: The Rand Corporation. Santa Monica, Net er, J., Maynes, F. S., and Ramanathan, R. 1965 The effect of mismatching on the measurement of response errors. 60: 1005-1027. Sudman, S., and Bradburn, N.M. 1973 Effects of. time and memory factors on responses in surveys. ~ _ 68:805-~15. Sudman, S., Wallace, W., and Ferber, R. 1974 The Cost-Effectiveness of Using the Diary an an Instrument for Collecting Health Data in Household Surveys. Survey Research Laboratory, University of Illinois.

OCR for page 71