Click for next page ( 26


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 25
CHAPTER 2 AFTER THE SEMINAR Chapter ~ summarized the views expressed by the CASM project participants at the St. Michaels and Baltimore meetings, with emphasis on their suggestions for cro~s-disciplinary research by cognitive scientists and survey researchers. This chapter describes some of the outcomes of the project, including research plans and activities developed by partici- pants after the St. Michaels meeting and dissemination of project results through the publication or presentation of papers and other means. Each of the first six sections of this chapter describes a research program or activity initiated by CASM participants working as individuals or in small groups. The first four sections describe plans for rather substantial research efforts. The first section describes a multiyear collaborative research program involving cognitive scientists and survey researchers . The plan for this program, which is already under way, was developed by CASM participants Sirken and Fuch~berg for the National Center for Health Statistics (NCHS ~ . The program, described in the second section was developed by CASM participants Tourangeau, Salter, D'Andrade, and Bradburn, along with other cognitive scientists. The program, whose objective is to study the cognitive underpinnings of the survey interview process, is also funded and under way. The third and fourth sections contain prospectuses for survey collections of data that would be of considerable interest to cognitive scientists. Converse and Schuman propose to investigate personal interpretation of recent historical events for a sample of the U.S. population; funding for this project is expected soon. Tulving and Press present a proposal for a national memory inventory in which memory capabilities and other cognitive abilities would be tested for a large probability sample of the U.S. population; although the authors are not now in a position to pursue their proposal, they welcome and would cooperate with efforts by others to undertake the proposed research. The fifth and sixth sections describe research done by students under the direction of Loftus and Ross, two of the cognitive scientists who participated in the CASM project. The last section of. this chapter describes outreach activities: steps taken by the CASM participants to share the ideas developed during and after the seminar with others and to recruit new members of the interdisciplinary network that has been established. 25

OCR for page 25
LABORATORY-BASED RESEARCH ON THE COGNITIVE ASPECTS OF SURVEY METHODOLOGY National Center for Health Statistics (Monroe Sirken and Robert Fuchaberg) The research project outlined in this plan uses the National Health Interview Survey (NHIS) as a test bed for research and experimentation of the sort discussed at the CASH seminar. Purpose and Objectives Questionnaire design and data collection procedures are among the weakest links in the survey measurement process, and past efforts to improve the quality of survey instruments and procedures have posed serious and difficult methodological problems that are unlikely to be recolored by traditional survey research methods. Therefore, it is essential to test nontraditional modes for conducting research on survey methods. The objective of this project is to investigate the cognitive laboratory as the setting for conducting research on the cognitive aspects of. survey methodology. It will tackle three of the most important questions to emerge from CASH. Namely, under what conditions are laboratory methods 1 ikely to: (~) produce results similar to or different from traditional field methods? succeed where traditional methods have failed? enhance the results obtained by traditional methods? Although survey researchers and cognitive scientists are both concerned with the manner in which individuals handle information, their approaches to the problem and the methods used to study the problem are quite different, and there has been very little communication between them. Surrey researchers are concerned about the survey measurement process and use field experiments to test response effects in terms of the wording, response categories, and orderings of. questions. They make very little, it any, use of controlled laboratory experiments to investigate the ma=er in which the respondents and interviewers process the information presented by the survey instrument. The traditions method of developing, testing, and evaluating survey instruments involves sizeable field pretests and pilot studies of questionnaires that are developed by survey statisticians and tested under Formals survey conditions by trained interviewers. Cognitive scientists, on the other hand, are concerned about the system individuals use in processing information. Cognitive psychologists conduct controlled experiments in a laboratory setting involving direct and intensive interaction with relatively small samples of subjects to investigate the mental procedures by which information is processed. A mayor objective of this project is 26

OCR for page 25
27 to contribute to the advancement of both disciplines, and to effect communication between them, including collaboration in research studies. The demonstration will conduct laboratory-based research on the cognitive aspects of survey design using the combined methods of the cognitive and statistical sciences. Cognitive knowledge and techniques will be used to gain a better understanding of the effects of cognitive factors in the survey measurement process. From these laboratory findings statistical models will be developed for controlling survey measurement errors. Benefits The laboratory is the ideal setting for conducting interdisciplinary research in which the combined technologies of the cogniti Ire, social, biological, and computer sciences can be applied in researching the cognitive aspects of survey methodology. Participation of NCHS staff in the interdisciplinary laboratory, as described later in this plan when discussing collaborative arrangements, will help to bridge the gap that currently exists between government agency survey methodologists and university survey researchers and social scientists. There will be potential benefits for all disciplines. The project will provide additional methodologies for researching cognitive issues in surveys, new phenomena to examine in basic research in the cognitive and related sciences, and tested strategies for producing improvements in the methods and statistics of federal statistical surveys in general, and in NHIS, in particular. A note of caution is in order about the potential benefits of this project. It is not expected that the project will produce definitive substantive findings with respect to any cognitive issues, although it may provide important leads for subsequent research. The major emphasis will be methodological rather than substantive. Even so, it is recognized that the methodological findings obtained in thin or any single study will not be conclusive until verified by other researchers in subsequent trials. Collaborative Arrangements This demonstration project will be conducted in a collaborative mode. Since NCHS has neither a cognitive research laboratory nor a staff of cognitive scientists, it will make contractual arrangments with universities to have the experiments conducted in their laboratories and with their scientists. Not only will this arrangement be cost-effective for this project, but it will, as noted earlier, have the major long-term benefit of establishing closer research ties between the federal statistical establishment and universities. The NCHS and university laboratory staffs will collaborate in all research aspects of this project, and in the preparation of research reports, many of which will be suitable for publication in scholarly

OCR for page 25
28 journals. The NCHS will be primarily responsible for the -surrey and statistical methods and the contractor for the cognitive methods. Work Plan Experiments will be conducted to tent the application of laboratory-based methods for two broad types of questionnaire design problems: development of survey instruments (2) investigation of specific cognitive issues The NHIS questionnaire will be used by the laboratory as the survey instrument for both types of experiments. The supplement to the NHIS questionnaire will be used to test the development of survey instruments. Thin part of the questionnaire collects information on specific health topics (child care, health promotion, prescribed medicine, etc.) and changes annually. The specific cognitive issues will be generic to surveys and could arise in either the NHIS supplement or the core of the NHIS questionnaire. The latter collects basic information about the nation's health (health status, utilization of health services, eSc.) and undergoes virtually no change from year to year. The workplans for developing and pretesting a supplement to the NHIS questionnaire and for conducting laboratory research on specific cognitive issues relating to the NHIS questionnaire, respectively, are discussed in the next two sections. These plans were developed within the context of the collaborative mode in which the project will be conducted. On the one hand, these plans are intended not to overly restrict the Center 'a or the subcontractor' creativity as the research progresses. This is a concession to the nature of this research project, and also a major advantage of having the laboratory research conducted outride NCHS. On the other hand, the plans were developed with the view to pursuing certain objectives and producing particular products within a specified time frame. This is a requirement necessary to ensure project accountability, and also a major advantage of having the project administered by NCHS. Survey Instrument Development The Center's schedule of activities to develop and tent the 1987 NHIS supplement and a proposed schedule of the laboratory's activities are presented below.

OCR for page 25
29 Date ~.~e~ 1785 Develop analysis plan 6/85 Complete first draft of supplement 10/85 Prepare pretest version of supplement 12/85 Start 0~3 pretest clearance 3/86 4/86 Conduct field pretest Participate in field pretest 6/86 Prepare pilot study version of supplement 7/86 Start of OMB pilot study clearance 10/86 Conduct pilot study 10/86 Design NHIS supplement 1/87 Start 1987 NHIS Complete first draft of supplement Complete testing of first draft of supplement Complete tenting of pretest version of the supplement ~ contingent on a three- month extension of the laboratory subcontract In accordance with the Center's established timetable for constructing its annual supplements to the NHIS questionnaire, the topic for the 1987 NHIS supplement will be selected during 1984 and the literature search will be completed by January ~ 985, exact' y two years before the NHIS commences. During the two-year period, January 1985-1 987, the Center staff will be engaged in the tightly scheduled set of activities as noted above. These activities exemplify the traditional method of constructing survey instruments. The sine qua non of this method is that the instruments are field tented under conditions that simulate the actual survey conditions as closely as possible. This approach is in sharp contrast to the proposed laboratory activities which would be conducted under controlled laboratory conditions. The schedule of laboratory activities is linked to the NHIS schedule of established activities so as to maximize the laboratory 's potential contributions in developing and testing the ~ 987 NHIS supplement, subject to the condition that these activities should not interfere with nor Jeopardize the basic integrity of the NHIS established testing practices. This phase of the prospect will delineate the potential role of. laboratory-based research in developing and testing survey instruments. The project could result in the development of improved NHIS pretesting protocols, including improved field pretesting methods for training and debriefing interviewers, and ninnovative" laboratory-based methods for conducting unstructured interviews and group interviews. Developing the First Draft of the NHIS Supplement The laboratory will devote the five-month period, January-May 1985, to developing the first draft of the NHIS supplement. During this same period the NCHS staff will be independently developing its own first draft of the NHIS supplement.

OCR for page 25
So In early January 1985, NCHS will provide the laboratory with the items of information that will be collected in the 1987 NHIS supplement, and the contractor will transform these items into survey questions and procedures. Using cognitive techniques, such an protocol analysis, the laboratory will investigate the manner in which respondents process the required information, and on the basin of these findings, draft questions and design its own first draft of the NHIS supplement. Comparing the first drafts of the NHIS supplements that are developed separately by the NCHS and the laboratory will indicate the extent to which the laboratory method is a surrogate for the traditional NHIS method and to what extent it produces different results. Merging what appear to be the best features of both versions of the questionnaire and comparing the combined result with the questionnaire that was developed entirely by traditional methods will provide a basis for assessing the enhancement value, if any, of developing questionnaires in the laboratory as an adjunct to, or in place of, traditional developmental methods. Testing the First Draft of the NHIS Supplement During the six-month period from June 1985 to November 1985, the first drafts of the NHIS supplement will be pretested in the laboratory. Possibly three different versions will be laboratory tested: one that was developed by the NHIS staff, another developed by the laboratory staff, and possibly a third which incorporated what are Judged to be the best features of the other two versions. The laboratory will assess whether the drafts of the NHIS supplement are eliciting the kinds of information they are supposed to elicit. Laboratory testing will be performed on a variety of subjects who will be selected because they are expected to experience different types of cognitive problems with the questionnaire. The criteria for selecting sub jects will depend somewhat on the topic covered by the THIS supplement, but certainly they will reflect demographic, ether and economic differences in the population. The laboratory pretest findings will be discussed with NCH~ staff during November 1985, so that they can be incorporated into an improved draft of the questionnaire that the NHIS staff would be preparing to accompany its request [or OMB clearance in December 1985 to conduct a field pretest in 1986. Testing the Field Pretest Draft of the NHIS Supplement During the four-month period, December 1985 until the subcontract ends in March 1986, the field pretest versions of the NHIS supplements will be pretested in the laboratory and then independently field pretested during April 1986. Pretesting identical drafts of the NHIS supplement by both laboratory and field methods will make it possible to compare and evaluate how well each method assessed whether the NHIS supplement was doing what it is supposed to be doing and, if not, what revisions were needed. Relative costs and turnaround times of conducting pretests by each method would also be compared.

OCR for page 25
31 Specific Cognitive Issues As noted earlier, the manner in which people handle information is of common interest to cognitive and survey scientists, but objectives and methods of the two sciences are quite different. Survey scientists conduct field experiments to evaluate the quality of responses elicited by survey instruments. Cognitive scientists, on the other hand, establish generalizations about the mental systems people use for processing information by conducting laboratory experiments. The mission of this project is to demonstrate the enhancement value to both scientific fields of conducting laboratory research on particular cognitive issues that have been implicated by survey scientists as adversely affecting the quality of survey responses. Cross-fertilization of the two scientific fields is the keynote of this pro ject. Cognitive issues that arise in surveys are representative of. wider classed of cognitive phenomena that are being studied in cognitive science, but under restricted and unnatural laboratory conditions. Therefore, it is believed that bringing the survey cognitive issues and the survey experience with there issues into the cognitive laboratory will generate ideas for basic and applied research in cognitive science. And feeding the laboratory research findings on these cognitive issues back to survey scientists will, in turn, --simulate the development of improved statistical models of survey measurement errors and improved methods of constructing survey instruments. For cognitive science, the ultimate benefit will be a better understanding of the way people process information, and for survey science, it will be improved control over the cognitive component of survey measurement. Three well-known, but largely unresolved, survey problems are presented as possible candidates for laboratory research. They are: telescoping, conditioning, and the respondents ' perceptions of the confidentiality of their responses. Each problem involves cognitive issues that are poorly understood and, as will be explained later, seem to present interesting material for laboratory research. For example, for unknown reasons the effects of conditioning and telescoping are asymmetric. The conditioning effects of ordering questions or response categories are often more pronounced when ordered one, rather than another, way. Similarly, the telescoping effects of allocating events either to earlier or later periods than those in which they actually occurred usually results in inaccurately allocating fewer events to lens recent than to more recent periods. Telescoping Failure of respondents to recall events and to recall correctly when the events occurred are mayor sources of error in the collection of survey data. The errors associated with these two cognitive sources are often confounded in surveys, which may help to explain why the classical negative accelerated forgetting curve predicted by cognitive science does not necessarily hold in surveys.

OCR for page 25
32 The tendency of survey respondents to allocate events either to earlier or later periods than those in which the events actually occurred is called telescoping. Typically, survey respondents report retrospectively about events that occurred during a reference period, which is a calendar period of specified length that precedes the interview date. The telescoping phenomenon has been observed both for reference periods that are bounded by prior interviews and for unbounded reference periods. The findings of survey research indicate that unbounded recall has a net forward telescoping effect, that is, more events are shifted forward in time and erroneously reported in the reference period than are shifted backward and erroneously not reported in the reference period. In addition, events that are correctly placed within the reference period tend to be reported as having occurred more recently than they actually did. The importance of the event, the length of the reference period, and the characteristics of respondents all appear to have an effect on the telescoping phenomenon. With bounded recall there appears to be telescoping within the reference period itself, with the net forward effect being greatest for the most recent part of the reference period. The telescoping phenomenon is representative of a wider class of cognitive phenomena involving temporal judgments. There is no doubt that cognitive scientists appreciate that the process of making temporal judgments is an important component of event memory; however, their understanding of the event-dating process is based primarily on laboratory experiments which involve neither naturally occurring events nor long-term memory. Consequently, it is unclear how well existing cognitive theory on temporal judgment applies to the real world of personal events such as those respondents are asked to recall in surveys. Apparently, the telescoping phenomenon, per se, has not been investigated in the cognitive laboratory, and it is proposed that the survey experience with this phenomenon may offer interest)" leads for designing innovative laboratory experiments. Conditioning All scientific investigations are subject to the risk that the measuring instruments will disturb the phenomenon under observation and thereby affect the accuracy of its measurement. In this broad sense, the conditioning concept in survey science is analogous to Heidelberg 's uncertainty principle in physics, but without the latter ' ~ specificity. Conditioning in survey research usually refers to the distorting effect of the total survey measurement process on survey responses, but in the narrower sense used here it refers more specifically to the response effects of collecting an item or ret of items of information on another item or set of information items. For example, it refers to the response effects of adding one set of questions to another set of questions, such as the effects of NHIS core questions on the questions in the NHIS supplement or vice versa. It also refers to the effects of. ordering a particular set of questions or response categories, or reinterviewing the same respondents, an in panel and quality check

OCR for page 25
33 surveys. The narrower definition is adopted here because it makes the survey conditioning phenomenon more amenable to laboratory experimentation. Although there are many examples of conditioning effects in THIS and other scientific surveys, problems often arise unexpectedly since the phenomenon is not well understood by survey scientists. For example, in his CASH paper, Bradburn refers to a mysterious asymmetric effect of question and response ordering. He notes that a different effect is observed when questions are ordered in one way than when they are ordered in another way. It is proposed that this curious effect of conditioning in surveys may offer leads for designing critical laboratory experiments on the conditioning effects of rotating the order in which the material is presented. Perceptions of Confidentiality The response effects of asking for information about sensitive topics is a major survey concern because (1) policy makers and other users of health survey data often require this type of information, and (2) respondents are usually reluctant to provide this information and the quality of the information reported is often suspect. Examples of sensitive topics are: illicit behavior such as drug use, drunk driving, low-esteem behavior such as excessive drinking, overeating, and diseases with social stigma such as cancer, venereal diseases, tuberculosi a, mental illness, etc . Although scientific surveys subscribe to a strict policy of protecting the confidentiality of the reported information, assurances of this policy are often insufficient to overcome the suspicions of respondents that their responses may be disclosed in an identifiable form to third parties or their reluctance to report sock ally undesirable behavior to an interviewer. Survey scientists try to reassure respondents by using data collection techniques that seek to preserve the anonymity of the persons for whom sensitive information is reported in household surveys. Although it seems obvious that the success of these techniques would be greatly affected by the respondents' perceptions of the confidentiality protection afforded by these techniques, their perceptions have not been subjected to in-depth research and hence they are largely unknown. There are three survey techniques often used for preserving respondent anonymity: (1) Randomized response--a respondent is simultaneously presented the sensitive question and another non-sensitive question, each of which can be answered yes or no. He/she answers only one question and he/she alone knows which one, because he/she selected the question to be answered by a random process such as flipping a coin. (2) Network sampling--the respondent serves as an informant for other persons to whom he/she is linked by virtue of kinship, friendship, or some other designated relationship, but who are otherwise unidentified.

OCR for page 25
34 (3) Self-enumeration--the respondent writes the answers to the sensitive questions on a blank sheet of paper that he/she seals in a elf-addressed envelope and mails to the survey organization. The problem of respondent compliance in surveys on sensitive topics is representative of a wider class of cognitive phenomena in which people are faced with the task and the rink of making decisions on the basis of information that they may neither fully comprehend nor believe. Numerous applications of the anonymity techniques in surveys on sensitive topics have produced mixed results that traditional survey research methods have been unable to explain satisfactorily. It is expected that laboratory research on respondents' perceptions of these techniques under varying conditions may improve the design of surveys on sensitive topics and may lead to an improved understanding of the cognitive processes by which people assess risks on the basis of incomplete information.

OCR for page 25
COGNITIVE PROCESSES IN SURVEY RESPONDING: PR W ECT SUMMARIES Roger Tourangeau, Villiam Salter, Roy D'Andrade, Norman Bradburn, and associates Researchers at the National Opinion Research Center (NORC), Yale University, and the University of Chicago have proposed three interrelated research programs to carry out a series of studies on the cognitive underpinnings of the survey interview process. All three projects share a common framework, which is described here briefly. The framework assumes that respondents in surveys proceed through three mayor stages in answering survey questions: they interpret the question, retrieve the relevant information, and formulate a response. It is our shared belief that response effects in surveys can bent be understood by examining these processes in detail. Since different types of questions make different demands at each stage, we have also adopted a simple scheme for classifying survey quest ~ ons. We distinguish three broad classes: questions that elicit attitudes or opinions; questions that ask about behaviors; and questions that concern the causes or reasons for behavior. The framework thus suggests nine areas of. investigation def ined by the three stages of survey responding and the three types of questions. The NORC research program, developed by Roger Tourangeau, Roy D'Andrade, and Norman Bradburn, is entitled Recognitive Processed in Survey Responding: Attitudes and Explanations. ~ It concerns two types of survey questions--those concerning attitudes and reasons--and includes studies on all three stages of survey responding--interpretation, retrieval, and Judgment. The Yale program, developed by Robert Abelson, is entitled "Cognitive Processed in Survey Responding: Multiple Schemas and the Role of Affect. It deals with the same two classes of survey questions as the NORC research program and offers a complementary perspective on some of the same issues explored there. It dovetails with the NORC work in other ways--it extends the analysis of attitudes in terms of cognitive schemata and incorporates studies on the role of affect in survey responses. The University of Chicago program, developed by William Salter, Steven Shevell, Lance Rips, and Norman Bradburn, in entitled Recognitive Processes in Survey Responding: Time and Frequency Estimation.8 It focuses on the remaining class of survey questions, those that concern behavior. It includes studies on the retrieval and Judgment processes and on how these processes interact when respondents must Judge the timing or frequency of events. All three research programs share the interdisciplinary perspective of the Advanced Research Seminar on Cognitive Aspects of Surrey Methodology. The research teams for each project include researchers who have done cognitive research or survey research or work in both fields. In addit ion, all three include a commitment to the replication and extension of laboratory findings to the field setting. Each project incorporates plans for split-ballot studies to be conducted within the context of a national surrey. 35

OCR for page 25
60 References Cralk, P. I. M. 1977 Age differences in human memory. In J.E. Birren and R.W. Schaie, eds., i. New York: Van Nostrand Reinhold. Craik, F.I.M, and Levy, B.A. 1976 The concept of primary memory. In W.~. Estes, ea., Hand~ok ~ , Vol 4. Hillsdale, N.~.: Lawrence Erlbaum Associates. Kelley, H. P. 1964 Memory abilities: a factor analysis. PA o ~aDhs ~ 1. Tulving, E. 1972 Epinodic and semantic memory. In E. Tul~ring and W. Donaldson. eds.. A. New York: Academic 1 983 Press. New York: Oxford Oni~rersity Press. Tulving, E., Schacter, D. L., and Stark, H. A. 1982 Priming effects in word-fragment completion are independent of recognition memory. ~ ~ 8: 336-342. Underwood, B.~., Boruch, R.F., and Mali, R.A. 1978 Composition of episodic memory. ~;_~ ~ 107: 393-419. . Waugh, N. C. ~ and Norman, D. A. 1965 Primary memory. ~ Rim 72:89-104. Wechaler, D. 1945 A standardized memory -scale for clinical use. Psychos on ~ 9 : 87-95.

OCR for page 25
PROTOCOL JlNALYSIS OF RESPONSES TO SURVEY Rl3CILL QUESTIONS Elizabeth Loflus One idea that was received enthusiastically at the St. Michaels seminar wan the suggestion for using protocol analysis to study how survey respondents retrieve information from memory to respond to questions about pant events. The first section of this item, prepared a few weeks after the seminar, expands on this idea and presents some results from five pilot interviews conducted to explore the potential utility of this technique. Subsequently, David Fathi, a student of the author, has conducted research in this Dame area for an honors thesis. The second section of this item describes some results from 23 protocol analyses of responses to two questions; one on use of. health care facilities and one on deposits to a credit account. One of the questions arising in this research was how to relate different methods of retrieval to the validity of responses. Since it proved difficult to obtain verification data for the health and credit deposit questions, some subsequent work by Fathi has related to questions asking students in an undergraduate psychology course to recall the exact dates of examinations given in the course. Experimental ProJect: Protocol Analysis In many national surveys, respondents are asked to recall personal events Prom their lives. For example, in the National Health Interview Survey, respondents are asked, "During the past 12 months, about how many times did (you) see or talk to a medical doctored In the National Crime Surrey, respondents are asked, non the last six months, did anyone beat you up, attack you, or hit you with something, such as a rock or bottle?. Very little is known about the precise strategies for retrieving personal information of thin sort. One method for learning about cognitive strategies is through the use of protocols (Ericsson and Simon, 1980~. In the protocol technique, people are asked to think aloud as they answer specific questions. The verbalizations produced are called protocols, and they can subsequently be transcribed and analyzed. This method has an advantage over the similar technique of asking people after the fact to describe how they arrived at a particular answer or estimate. The ~atter-the-fact. technique has the disadvantage that people often provide reasons or rationalizations for their bobavior that are not the true reasons but rather are strategies that subjects believe should have been appropriate (Nisbett and Ross, 1980~. To explore the feasibility of a protocol analysis approach to the problem of how people retrieve personal experiences of the type required on, say, the National Health Interview Survey, we asked five pilot subjects to think aloud while answering specific questions. We first gave subjects some practice questions so they could gain experience in Verbalizing their thought processes. Those were questions such an, .~n 61

OCR for page 25
62 the last 12 months, have you eaten lobster?. Then we asked some health-related questions and Rome crime victimization questions. For example, we asked subjects, .In the last 12 months, how many times have you gone to a doctor, or a dentist, or a hospital, or utilized any health care specialist or facility?. and, sin the last 12 months, have you been the victim of a crime? Many specific questions can be answered by examining the protocols produced by subjects. For example, one specific question Ad this: Do people answer the health question by starting from the beginning of the 12-month period and moving toward the present (the past to present approach), or do they start from the most recent event and move backward (the present to past approach)? One might predict that respondents would begin with the most recent events, since these might be more "available in memory (Tversk~r and Rahneman, 1973~. While our results must be considered preliminary, they indicate that, to the contrary, the past to present approach is the favored one. For example, one female respondent answered the health question by saying, Let's see . . . six . . . six months ago ~ went to the dentist. Last month I went to the doctor. I think that' s it . ~ It this tendency to pref er the past to present retrieval sequence for the health question were to be documented in a full study, it would suggest that people might be most efficient at retrieving information if prompted to do so by cues that allowed them to start in the past and work toward the present. Of course, this hypothesis would need to be explicitly tested mince we know that simply because most people perform acts in a particular way does not necessarily mean this is the most efficient way to do so. Another specific issue that could be addressed by analysis of the protocols is the extent to which respondents produce new information when asked further questions that relate to ones that were asked earlier. For example, the response of the female quoted above indicated two health related contacts. However, later this respondent was asked, nIn the last 2 months have you been to a dentistry Her answer: Let's see . . . I had my teeth cleaned six months ago, and no . . . and then I had them checked three months ago, and I had a tooth . . . yeah, I had a toothache about Harch . . . yeah. So, yeah, I have. n (Interview conducted in July 1983.) This protocol again indicates a preference for the past-to-present retrieval sequence, but also indicates the production of two additional dentist visits that were not provided earlier to the more general question. Although it is well suspected that additional questions will produce additional instances, it is not known why. Protocols could shed light on this issue. Furthermore, it is not known whether beginning an interview with a general question (e.g., shave you been to a specialist? is the optimal technique. It is possible that after having said, anon to the general questions, subjects may be less likely to search memory in an eSfort to answer the specific question than they might have had they not been asked the general question to begin with. He simply do not know whether this is the case. However, an examination of protocols given to specie to quest ions that either are or are not preceded by general ones would be more informative.

OCR for page 25
63 One interesting observation from the five pilot protocols is the large number of instances in which people change their answer as they are in the midst of speaking. For example, one female subject who was asked the crone question answered: "No, not that I can think of, unless . . . Oh, I had two dollars stolen at work, but that's it. n Another said: "No, I haven't, that I can remember . . . Yes, ~ was--T was thinking about my car, and I had come tapes stolen from my car, in Montlake, about six months ago." We could speculate that if subjects had been responding using a more formal checklist technique in which they simply had to Ray eyed or "no" that these two instances might never have been reported. Under the more leisurely approach provided by the protocol technique, the instances emerged from memory. One question that naturally comes to mind is whether we can improve on current interviewing techniques to take advantage of this possible discovery. For example, if respondents were asked to think for a minute, and then answer the question, would we be able to accomplish the same benefits within the context of the more typical interviewing procedures? In short, many interesting issues can be explored through the use of protocols. Specific hypotheses can be tested concerning how personal information is retrieved by people. Moreover, methods for improving the interview process can be tested in this fashion. Order of Retrieval in Free Recall of Autobiographical Material Six subjects were asked, "In the last 12 months, how many times have you gone to a doctor, or a dentist, or a hospital, or utilized any health care specialist or facility? Three subjects were asked, "In the last 12 months, I'd like you to try and recall all the times that you deposited money in your A La Card account., and for each time, try and give me the date as accurately as possible, n or a slight variation on this question. Seventeen subjects were asked both there questions, with the A La Card question coming first. Subjects were instructed to Think out loud" an they responded, and their remarks were taped and transcribed verbatim, yielding 23 protocols in response to the health care question and 20 in response to the A La Card question. Sixteen of the 23 health care protocols and ~ of the 20 A La Card protocols contained fewer than two instances of the behavior in question, thus yielding no information about order of retrieval. For those protocols containing 2 or more instances of the behavior in question, a "+~ was assigned for each time a subject went from a temporally more distant instance to one more recent, and a In for each tome the subject moved from a more recent instance to one in the more distant past. This was regarded as a rough way to quantify the degree to which subjects tended spontaneously to retrieve these autobiographical memories in one direction or another. MA system under which costs of meals eaten by students are charged against an account to which periodic deposits are made.

OCR for page 25
64 When all the + and - signs were counted up across all the protocols, the results were an follows: Health Care + 7 3 A La Card ~ 23 6 - - When the protocols were clas~ified according to whether the direction of recall was consistently forward (i.e., all +'s), consistently backward (all -'s), not clearly in one direction or another (both +'s and -'s), or there was no information about direction of recall available (fewer than 2 instances of the behavior in question produced), the results were as follows: Health Care: A La Card: all + all - both ~ and - no information 4 2 16 23 all + all - both + and - 3 no information ~ 20 7 2 These results were taken as evidence that, at least in response to these questions, subjects tend to retrieve autobiographical memories in a predominantly past-to-present, or forward, direction. References Ericsson, E.A., and Simon, H.A. 1980 Verbal reports an data. Nisbett, R., and Ross, L. 1980 ~ud~-e~t. Englewood Cliffs, N.J.: Prentice-Hall. T~ersky, A., and Rahneman, D. 1973 Availability: a heuristic for Judging frequency and probability. 0~;~ ~10~ 5:207-232. P~xrholu~al B~x 87:215-251.

OCR for page 25
TROUGHS AND RESEARCH ON ESTIMATES "OUT PAST IND FUTURE BEHAVIOR Lee Ross The Strategy and Tactics of Survey Methodology as They Pertain to Determining Past Actions and Out comes basic issue raised at the St. Michaels coherence concerned the special status of surveys, such as the National Hearth Interview Survey, which focuses primarily on relatively ob~t~ past actions and outcomes rather than opinions, preferences, fears, intentions, or other largely ~ b ~ ive responses. A great deal of the traditional science and art of the survey methodologist has focused on the problem of potential instrument or interviewer bias. Many strategic decisions about instructions, wording of items, and the role of the interviewer are designed to minimize such potential for bias by minimizing the role of both the instrument and interviewer in defining terms, suggesting response strategies, and especially in providing feedback about the responses themselves. While such precautions may be entirely appropriate in the context of political surveying or other attempts to ascertain attitudes, beliefs, or other subjective states of the respondent, they may be less necessary when the responses in question deal with specific concrete past actions and outcomes by the respondent. Furthermore, in attempting to determine the respondent's past actions and outcomes through measurement of his or her recollections or estimates, a rather different set of potential sources of error or bias come into play--i.e., the types of factors with which cognitive psychologists have long been concerned in their study of human memory and Judgment. Cognitive psychologists have identified many factors that impair performance or introduce error in recall or Judgment and, perhaps even more pertinent to present concerns, they have identified factors or strategies that lead to improvement in performance. If such psychologists were confronted with the problem of designing instruments and interviewer protocols for facilitating accurate recur and estimation of past actions and outcomes, ~ suspect that they would worry relatively little about traditional instrument or interviewer effects and a great deal about how to help the respondent remember the events in question or estimate the relevant magnitudes or frequencies associated with such events. They would worry about the impact of the respondents' general theories, scbemas, or expectations on their recall or estimates. They would worry about the effect of the respondents' concerns with self-presentation (and perhaps selt-perception and evaluation as well). Most importantly, perhaps, they would design instruments and procedures to overcome such obstacles through trial and error testing which measured accuracy of recall or esti~^tion--i.~., compared recollections and estimations to direct measures of the actions and outcomes in question. ~ could offer many Radicals suggestions about techniques that could enhance accuracy of recollections or estimates. One might make heavy use of models--i.e., letting the respondent see someone doing a good Job of being systematic and complete, or using specific memory aids or mnemonics 65

OCR for page 25
66 in recalling their medical history, consumer behavior, crime victimization experiences, or whatever topic is the focus of the survey. One might encourage the interviewer to offer suggestions about how to remember or estimate the responses or outcomes in question (and train that interviewer in how to prompt such recalls One might explicitly warn the respondent about common biases or errors. One might encourage the respondent to describe events in his or her own words before attempting to answer specific questionnaire items. Perhaps the most extreme possibility would be furnishing the respondent with informative nanchors~--e. g., mean Judgments, typical responses, or ~base-rates. for people in general or people who are like them in terms of pertinent demographic charact ecliptics . Research on Apart ~ and futures Behavioral Estimates for Self and Other One preliminary piece of research has been undertaken in my laboratory that was prompted by the foregoing comments, even though its direct relevance may not be immediately apparent. It was our thesis that recollection of past events, at leant in cases where Episodic recalls is likely to be imperfect at best, or even nonexistent, is closely akin to other related Judgment tacks. Specifically, we sought to compare estimates of specific past performances to parallel predictions about future response and to compare estimates and predictions about one's .XP responses with parallel estimates and predictions about the responses of a peer whom one knows rather well and has ample opportunity to observe on a day-to-day basis. Ultimately, our concern will be with relative accuracy, and with the relationship between accuracy and confidence, in these four different domains (i.e., set[/other x past/future). Pursuing this concern, however, will demand that we choose responses for which we can independently assess actual behavior to which the relevant estimates or predictions can be compared. This will pose significant methodological hurdles and tax our ingenuity. Undoubtedly, it will also restrict our domain of. inquiry to responses that are normally recorded ~ e. g., checks written, purchases made, books checked out of the library, time logged on computer, long distance telephone calls ~ or at least recordable by an observer (behavior in contrived experimental nettings, perhaps television watching, study the, class attendance, etc.~--domains which may or may not be representative of those that figure in survey concerns and domains and which may or may not be typical in terms of difficulty of. recall or estimation. For now, we have chosen to ignore accuracy per se, and to focus on confidence intervals--i. e., to compare subjects' certainty (or, to be more precise, the range of their uncertainty) about the frequency and magnitude of their own past behaviors or outcomes with their certainty about frequencies for parallel future responses. Furthermore, we compare confidence intermurals regarding self-estimates and predictions with parallel estimates and predictions for other people. In other words, we are comparing estimates or recollections about one's past behavior, about

OCR for page 25
67 which one might have a basis for considerable certainty, with three types of estimate that one would expect to be highly uncertain and heavily Theory basely rather than data based. In our single research effort, Stanford students were asked to make estimates about a variety of pant and future responses either for themselves (self-estimate) or for their roommates (roommate estimates). Items included number of checks written (or to be written) in a 30-day period, money spent in restaurants, hours spent watching television, number of long distance telephone calls, and so forth. For each item they made a best guess and then bracketed that best guess with an upper and lower confidence limit. In [act, they furnished two different confidence limits: 50 percent limits (such that they thought the probability of the upper limit being too low was .25 (or 25 percent) and that the probability of the lower limit being too high was also .25 (or 25 percent) and 80 percent limits ~ such that the probabilities of the limits being too low or too high were each NO percent). They also rated the ease or difficulty of making each estimate, and indicated how "surprisedn they would be if the "right answers was not contained within the confidence limits that had been specified (although these items shall not be dealt with in thin brief report). The research design made use of both within- and between-~ubJect comparisons--with self versus other a between-subject factor and past versus future a within-subJect factor (order of past vs. future was counterbalanced , as was the order of specific items). The results of this pilot effort (see Table 1) can be summarized succinctly. Confidence limits for predictions of the future were only TABLE ~ Relative Width of 50 Percent Confidence Intervals for Estimates of Past Behavior and Predictions of Future Behavior Estimate of Prediction of Past Behavior Future Behavior Combined Judgment about self ~ 00 ~ 3 ~ ~ ~ 5 Judgment about roommate 1 1 1 124 117 Combined ~ 05 ~ 28 Note: All intervals were transformed to reflect magnitude relative to interval for past behavior of self. Means reported summarize results for 1-2 behavioral estimates and predictions. A total of 25 subjects offered confidence intervals for self only (both pant and future behavior) and an equal number offered confidence intervals for roommate only (both past and future behavior).

OCR for page 25
68 modestly wider than confidence limits for estimates about the past regardiens of whether it was the self or ones roommate who was the target. Furthermore, confidence internals about one's roommate were no wider, overall, than confidence levels about oneself; for estimates about the past, the confidence interval was slightly wider for roommate than self; for predictions about the future, the reverse wan true. None of theme main effects or interaction effec~c^, moreover, appear to be statistically significant (although this may change somewhat when data from a second cohort of subjects in added to our analysis. It is particularly noteworthy that the confidence intervals for ostensively data-based estimates about one's pant behavior were only marginally narrower than for the ostensively theory-based estimates about the future responses of one's roommates. Such data certainly prompt one to wonder exactly how data-based estimates about ones past really are! They also encourage the type of speculation offered earlier--i. e., that the accuracy of any inferences we might want to make about that behavior would be facilitated by procedures that facilitated recall or encouraged more accurate estimation strategies on the part of the respondent. Finally, the data comparing self-est imates and roommate estimates are interesting in their own right, beyond any relevance to concerns of survey methodology. People apparently believe that they can mare as accurate and confident estimates about other people' ~ responses as they can about their own, particularly when it is future rather than past responses that are the subject of such estimates. It is obviously tempting to find out whether such relative immodesty regarding one's social predictions, and modesty regarding self-prediction, is ''ustified ~ Just as it is tempting to find out whether one's ability to predict the future is really as good, and one's estimates about the pant are really as bad, as suggested by the confidence limits offered in our study). The need for follow-up research is e~rident--research in which accuracy, and therefore calibration of such confidence intervals, can be assessed directly.

OCR for page 25
69 OUTREACH ACTIVITIES In the months that have elapsed since the January 1984 meeting in Baltimore, several of the seminar participants have been active in publicizing the activities and outputs of CASH and in seeking to encourage others to work in this exciting cross-disciplinary field. The principal method of outreach has been the presentation and publication of papers, but other means are also being used. On January 26, 1984, Norman Bradburn attended a seminar, Problems of Measuring Behavior,. sponsored by the Economic and Social Research Council in London, England, and presented a paper, Potential contributions of cognitive sciences to survey questionnaire design. ~ In February Bradburn made presentations on the same subject to three organizations in West Germany: the Zentrum fur Umtragen und Methodische Analyze (Z0M4, in Mannheim, the Institut fur Demo-~kopie in Allenabach, and the Max Planck Society in Munich. In July 1984 Roger Tourangeau attended a ZUMA seminar entitled Asocial Information Processing and Surrey Methodology and presented a paper, "Question order and context effects. n At the 39th Annual Conference of. the American Association for Public Opinion Research held on May 17-20, 1984, at Lake Lawn Lodge, Dele~ran, Wisconsin, Judith Tanur chaired a session on contributions of cognitive psychology to survey research that included a paper entitled Ran information processing approach to recall in surveys. by Norman Bradburn and a paper entitled Attitude measurement: a cognitive perspectives by Roger Tourangeau. The discussant was Elizabeth Martin of the Bureau of Social Science Research. At the annual meeting of the American Statistical Association held on August ~ 3- ~ 6, ~ 984, in Philadelphia, Pennsylvania, a session on cognitive aspects of survey methodology was sponsored by the Section on Survey Research Methods and cosponsored by the Social Statistics and Statistical Education Sect ions . Organized and chaired by Judith Tanur, the session included a paper by Roger Tourangeau entitled Interchanges between cognitive science and survey methodology and a paper authored by David C. Fathi, Jonathan W. Schooler, and Elizabeth F. Lortus, presented by Elizabeth Loftus, entitled Moving survey problems into the cognitive psychology laboratory. The discussants were two of the CASH guests at St. Michael~, Dr. Jacob J. Feldman of the National Center for Health Statistics and Professor Phillip J. Stone of. Harvard University. A paper entitled Recognitive psychology meets the national survey. by Elizabeth F. Lofts, Stephen E. Fienberg, and Judith M. Tanur has been prepared in response to an invitation from the ~rtr~ and is expected to appear ~ n December 1984. The editor of the ~ _~ has invited members of the CASH group to prepare papers. A paper entitled nCognitive aspects of health surveys for public information and policy. is being prepared by Stephen E. Fienberg, Elizabeth F. Loftus, and Judith M. Tanur. A companion piece being prepared by Monroe Sirkin and Judith Lessler stems from their laboratory-based research project at the National Center for Health Statistics. The editor of the ~v in inviting several

OCR for page 25
JO Health Statistics. The editor of the ~a~y is inviting several discussants for these papers. In response to a preliminary proposal presented at its board meeting in June 1984, the Social Science Research Council (SSRC) is organizing a working group on cognition and survey research in order to prepare a detailed plan for the activities of a possible SSRC committee that would bear thesame name. Cochaired by Robert Abelson and Judith Tanur, the working group includes Roy D'Andrade, Stephen Fienberg, Robert Groves, Robin Hogarth, Don Kinder, and Elizabeth LofLus. The staff person responsible for this effort at SSRC is Robert Pearson, a guest at CASM's Balt imore meet ing. Theme are the outreach activities that the editors have been able to identify as this report goes to press; there may well be others that have escaped their attention. It seems reasonable to predict, on the basis of. this record, that we can look forward to a continuing round of relevant reports and discussions an further results emerge from the cross-disciplinary research programs and activities genera~ced by the CASH project .