1

The Growing Problem of Nonresponse

The problem of nonresponse in social science surveys in the United States has been succinctly summarized by the sponsor of this study, the Russell Sage Foundation (2010), in a foreword to the foundation’s description of this study:

Household survey responses rates in the United States have been steadily declining for at least the last two decades. A similar decline in survey response can be observed in all wealthy countries, and is particularly high in areas with large numbers of single-parent households, families with young children, workers with long commutes, and high crime rates. Efforts to raise response rates have used monetary incentives or repetitive attempts to obtain completed interviews, but these strategies increase the costs of surveys and are often unsuccessful. Why is response declining and how does it increase the possibility of inaccurate survey results? Most important, with the advancement of reliable social science research at stake, what are effective methods for increasing the response rate for household surveys?

In examining the extent and effect of nonresponse in social science surveys and the steps that can be taken to reduce the effects of nonresponse, an overriding concern is that the validity of social science studies and their conclusions is indeed at stake. Apparent trends in nonresponse, if not arrested, are likely to weaken the validity of inferences drawn from estimates based on household surveys and undermine, perhaps fatally, their potential to elicit information that assists in understanding social and economic issues. There has always been incomplete acceptance that social science is science. Doubts about the validity of data from surveys give ammunition



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 7
1 The Growing Problem of Nonresponse T he problem of nonresponse in social science surveys in the United States has been succinctly summarized by the sponsor of this study, the Russell Sage Foundation (2010), in a foreword to the founda- tion’s description of this study: Household survey responses rates in the United States have been steadily declining for at least the last two decades. A similar decline in survey re- sponse can be observed in all wealthy countries, and is particularly high in areas with large numbers of single-parent households, families with young children, workers with long commutes, and high crime rates. Efforts to raise response rates have used monetary incentives or repetitive attempts to obtain completed interviews, but these strategies increase the costs of sur- veys and are often unsuccessful. Why is response declining and how does it increase the possibility of inaccurate survey results? Most important, with the advancement of reliable social science research at stake, what are effective methods for increasing the response rate for household surveys? In examining the extent and effect of nonresponse in social science sur- veys and the steps that can be taken to reduce the effects of nonresponse, an overriding concern is that the validity of social science studies and their conclusions is indeed at stake. Apparent trends in nonresponse, if not ar- rested, are likely to weaken the validity of inferences drawn from estimates based on household surveys and undermine, perhaps fatally, their potential to elicit information that assists in understanding social and economic is- sues. There has always been incomplete acceptance that social science is science. Doubts about the validity of data from surveys give ammunition 7

OCR for page 7
8 NONRESPONSE IN SOCIAL SCIENCE SURVEYS to critics of social science and other skeptics reluctant to accept the conclu- sions of social scientific investigations. Moreover, these trends undermine researchers’ (and the public’s) confi- dence in the use of the survey as a tool in the development of public policy responses to social problems. This growing lack of confidence stands in stark contrast to the recent trends of very rapid growth in the use of surveys in social science and in the variety of methods used to conduct surveys. In documenting trends in government-sponsored social science survey research over a recent two-decade span, Presser and McCulloch (2011) found that the number of surveys increased by more than 50 percent over the period 1984 to 2004 and that the number of respondents approved for studies cleared by the Office of Management and Budget (OMB) increased by more than 300 percent. The authors point out that the relative increase in the number of surveys was many times greater than the relative increase in the size of the general population during the same time frame. The modes employed by surveys in the public and private sectors have proliferated as new technologies and methodologies have emerged, but each of these new technologies and processes has its own set of methodological issues and its own nonresponse profile. To the traditional trio of mail, tele- phone, and face-to-face surveys, researchers have added interactive voice response (IVR), audio computer-assisted self-interviewing (ACASI), and Web surveys. In addition, new methods of sampling (such as address-based sampling) have emerged. The proliferation of modes of data collection has been accompanied by a growing public and private research agenda focused on seeking solutions to aspects of the problem of survey nonresponse, rang- ing from better training and deployment of interviewers, to increased use of incentives, to better use of information collected during the survey process, and to increased use of auxiliary information from other sources in survey design and estimation. In this chapter, we set the stage for subsequent discussions of the is- sues surrounding survey nonresponse, its consequences, and proposed approaches to these issues by examining the nature and extent of nonre- sponse in social science surveys. We lay out evidence regarding the trends in nonresponse rates overall and by survey mode, and we consider evidence that increasing levels of nonresponse are an international phenomenon. We also explore some of the reasons for growing nonresponse rates. Finally, we discuss the current state of knowledge about some of the practical implications of growing nonresponse rates for survey costs and survey management. The evidence presented in this chapter leads to several rec- ommendations for a research agenda aimed at filling some of the current gaps in our knowledge concerning trends in nonresponse as well as the reasons for those trends.

OCR for page 7
THE GROWING PROBLEM OF NONRESPONSE 9 CONCEPTUALIZING AND DEFINING NONRESPONSE Great strides have been made over the past three decades in conceptual- izing and defining survey nonresponse, and this work has led to a growing consensus about the root causes and the definition and classification of types of survey nonresponse—a necessary first step in the development of metrics about nonresponse and in coming to grips with the consequences. The decision of a person whether to respond to a survey involves several key factors, and an examination of those factors can be laid out as a conceptual framework for considering survey nonresponse issues. The elaboration of these factors provides a convenient conceptual point of de- parture for this review. A main, overriding factor is a cost-benefit analysis: People respond to surveys when they conclude that the rewards outweigh the costs. Societal factors also enter in. If there is social disorganization or high crime rates, for example, some respondents will be more likely to refuse. There may be differences in the likelihood to respond among different sociodemo- graphic groups, with some more or less prone to take part in surveys. The survey setting also may influence the decision to respond—which in most cases, particularly in a telephone environment, may be a near-instantaneous decision—and interviewers play a key role in determining cooperation, especially in face-to-face surveys. In the larger sense, a growing societal concern over intrusions on people’s time and privacy may have an influence. Attitudes also play a part. The growth in the number of surveys has led to stronger attitudes about surveys in general, apart from attitudes about the topic. The salience of the topic plays a role, and participation may be partly an emotional decision. The mode of data collection may also affect response propensities. And finally, there may be a random component to the decision whether or not to cooperate. These factors are each explored in some depth in this report.1 The advances made toward arriving at a definitional consensus on nonresponse have been largely the result of ongoing work by the American Association for Public Opinion Research (AAPOR), which in the late 1990s began publication of its Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. This volume has been subsequently amended and updated, most recently in 2011.2 The survey research community has, under the leadership of AAPOR, 1  he T conceptual framework that is presented here was drawn largely from the survey methodology field, with the understanding that every field of social science has concepts and frameworks that might shed light on the problem of growing nonresponse, its causes, and its consequences. 2  he AAPOR work in this field was preceded by a project to develop standard definitions T by the Council of American Survey Research Organizations.

OCR for page 7
10 NONRESPONSE IN SOCIAL SCIENCE SURVEYS focused on four operational definitions of rates of survey participation: cooperation rates, contact rates, refusal rates, and response rates. These definitions differ in more than nuance. According to the definitions in the AAPOR Standard Definitions volume referred to above, cooperation rates are the proportion of all cases interviewed of all eligible units ever con- tacted; contact rates are the proportion of all cases in which some house- hold member was actually reached; refusal rates are the proportion of all potentially eligible cases in which the housing unit or the sample member refuses to be interviewed or breaks off the interview;3 and the most familiar rate, the response rate, represents the proportion of complete interviews, with responding units divided by the number of eligible units in the sample (American Association for Public Opinion Research, 2011). In view of the importance of response rate as a descriptor of the quality of a survey, AAPOR provides six related formulas for calculating response rates (see Box 1-1) and also defines different measures for different modes of survey collection. There is now general concurrence with the overall AAPOR framework throughout the social science survey community. Indeed, several journals now require that one of the AAPOR response rates be reported when sur- vey results are presented. Even with the rich variety of AAPOR definitions, however, some survey organizations have had to supplement these defini- tions with ancillary concepts and definitions based on their own needs and buttressed by their own research. Yet, by and large, the definitions have leveled the playing field and have held up well, except in one case: The distinction between contact and cooperation rates has been a source of confusion for some. Calculating household-level contact and cooperation rates requires obtaining information on contact or lack of contact with the household. The distinctions between them are often blurred, and in an era when more and more people are taking steps to limit their accessibility, research is needed on whether the distinction between contact and coopera- tion is still useful to maintain. Recommendation 1-1: Research is needed on the different factors af- fecting contact and cooperation rates. The process of reaching a consensus definition of nonresponse rates is illustrated by the process within the federal government, which spon- 3  Break-offs may occur at several points in the interview. As a rule of thumb, an interview is considered a break-off refusal when less than 50 percent of applicable questions are answered; those with 50–80 percent answered are considered partial responses; and those with 80 percent or more answered are considered completed responses (American Association for Public Opinion Research, 2011, p. 14).

OCR for page 7
THE GROWING PROBLEM OF NONRESPONSE 11 BOX 1-1 What Is a Response Rate? Response Rate 1 (RR1), or the minimum response rate, is the number of com- plete interviews divided by the number of interviews (complete plus partial) plus the number of non-interviews (refusal and break-off plus non-contacts plus others) plus all cases of unknown eligibility. Response Rate 2 (RR2) counts partial interviews as respondents but otherwise is identical to RR1. Response Rate 3 (RR3) estimates e, the proportion of cases with unknown eligibility that are actually eligible. In estimating e, researchers should be guided by the best available scientific information on what share eligible cases make up among the unknown cases and should not select a proportion in order to boost the response rate. The basis for the estimate must be explicitly stated and detailed. Response Rate 4 (RR4) allocates cases of unknown eligibility as in RR3, but also includes partial interviews as respondents as in RR2. Response Rate 5 (RR5) is either a special case of RR3 in that it assumes that e = 0 (i.e., that there are no eligible cases among the cases of unknown eligibility) or the rare case in which there are no cases of unknown eligibility. Response Rate 6 (RR6) makes the same assumption as RR5 and also includes partial interviews as respondents. RR5 and RR6 are appropriate only when it is valid to assume that none of the unknown cases are eligible ones, or when there are no unknown cases. RR6 represents the maximum response rate. SOURCE: Excerpted (without equations) from American Association for Public Opinion Research (2011, pp. 44–45). sors and conducts some of the largest and most significant social science surveys in the United States. The federal government ventured into docu- menting and defining trends in unit nonresponse in reaction to studies con- ducted by the Federal Committee on Statistical Methodology (FCSM) in the 1990s (Johnson et al., 1994; Shettle et al., 1994). These studies found little consistency in how the federal surveys that they reviewed measured and reported nonresponse rates. These inconsistencies were found to have been due mainly to differences in sample design across the surveys. Still, as recently as 2000 it was observed that the official publication of many response rates from government surveys was “fragmented, sporadic, and non-standardized” (Bates et al., 2000). The lack of standardized reporting

OCR for page 7
12 NONRESPONSE IN SOCIAL SCIENCE SURVEYS of response rates for federal surveys was again documented in FCSM Work- ing Paper 31 (Office of Management and Budget, 2001). The comparability issues are exacerbated for international comparisons where refusals are handled differently. OMB has subsequently issued policy guidance to standardize nonre- sponse rate definitions and to set targets for acceptable survey response rates—no easy task since government surveys vary widely in their mandate, sample design, content, interview period, mode, respondent rules, and peri- odicity. The main source of this guidance is the OMB Statistical and Science Policy Office release titled Guidance on Agency Survey and Statistical Infor- mation Collections—Questions and Answers When Designing Surveys for Information Collections (Office of Management and Budget, 2006). This document provides information on the OMB clearance process for surveys and other statistical information collections. Included is information about defining response rates. In summary, the document encourages agencies to use the AAPOR standard formulas in calculating and reporting response rates in clearance submissions to OMB, while permitting agencies to “use other formulas as long as the method used to calculate response rates is documented” (Office of Management and Budget, 2006, p. 57). The existing formal definitions of nonresponse rates take the outcome of a case, response or nonresponse, as given. Although there is relatively little ambiguity about what constitutes response, nonresponse may cover a broad range of possibilities beyond “refusal” and “no contact.” Some nonresponse may reflect refusals that are so adamant that conversion is a virtual impossibility, but in other cases there is a degree of judgment about the utility of following up, perhaps with another interviewer or a spe- cialized refusal convertor in an interviewer-mediated survey. Respondents who are not contacted might not be reached because they are away for a prolonged period, because contact was poorly timed, or simply because contact is random and more attempts would have been needed. It is only relatively recently that it has become possible, with the use of electronic records to quantify attempts on individual cases, to make a broad explora- tion of effort and nonresponse. The availability of such electronic records opens the doors to new avenues of research, and additional work in this area is needed. LONG-TERM TRENDS IN RESPONSE RATES Evidence concerning trends in response rates in general population surveys has been accumulating for over three decades. In an early trend study, Steeh (1981) found that refusals were increasing in U.S. academic surveys conducted between 1952 and 1979. In 1989, Groves summarized the literature and concluded that “participation in social surveys by sample

OCR for page 7
THE GROWING PROBLEM OF NONRESPONSE 13 households appears to be declining in the United States over time.” Later, in the early 1990s, the concern grew to a crescendo with new evidence offered by Bradburn (1992) that response rates were declining and had been declin- ing for some time. It became apparent that this decline in response rates was becoming widespread. This concern was buttressed by the experience of the 1990 Census of Population and Housing, which saw an unexpected and significant decline in mailback responses (Fay et al., 1991). Brehm (1994) found that all sectors of the survey industry—academic, government, busi- ness, and media—were suffering from falling response rates. Groves and Couper (1998) likewise concluded that the decline in response rates was being experienced by government, academic, and commercial surveys. Their summary of data from other countries was based on surveys with long time series and found, for example, that while response to the Canadian Labor Force Survey appeared to be stable over the years, nonresponse appeared to be increasing in Holland, Japan, and Sweden. These repeated surveys are thought to be especially valuable indicators of time trends because their de- signs remain stable across years. Campanelli et al. (1997) also corroborated this trend, but they found that there were some surveys that had managed to maintain their response rates, although at the expense of a larger invest- ment in time and money. This direct evidence was accompanied by a growing literature exhibit- ing concern about the size and effects of nonresponse, evidenced by studies such as National Research Council (1983), Goyder (1987), Lessler and Kalsbeek (1992), Brehm (1993), Groves and Couper (1998), De Leuw and De Heer (2002), Groves et al. (2002), and Stoop (2005). In addition, the Journal of Official Statistics devoted special issues to nonresponse in 1999, 2006, and 2011. The most recent in this series evidenced the growing matu- rity in the field, emphasizing assessment of nonresponse bias and its impact on analysis (Blom and Kreuter, 2011). In her introduction to the 2006 special edition of Public Opinion Quarterly, Singer (2006) cited multiple sources to support the consensus view that nonresponse rates in U.S. household surveys have increased over time. Although not discussed in this paper, household survey nonresponse is a matter of growing concern in many countries.4 4 nternational comparative data on response rates and types of nonresponse in combina- I tion with data on design and collection strategies are scarce. While studies have shown that response rates differ across countries and over time, international data should be interpreted with caution. There are differences in survey design (such as mode of data collection and whether the survey is mandatory) and survey organization and implementation. Comparisons from the European Social Survey, a biennial face-to-face survey of attitudes, opinions, and beliefs in around 30 European countries, finds that, although the target response rate is 70 per- cent, in practice response rates are often lower and vary across countries (Stoop et al., 2010, p. 407). At our panel’s workshop, Lilli Japec of Statistics Sweden reported that nonresponse

OCR for page 7
14 NONRESPONSE IN SOCIAL SCIENCE SURVEYS Based on the above evidence, we are able to conclude that response rates continue on a long-term downward path, but we are concerned that solid evidence about the reasons for the decline is still elusive. Recommendation 1-2: Research is needed to identify the person-level and societal variables responsible for the downward trend in response rates. These variables could include changes in technology and com- munication patterns, which are also reflected in changes in methods of collecting survey data. Recommendation 1-3: Research is needed on people’s general attitudes toward surveys and on whether these have changed over time. Recommendation 1-4: Research is needed about why people take part in surveys and the factors that motivate them to participate. RESPONSE RATE TRENDS IN CROSS-SECTIONAL SURVEYS This section reviews the recent experiences with response rates for sev- eral large and high-visibility repeated cross-sectional surveys. The surveys were selected to be illustrative of some of the more prominent surveys that produce information used in the social sciences. The discussion is abstracted from a paper prepared for this study by Brick and Williams (2013), who conducted an extensive literature review and examined four surveys from the late 1990s to the late 2000s. Two of them were face-to-face surveys— the National Health Interview Survey (NHIS) and the General Social Survey (GSS)—and two were list-assisted random digit dialing (RDD) telephone surveys—the National Household Education Survey (NHES) and the Na- tional Immunization Survey (NIS). We have added discussions of two ad- ditional telephone surveys: the Behavioral Risk Factor Surveillance System (BRFSS) and the Survey of Consumer Attitudes (SCA). We supplement the work of Brick and Williams with a discussion of results of analysis performed by the Census Bureau, the Bureau of Labor Statistics, and the Federal Reserve Board on response rates in the surveys they conduct. in the Swedish Labor Force survey went from 2 percent to nearly 25 percent between 1963 and 2010. Among her findings were that response rates were higher when interviewers found the survey to be very interesting and among interviewers who had a positive attitude toward persuasion. At the sampled person level, there were lower response rates for non-Swedish citizens and persons 64 years of age and older and higher response rates for married persons.

OCR for page 7
THE GROWING PROBLEM OF NONRESPONSE 15 National Health Interview Survey The NHIS samples households as well as one adult and child within each sample household, so the rates shown in Table 1-1 reflect the initial household screening rate as well as the unconditional rate of response for the various survey target groups—the family and one sample adult and one child. The National Center for Health Statistics computes both conditional and unconditional response rates. We use the unconditional response rate (which is equivalent to RR1). The NHIS underwent a major redesign, including changes in the sur- vey questionnaire and moving from paper to computer-assisted personal interviewing (CAPI) administration in 1997. As a result, the response rate series begins with 1997, the first year of the new design. Nonresponse rose sharply at the beginning of this period but then fell slightly before rising again at the end of the period. Table 1-1 shows the unconditional response rates for the period 1997 to 2011. General Social Survey The GSS is a social survey that collects data on demographic char- acteristics and attitudes. It is conducted by NORC at the University of Chicago using face-to-face interviews of a randomly selected sample of non-­ TABLE 1-1  National Health Interview Survey Unconditional Response Rates, 1997–2011 (in percentage) Survey Household Family Sample Child Sample Adult Year Module Module Module Module 1997 91.8 90.3 84.1 80.4 1998 90.0 88.2 82.4 73.9 1999 87.6 86.1 78.2 69.6 2000 88.9 87.3 79.4 72.1 2001 88.9 87.6 80.6 73.8 2002 89.6 88.1 81.3 74.3 2003 89.2 87.9 81.1 74.2 2004 86.9 86.5 79.4 72.5 2005 86.5 86.1 77.5 69.0 2006 87.3 87.0 78.8 70.8 2007 87.1 86.6 76.5 67.8 2008 84.9 84.5 72.3 62.6 2009 82.2 81.6 73.4 65.4 2010 79.5 78.7 70.7 60.8 2011 82.0 81.3 74.6 66.3 SOURCE: National Center for Health Statistics (2012).

OCR for page 7
16 NONRESPONSE IN SOCIAL SCIENCE SURVEYS institutionalized adults (18 years of age and older). The survey was con- ducted annually every year from 1972 to 1994 except in 1979, 1981, and 1992, and since 1994 it has been conducted every other year. The 2010 national sample had 55,087 respondents. Response rates for this survey are reported using the AAPOR RR5 defi- nition. Smith (1995) reports that from 1975 to 1993, the GSS exemplified a survey that did not experience increased nonresponse. After peaking at 82.4 percent in 1993, however, the response rate declined for a number of years. Since 2000, the rate has held steady in the vicinity of 70 percent; it was 70.3 percent for the most recent collection in 2010. Although the rate has held steady, the reasons for the decline in response rates have varied by survey round. Refusals have risen most as the reason for nonresponse over this period (NORC, 2011). National Household Education Survey The NHES is a biannual set of surveys that, except for a screening survey, vary in content from year to year. These surveys are sponsored by the National Center for Education Statistics (NCES) and are conducted via CATI using RDD samples. The survey has changed significantly over the years, and completion and response rates varied in different administrations of NHES. These various versions of the NHES are described in a series of working papers and technical papers from NCES for the period 1999–2007 (Brick et al., 2000; Hagedorn et al., 2004, 2006, 2009; Nolin et al., 2004). In their study of response rate trends for the early years of the survey, Brick et al. (1997) looked at response rates, refusals, and cooperation rates from the four NHES surveys. Zuckerberg (2010) reported screener response rates for 1991–2007. Table 1-2 presents the NHES screener survey completion rates for 1991 through 2007. Overall response rates for NHES screeners have fallen from 81 percent in 1991 to 52.5 percent in 2007. There was a decline of 11 percentage points in response rates overall from the 2005 cycle. NCES undertook a major redesign of the survey to boost the response rates, con- ducting an initial feasibility test in 2009. The test involved a two-phase mail-out/mail-back survey using an address-based sampling (ABS) frame instead of the CATI data collection and RDD sampling, which had been used for NHES from 1991 to 2007. A large-scale field test took place in 2011. The most recent cycle of NHES data collection took place in 2012. (See Chapter 4 for a discussion of the ABS field test.) In their analysis of the response rates for the NHES screening and the conditional adult rate, Brick and Williams (2013, p. 41) concluded that NHES has experienced a much greater increase in nonresponse rates, with an annual increase of 2.3 percentage points per year over the period, than

OCR for page 7
THE GROWING PROBLEM OF NONRESPONSE 17 TABLE 1-2  Number of Completed Interviews and Weighted Screener Response Rates in NHES, by Survey Year and Component, 1991–2007 Number of Screener Response Survey Year Completed Interviews Rate (%) NHES:91 60,322 81.0 NHES:93 63,844 82.1 NHES:95 45,465 73.3 NHES:96 55,838 69.9 NHES:99 55,929 74.1 NHES:01 48,385 67.5 NHES:03 32,049 61.7 NHES:05 58,140 64.2 NHES:07 54,034 52.5 NOTE: NHES = National Household Education Survey. SOURCE: Zuckerberg (2010). either the GSS or the NHIS. Like the NHIS, the linear fit is not very descrip- tive of the pattern of nonresponse even though the R2 is high (R2 = 0.80). If the 2007 data point were excluded, then the annual increase in nonresponse would only be 1.7 percentage points (similar to that reported by Curtin et al. in 2005 for another RDD survey). National Immunization Survey The NIS is a national RDD survey of telephone households with chil- dren who are between 19 and 35 months old. It is sponsored by the Centers for Disease Control and Prevention. The first data collection effort took place in 1994, and the survey has been conducted on a quarterly basis since April of the same year. Each year, 10 million telephone calls are made to identify households with 35,000 age-eligible children (Ballard-LeFauve et al., 2003). Because the population of interest is children, it is necessary to screen households to find out whether they include children who are eligible for the survey. Information on a child’s vaccination history is provided by a household respondent. Since 1995, data have been sought from the child’s medical provider, as household respondents often have problems recalling a child’s vaccination history. Zell et al. (1995) and Battaglia et al. (1997) pointed out that using data from two sources provides a more accurate estimate of vaccination coverage. NIS response rates have fallen fairly consistently over a period of 16 years (see Table 1-3). Brick and Williams (2013, p. 41) noted that the in-

OCR for page 7
THE GROWING PROBLEM OF NONRESPONSE 29 TABLE 1-8  Overall Retention Rates for Each Sample at Each Wave After the First (in percentage) Year(s) of Data Collection Sample 1994 1995/96 1998 2000 2002 2004 2006 2008 HRS 89.4 86.9 86.7 85.4 86.6 86.4 88.6 88.6 AHEAD 93.0 91.4 90.5 90.1 89.4 90.6 90.7 CODA 92.3 91.2 90.1 91.4 90.4 WB 90.9 90.6 87.9 88.1 87.0 EBB 87.7 86.3 Total (by year) 89.4 89.2 88.3 88.0 88.4 87.6 88.9 88.4 NOTE: AHEAD = Asset and Health Dynamics Among the Oldest Old Survey, CODA = Children of the Depression Age Sample, EBB = Early Baby Boomer Survey, HRS = Health and Retirement Study, WB = War Baby Survey. SOURCE: National Institute on Aging (2011b). and 68.7 percent, respectively.9 Table 1-8 reports the retention rates from subsequent waves of the five samples. The baseline (first wave) response rate is based on completed interviews of all individuals deemed eligible for HRS. The wave-specific response rate is based on the number of individuals from whom an interview was sought in that wave. The declines in response rates over time that are apparent in cross-sectional surveys are not apparent here. Most of the interviews in these five samples are conducted by telephone. Exceptions are made if the respondent has health issues or if the house- hold does not have a telephone. Face-to-face interviews were conducted for HRS, WB, and CODA respondents in each sample’s first wave and for AHEAD respondents age 80 years and above. Mode experiments were done on AHEAD respondents in the second and third wave and on HRS respondents in the fourth wave. The mode experiment in the second wave of AHEAD compared face-to-face and telephone interviews. A similar ex- periment was done on HRS respondents in the birth cohort of 1918–1920. These experiments assessed the impact of interview mode on measures of cognitive functioning. The average cognitive scores of telephone-mode ­ respondents and face-to-face mode respondents did not differ significantly in the AHEAD- and HRS-mode experiments (Ofstedal et al., 2005). In a mail-out experiment conducted on the 1998 wave of the HRS, an advance letter was sent, followed by the mail questionnaire and a $20 incentive 9  escriptions of HRS, AHEAD, WB, CODA, and EBB are taken from National Institute D on Aging (2011b).

OCR for page 7
30 NONRESPONSE IN SOCIAL SCIENCE SURVEYS check; the response rate achieved was 84 percent after excluding the exit cases (National Institute on Aging, 2011b). Panel Study of Income Dynamics The PSID is a longitudinal household survey directed by the University of Michigan Survey Research Center. The first round of data collection took place in 1968; the sample consisted of 18,000 individuals living in 5,000 families. From 1968 to 1997, data were collected every year. In 1999 PSID started collecting data biennially. The mode of data collection changed from in-person interview to telephone interview in 1973 to reduce costs. Exceptions are made for households with no telephone or in other circumstances that do not permit a respondent to give a telephone inter- view. The average in-person interview was around one hour long. Despite attempts at streamlining the survey over time, the cumulative response rate dropped from 76 percent in the baseline year, 1968, to 56 percent in 1988 (Hill, 1999). Further changes were made in 1993 when PSID moved from using paper-and-pencil telephone interviews to CATI. CATI has been used for PSID ever since. Table 1-9 shows response rates for different segments of the PSID sample for the 2003 and 2005 waves.10 A response rate of 97 percent is consistently achieved for the core segment and 88 percent for the immigrant segment. McGonagle and Schoeni (2006) documented the factors they be- lieve are responsible for the high response rates in the PSID. These include incentive payments, payments for updating locating information during non-survey years, the use of experienced interviewers, persuasion letters sent to reluctant respondents, informing respondents about upcoming data collection waves, sending thank-you letters to responding households, and training interviewers to handle refusals. Even though response rates are exemplary in the case of the PSID relative to other panel surveys, it does suffer from high cumulative nonresponse rates due to an attrition rate of 50 percent (Cellini et al., 2008). REASONS FOR NONRESPONSE The reasons given by nonrespondents for not taking part in surveys have not changed much over time. A comparison of nonresponse reasons from two U.S. surveys—the 1978 National Medical Care Expenditure Survey (NMCES) and the 2008 NHIS—shows that the reasons reported 10  he response rates are shown for two different family types—those that have remained T stable (non-split-offs) and those in which family members have split off from the basic family unit to form a new family unit between waves.

OCR for page 7
TABLE 1-9  PSID Response Rates and Sample Sizes, 2003–2005 Non-Split-Offs Split-Offs Response Rate Number of Response Rate Number of Total Number Sample (%) Interviews (%) Interviews of Interviews 2003 Actual Core reinterview 97 6,554 83 561 7,115 Core recontact 65 200 56 15 215 Core subtotal 95 6,754 82 576 7,330 Immigrant reinterview 94 459 61 36 495 Immigrant recontact 51 42 43 3 45 Immigrant subtotal 88 501 59 39 540 Total 95 7,255 80 615 7,870 2005 Actual Core reinterview 98 6,756 89 516 7,272 Core recontact 61 194 60 3 197 Core subtotal 96 6,950 88 519 7,469 Immigrant reinterview 93 500 74 45 545 Immigrant recontact 42 27 — 0 27 Immigrant subtotal 88 527 74 45 572 Total 95 7,477 87 564 8,041 SOURCE: McGonagle and Schoeni (2006). Reprinted with permission. 31

OCR for page 7
32 NONRESPONSE IN SOCIAL SCIENCE SURVEYS TABLE 1-10  Reasons for Nonresponse from Two Face-to-Face Surveys, 1978 and 2008, in Priority Order NMCES Reasons for Nonresponse (1978) NHIS Reasons for Nonresponse (2008) Not interested Not interested/does not want to be bothered Unspecified refusal Too busy No time to give Interview takes too much time Poor physical health/mental condition of Breaks appointments respondent Antipathy to surveys in general Survey is voluntary Wants to protect own privacy Privacy concerns Third-party influences respondent to refuse Anti-government concerns Generalized hostility to government Does not understand survey/asks questions about survey Other reasons Survey content does not apply Objects to government invasion of privacy Hang-up/slams door Hostile or threatens interviewer Other household members tell respondent not to participate Talk only to specific household member Family issues NOTE: NHIS = National Health Interview Survey, NMCES = National Medical Care Expen- diture Survey. SOURCE: Brick and Williams (2013, p. 39). and their relative order were similar in the two surveys, indicating that the increase in nonresponse rates over time cannot simply be attributed to a change in subjects’ reasons for not responding (see Table 1-10).11 This sug- gests in turn that there is no simple way of identifying mechanisms for the increase in nonresponse rates over time by examining the reasons given for nonresponse (Brick and Williams, 2013). When the authors examined the relationship between survey nonre- sponse rates and nine selected characteristics that might be expected to influence response rates by affecting accessibility or cooperation, they were able to identify four variables that were highly correlated with nonresponse rates for the four surveys they studied: the percentage of families with chil- dren under the age of six; the percentage of single-person households; the violent crime rate; and travel time to work. However, it is unclear whether these trends just happened to coincide with the increase in nonresponse rates or whether they represent causal factors in that rise. 11  he T data on the NMCES are from Meyers and Oliver (1978) based on non-interview report forms completed by field interviewers and those for the NHIS are from Bates et al. (2008) utilizing automated contact history records of verbal and non-verbal interactions recorded by interviewers during contact with households.

OCR for page 7
THE GROWING PROBLEM OF NONRESPONSE 33 Groves and Couper (1998) and Harris-Kojetin and Tucker (1999) conducted similar time-series analyses. The latter study found that outside influences also played a role. The authors found evidence that changes in presidential approval, consumer sentiments regarding the economy, and the unemployment rate were reliably related to changes in refusal rates in the Current Population Survey. THEORETICAL PERSPECTIVES ON NONRESPONSE Survey methodologists have proposed several theories to explain why people participate in surveys. According to Goyder et al. (2006), the devel- opment of a theory of survey response has centered on several key themes: People respond to surveys when they conclude that the rewards outweigh the costs; societal factors, such as social disorganization or high crime rates, cause some respondents to refuse; different sociodemographic groups are more or less prone to take part in surveys; factors in the survey set- ting may influence a near-instantaneous decision; there is growing concern over intrusions on people’s time and privacy; the growth in the number of surveys has led to stronger attitudes about surveys in general, apart from topic; interviewers play a key role in gaining cooperation, especially in face-to-face surveys; the topic salience plays a role; participation is partly an emotional decision; response propensities can vary by mode of data col- lection; and there is a random component to these decisions. These ideas are captured to one degree or another in three main theories—social capital theory, leverage–saliency theory, and social exchange theory—which are summarized below. Social Capital Theory Social capital theory may help to explain the social and psychological underpinnings of the interpersonal relationships that promote trust and cooperation and thus promote the willingness to respond to surveys. Ac- cording to Robert Putnam, who popularized this theory in his work (1995, 2001), social capital refers to the trust that people gain through productive interaction and that leads to cooperation. The cooperation is manifested in community networks, civic engagement, local civic identity, reciprocity, and trust in the community. Social capital can be measured in the prevalence of community organizations. The decline in association memberships in recent years is associated with less confidence in public institutions. On an individual level, social capital is gained through education, which provides greater networking opportunities (Heyneman, 2000). Lower socio- economic status is associated with reduced levels of trust and cooperative behavior (Letki, 2006). Other individual-level attributes affect the build-up

OCR for page 7
34 NONRESPONSE IN SOCIAL SCIENCE SURVEYS of social capital; in particular, length of residence, home ownership, and the presence of spouses and children strengthen social capital. In considering the effect that social capital has on the likelihood to respond to surveys, Brick and Williams (2013) point out two aspects of the theory that should be considered: Social capital is a collective rather than an individual attribute, and the loss in social capital is partly due to generational change (p. 54). To the extent that nonresponse is generational, for example, it would suggest “exploring predictive models for reasons of nonresponse that are time-lagged and smooth data, rather than relying on the simple models that are considered here” (Brick and Williams, 2013, p. 55). Although social capital theory remains a possible explanation for increasing nonresponse, considerable research must be done to establish a link between the theory and survey nonresponse. Brick and Williams state that they would find “a rigorous investigation of the relationship between social capital and nonresponse rates to be extremely helpful” (Brick and Williams, 2013, p. 55). Leverage–Saliency Theory The leverage–saliency theory (LST) provides a second perspective on survey participation. Groves et al. (2004) introduced the LST of survey participation in 2000. As summarized in Maynard et al. (2010), the LST is a theory of how potential respondents make decisions to participate in a survey interview. The LST posits that people vary in the importance and value they assign to different aspects of a survey request (Groves et al., 2000). For example, for some individuals the topic may be important, for others the reputability of the organization conducting the survey may be significant, and for still others a chance to receive a cash reward may be of consequence. Accord- ing to the theory, the influence of each component of the request depends both on the weight accorded it by a sampled individual (leverage) and on its prominence in the request protocol (saliency). One application of the LST has shown that when the survey topic is a factor in the decision to participate, noncooperation will cause nonresponse error on topic-related questions (Groves et al., 2004). The LST assumes that a potential respondent has an expected util- ity for participating in a survey and agrees to participate if this expected utility is net positive. “Leverage” refers to a potential respondent’s assess­ ments (including valence and weight) of the survey’s attributes that make participation more or less appealing. For example, a cash incentive might have a positive valence and a greater weight as the size of the incen- tive increases; a long interview might have negative valence and a weight

OCR for page 7
THE GROWING PROBLEM OF NONRESPONSE 35 that increases with the length of the interview (Dijkstra and Smit, 2002). Whether an attribute has a positive or a negative leverage varies across sample persons. A specific survey topic may have positive leverage for in- dividuals who are more generally interested in talking about that topic and a negative leverage for those who are not (Groves et al., 2000). Likewise, different respondent groups may react differently. For example, because of different communication experiences people of different ages may be differentially sensitive to the length of a survey or to factors such as direct personal contact. There also may be strong ethnic and racial influences on a person’s assessment of the survey attributes, perhaps related to distrust of mainstream institutions. To the extent that distrust of mainstream institu- tions is intensifying, rising nonresponse rates may be the result. “Saliency”—or salience—refers to the prominence of different attri- butes of the survey request for a sample person who is deciding whether to participate. LST calls attention to the fact that sample persons may base their decisions on only a few attributes of the survey and also to how survey organizations and interviewers provide information to sample persons. A survey might provide a financial incentive and attempt to appeal to a sense of civic duty, for example, but an interviewer might emphasize only one of these aspects, making it more salient and poten- tially more influential as sample persons decide whether to participate. Consequently, requests for participation in a given survey might obtain different responses from the same person, depending on which attributes are made most salient. Box 1-2 indicates how topic saliency can improve response rates. BOX 1-2 How Topic Saliency Improves Response Rates Judging from behavior on the first contact containing the survey introduc- tion, we found that persons cooperated at higher rates to surveys on topics of likely interest to them. The odds of cooperating are roughly 40 percent higher for t ­opics of likely interest than for other topics, based on the four frames utilized in the experiment. Given these results and the deductions from leverage–salience theory, we suspect we could make the 40 percent much higher by making the topic a much more salient aspect of the survey introduction. It is important to note that the overall effects on total response rates of these effects are dampened by non-contact nonresponse, as well as by physical-, mental-, and language-caused nonresponse (Groves et al., 2004, pp. 25–26).

OCR for page 7
36 NONRESPONSE IN SOCIAL SCIENCE SURVEYS Social Exchange Theory Social exchange theory posits that the decision to respond to a survey is based on the assessment of the costs versus the benefits of taking part. Exchanges occur in purest form in economic transactions, and little social relationship is required for successful economic exchanges. Social exchange theory attempts to cover non-economic exchanges, where the costs and ben- efits include such intangibles as the maintenance of tradition, conformity to group norms, and self-esteem. The impetus for an exchange may be as simple as “liking” the inter- viewer by dint of personal interaction or prepaid incentives or as complex as establishing trust. One of the founders of social exchange theory, George Homans presents exchange as overlapping with psychological heuristics. Liking may even lead to a change in opinion in accordance with the liked person’s views (Homans, 1958, p. 602). Trust may be an important component in social exchanges. As Dillman (1978, 1991, 1999) has emphasized, in the survey setting it is important to minimize the costs to the respondents, clearly convey the nature and extent of the benefits, and establish trust so that the respondents are confident about the costs they can expect to incur and the benefits they can expect to receive. As demands on the time of respondents grow, increasing the cost of responding, and distrust of institutions multiplies, the social exchange theory would posit that nonresponse would grow unless perceived benefits of responding also increase. IDENTIFYING COSTS ASSOCIATED WITH APPROACHES TO MINIMIZE NONRESPONSE Costs and response trends have gone hand in hand as key driving fac- tors in the selection of social science survey modes and in the design and conduct of surveys. As the cost of face-to-face data collection grew over the years, survey researchers moved to landline RDD data collection. As the coverage of landline RDD survey designs eroded, survey researchers shifted to dual-frame RDD designs or ABS designs. As the cost of dual- frame and ABS designs rose—and driven in part by the lack of timeliness of the latter—survey researchers were drawn to Web survey designs. It is ­ possible that a significant portion of the downward trend in response rates is attributable to (a) survey budgets not keeping pace with rising costs even with (b) the increasing use of new frames and modes of interview to com- bat declining coverage. Combating declining response rates generally in- creases the cost of the survey, or, as stated quite succinctly in a 2004 study, “[S]urvey costs and survey errors are reflections of each other; increasing one reduces the other” (Groves, 2004, p. 49). The costs are incurred

OCR for page 7
THE GROWING PROBLEM OF NONRESPONSE 37 d ­ irectly in increased interviewer time and also indirectly in the increased management costs associated with approaches such as responsive design, which requires greater resources for developing and using paradata and other information systems. (Responsive design is discussed in Chapter 4.) Peytchev (2009) examines costs in the SCA, a telephone survey where a large part of the cost is interviewer time. The number of call attempts to complete an interview in the SCA doubled between 1979 and 1996, from 3.9 to 7.9 (Curtin et al., 2005); even so, the resulting non-contact rate more than tripled during this same period. Higher proportions of non-contacts and refusals also undoubtedly require more visits and more refusal conver- sion attempts in face-to-face surveys. Unfortunately, there is a dearth of evidence that relates cost to nonre- sponse. In an attempt to rectify this knowledge gap, in conjunction with survey sponsors, the U.S. Census Bureau has launched a major effort to identify costs associated with survey activities.12 Statistics Canada is ex- ploring how responsive design initiatives can be employed to yield cost information that is useful for analysis and decision making.13 Both of these initiatives are in their developmental stages, but they do constitute a serious attempt on the part of these agencies to understand the costs associated with response levels, and both are aided by the collection of data during the normal survey process. Recommendation 1-5: Research is needed to establish, empirically, the cost–error trade-offs in the use of incentives and other tools to reduce nonresponse. It is increasingly recognized that current trends in survey costs are un- sustainable. The fact that, across the Census Bureau, data collection costs have been rising as survey participation has declined led to the cost savings identification effort. Census Bureau task forces were charged with identify- ing the most promising opportunities to improve the cost efficiency of data collection procedures in surveys that the Census Bureau conducts for other agencies under reimbursable agreements. Three broad themes emerged from the task forces: (1) a need for better information on costs; (2) a need for less complex survey management; and (3) a need for continuous and cooperative cost management, especially dur- ing the data collection period. In regard to the first theme, it was pointed out that, to fully understand cost trends in the field, more detailed informa- 12  arbara B O’Hare of the U.S. Census Bureau summarized this work in her presentation to the panel on April 28, 2011. 13  rançois Laflamme of Statistics Canada summarized this work in his presentation to the F panel on April 28, 2011.

OCR for page 7
38 NONRESPONSE IN SOCIAL SCIENCE SURVEYS tion about how field interviewers spend their time is needed. The Census Bureau does not ask interviewers to break out their survey hours, although other survey organizations capture travel and administrative time separately from actual data collection. One method for monitoring costs is through information about the survey process via the collection of paradata (Couper, 1998). The Census Bureau has developed the contact history instrument (CHI), which captures the characteristics of household contact attempts and outcomes; other organizations have their own call record instruments. The information collected is used by field supervisors across organizations to manage cases. Call records, such as the CHI, provide valuable information on the level of effort by case and consequently can be a good proxy for cost. There is a need to go beyond call records alone and to consolidate and systematically monitor other paradata related to costs, such as daily work hours, travel, and case dispositions. When merged with information about nonresponse rates, missing data rates, and key indicator values, costs and data quality can be evaluated simultaneously, and data collection efforts may be man- aged more effectively. Despite these pioneering efforts, there is no common framework for assessing the relationship between cost and response rates and no quanti- tative model of the relationship. A common framework would enable the development of common metrics that would be valid across modes so that comparisons could be made and so that information on costs in an increas- ingly mixed-mode environment can be generated. The literature suggests that some of the cost elements and metrics that would apply to interview surveys would include information on the nature and number of contacts that people receive and on production units, such as days of interviewer project-specific training, numbers of call attempts, the length of the field period, the number of interviewers, and remuneration and incentives—all of which can be captured in paradata collecting and generating systems (see, for example, American Association for Public Opinion Research, 2010a). Recommendation 1-6: Research is needed on the nature (mode of con- tact, content) of the contacts that people receive over the course of a survey based on data captured in the survey process. In the panel’s discussion with field staff of a major ongoing survey, it became obvious that more significant costs are being incurred in the quest for high response rates.14 More expensive senior interviewers are often assigned for difficult refusal cases. Much time is spent in gaining entry to 14  athy C Haggerty and Nina Walker of NORC at the University of Chicago summarized their experiences in a presentation to the panel on February 17, 2011.

OCR for page 7
THE GROWING PROBLEM OF NONRESPONSE 39 gated and otherwise restricted communities in the hopes of gaining an in- terview with a sampled unit. Recommendation 1-7: Research is needed on the cost implications of nonresponse and on how to capture cost data in a standardized way. These “cost” data may be based on proxy variables (such as total in- terviewer hours per case or other measures of effort), but at a minimum it would be useful if these or similar aggregate cost measures were routinely published. Cost information is needed if researchers are to make informed decisions about whether additional efforts to combat nonresponse are worth the cost.