National Academies Press: OpenBook
« Previous: 4 Approaches to Improving Survey Response
Suggested Citation:"5 Research Agenda." National Research Council. 2013. Nonresponse in Social Science Surveys: A Research Agenda. Washington, DC: The National Academies Press. doi: 10.17226/18293.
×

5

Research Agenda

This report documents the rich set of extant findings about the causes, consequences, and remedies for the general decline in survey response throughout the developed world. This decline represents a growing threat to the quality of social science data in the United States and elsewhere. This report also identifies a number of gaps in that knowledge and promising paths to advance the state of the science and develop more effective remedies.

In the various sections of this report, the panel has recommended research on a long list of topics. These topics fall into three broad categories: (1) research that would deepen our understanding of the phenomenon of nonresponse and the causes of the decline in response rates over the past few decades; (2) research aimed at clarifying the consequences of nonresponse; and (3) research designed to improve our tools for boosting response rates or more effectively compensating for the effects of nonresponse statistically. The panel thus supports a series of research programs and projects that are brought together here. We believe that, together, these topics constitute a comprehensive and multifaceted research agenda.

The panel is aware that, in these times of increasingly limited human and financial resources for the social science survey community, a research agenda must reflect both costs and benefits. Where possible, priorities have been suggested in the report itself.

The panel does not attempt to assign responsibility for these research items among the various players that make up the social science survey community and the research and academic institutions that support that community. Much of the path-breaking basic research, which often does

Suggested Citation:"5 Research Agenda." National Research Council. 2013. Nonresponse in Social Science Surveys: A Research Agenda. Washington, DC: The National Academies Press. doi: 10.17226/18293.
×

not require significant investment of resources for testing and development, will come from academic and other research centers. Large data collection and analytical organizations in the private sector and in government would be responsible for conducting the research that requires new data collection, such as research on interviewer and mode effects. Organizations that provide a platform for integrating the research work generated in these various venues, such as the AAPOR, the American Statistical Association, the International Statistical Institute, and, within the federal government, the Federal Committee on Statistical Methodology, would also have important roles to play.

RESEARCH ON THE PROBLEM

The first set of research topics would help further define the problem, develop appropriate measures, and deepen understanding of the scope, causes, and extent of the problem:

• Research to identify the person-level and societal variables responsible for the downward trend in response rates. These variables could include changes in technology, in communication patterns, and in methods of collecting survey data.

• Research on people’s general attitudes toward surveys and on whether these have changed over time.

• Research about why people take part in surveys and the factors that motivate them to participate.

• Research on the factors affecting contact and cooperation rates.

• Research on the nature (mode of contact, content) of the contacts that people receive over the course of a survey based on data captured in the survey process.

• Research on the overall level of burden from survey requests and on the role of that burden in the decision to participate in a specific survey.

In considering burden, it is important to conduct basic research on the dimensions of response burden and how they should be operationalized. It would be useful to consider factors (e.g., time, cognitive difficulty, or invasiveness) that may determine how potential respondents assess the burden involved in taking part in a survey. These research paths should lead to more practical consideration of how interviewers, advance letters, or other explanatory or motivational material could effectively alter perceptions about the likely burden of a survey.

Suggested Citation:"5 Research Agenda." National Research Council. 2013. Nonresponse in Social Science Surveys: A Research Agenda. Washington, DC: The National Academies Press. doi: 10.17226/18293.
×

RESEARCH ON CONSEQUENCES

The second set of topics concerns statistical and other tools for understanding and mitigating the consequences of nonresponse:

• Research on the cost implications of nonresponse and on how to capture cost data in a standardized way.

• Research on the relationship between nonresponse rates and nonresponse bias and on the variables that determine when such a relationship is likely.

• Research to examine unit and item nonresponse bias and to develop models of the relationship between nonresponse rates and bias.

• Research on the theoretical limits of what nonresponse adjustments can achieve, given low correlations with survey variables, and on the effects of measurement errors, missing data, and other problems with the covariates.

• Research on the impact of nonresponse reduction on other error sources, such as measurement error.

• Research to quantify the role that nonresponse error plays as a component of total survey error.

• Research on the differential effects of incentives offered to respondents (and interviewers) and the extent to which incentives affect nonresponse bias.

The panel notes that research activities designed to expand knowledge of the relationship between response rates and nonresponse bias are likely to assist in the development of a much needed theory of nonresponse bias. A more comprehensive theory not only would further our understanding of the relationship between response rates and nonresponse bias but also would aid in the development of adjustment techniques to deal with bias under different circumstances.

RESEARCH ON COPING

The third set of research topics concerns methods for coping with nonresponse:

• Research to establish empirically the cost–error trade-offs in the use of incentives and other tools to reduce nonresponse.

• Research on and development of new indicators for the impact of nonresponse, including application of the alternative indicators to real surveys to determine how well the indicators work.

Suggested Citation:"5 Research Agenda." National Research Council. 2013. Nonresponse in Social Science Surveys: A Research Agenda. Washington, DC: The National Academies Press. doi: 10.17226/18293.
×

• Research on understanding mode effects, including the impact of mode on reliability and validity.

• Research leading to the development of minimal standards for call records and similar data in order to improve the management of data collection, increase response rates, and reduce nonresponse errors.

• Research on the structure and content of interviewer training as well as on the value of continued coaching of interviewers. Where possible, experiments should be supported to identify the most effective techniques.

• Research to improve the modeling of responses as well as to improve the methods to determine whether data are missing at random.

• Research on the use of auxiliary data for weighting adjustments, including whether weighting can make estimates worse (i.e., increase bias) and whether traditional weighting approaches inflate the variance of the estimates.

• Research to assist in understanding the impact of adjustment procedures on estimates other than means, proportions, and totals.

• Research on the impact that reduction of survey nonresponse would have on other error sources, such as measurement error.

• Research on how to best make a switch from the telephone survey mode (and frame) to mail, including how to ensure that the right person completes a mail survey.

• Research on the theory and practice of responsive design, including its effects on nonresponse bias, information requirements for its implementation, types of surveys for which it is most appropriate, and variance implications.

RESEARCH ON ALTERNATIVES

Finally, the panel recognizes the need to explore alternatives to traditional survey data collection. There are increasing suggestions that administrative data and Internet “scraping” can produce data that could substitute for surveys. The panel suggests that further research is needed to ascertain the quality of data gleaned from these sources, and makes two final recommendations:

• Research into the availability, quality, and application of administrative records to augment (or replace) survey data collections.

• Research to determine the capability of information gathered by mining the Internet to augment (or replace) official survey statistics.

Suggested Citation:"5 Research Agenda." National Research Council. 2013. Nonresponse in Social Science Surveys: A Research Agenda. Washington, DC: The National Academies Press. doi: 10.17226/18293.
×
Page 101
Suggested Citation:"5 Research Agenda." National Research Council. 2013. Nonresponse in Social Science Surveys: A Research Agenda. Washington, DC: The National Academies Press. doi: 10.17226/18293.
×
Page 102
Suggested Citation:"5 Research Agenda." National Research Council. 2013. Nonresponse in Social Science Surveys: A Research Agenda. Washington, DC: The National Academies Press. doi: 10.17226/18293.
×
Page 103
Suggested Citation:"5 Research Agenda." National Research Council. 2013. Nonresponse in Social Science Surveys: A Research Agenda. Washington, DC: The National Academies Press. doi: 10.17226/18293.
×
Page 104
Next: References and Selected Bibliography »
Nonresponse in Social Science Surveys: A Research Agenda Get This Book
×
Buy Paperback | $42.00 Buy Ebook | $33.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

For many household surveys in the United States, responses rates have been steadily declining for at least the past two decades. A similar decline in survey response can be observed in all wealthy countries. Efforts to raise response rates have used such strategies as monetary incentives or repeated attempts to contact sample members and obtain completed interviews, but these strategies increase the costs of surveys. This review addresses the core issues regarding survey nonresponse. It considers why response rates are declining and what that means for the accuracy of survey results. These trends are of particular concern for the social science community, which is heavily invested in obtaining information from household surveys. The evidence to date makes it apparent that current trends in nonresponse, if not arrested, threaten to undermine the potential of household surveys to elicit information that assists in understanding social and economic issues. The trends also threaten to weaken the validity of inferences drawn from estimates based on those surveys. High nonresponse rates create the potential or risk for bias in estimates and affect survey design, data collection, estimation, and analysis.

The survey community is painfully aware of these trends and has responded aggressively to these threats. The interview modes employed by surveys in the public and private sectors have proliferated as new technologies and methods have emerged and matured. To the traditional trio of mail, telephone, and face-to-face surveys have been added interactive voice response (IVR), audio computer-assisted self-interviewing (ACASI), web surveys, and a number of hybrid methods. Similarly, a growing research agenda has emerged in the past decade or so focused on seeking solutions to various aspects of the problem of survey nonresponse; the potential solutions that have been considered range from better training and deployment of interviewers to more use of incentives, better use of the information collected in the data collection, and increased use of auxiliary information from other sources in survey design and data collection. Nonresponse in Social Science Surveys: A Research Agenda also documents the increased use of information collected in the survey process in nonresponse adjustment.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!