THE 2014 REDESIGN OF THE
SURVEY OF INCOME
AND PROGRAM
PARTICIPATION
Panel on the Review and Evaluation of the 2014 Survey of
Income and Program Participation: Content and Design
Committee on National Statistics
Division of Behavioral and Social Sciences and Education
A Consensus Study Report of
THE NATIONAL ACADEMIES PRESS
Washington, DC
www.nap.edu
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
This activity was supported by contract YA1323-13-CN-0022 between the U.S. Department of Commerce’s Census Bureau and the National Academies of Sciences, Engineering, and Medicine. Support of the work of the Committee on National Statistics is provided by a consortium of federal agencies through a grant from the National Science Foundation, a cooperative agreement from the National Agricultural Statistics Service, and several individual contracts. Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.
International Standard Book Number-13: 978-0-309-46417-8
International Standard Book Number-10: 0-309-46417-X
Digital Object Identifier: https://doi.org/10.17226/24864
Additional copies of this publication are available for sale from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2018 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Academies of Sciences, Engineering, and Medicine. (2018). The 2014 Redesign of the Survey of Income and Program Participation: An Assessment. Washington, DC: The National Academies Press. doi: https://doi.org/10.17226/24864.
The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president.
The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. C. D. Mote, Jr., is president.
The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.
The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.
Learn more about the National Academies of Sciences, Engineering, and Medicine at www.nationalacademies.org.
Consensus Study Reports published by the National Academies of Sciences, Engineering, and Medicine document the evidence-based consensus on the study’s statement of task by an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and the committee’s deliberations. Each report has been subjected to a rigorous and independent peer-review process and it represents the position of the National Academies on the statement of task.
Proceedings published by the National Academies of Sciences, Engineering, and Medicine chronicle the presentations and discussions at a workshop, symposium, or other event convened by the National Academies. The statements and opinions contained in proceedings are those of the participants and are not endorsed by other participants, the planning committee, or the National Academies.
For information about other products and activities of the National Academies, please visit www.nationalacademies.org/about/whatwedo.
PANEL ON THE REVIEW AND EVALUATION OF THE 2014 SURVEY OF INCOME AND PROGRAM PARTICIPATION: CONTENT AND DESIGN
JOHN L. CZAJKA (Chair), Mathematica Policy Research, Inc.
MARIANNE P. BITLER, Department of Economics, University of California, Davis
PETER D. BRANDON, Department of Sociology, University of Albany, SUNY
MICHAEL R. ELLIOTT, Department of Biostatistics, University of Michigan
ANDREW HOUTENVILLE, Department of Economics and Institute on Disability, University of New Hampshire
JOHN ICELAND, Department of Sociology and Criminology, Pennsylvania State University
KOSUKE IMAI, Department of Politics and Center for Statistics and Machine Learning, Princeton University
DANIEL KASPRZYK, NORC at the University of Chicago
H. LUKE SHAEFER, School of Social Work, University of Michigan
ARLOC SHERMAN, Center on Budget and Policy Priorities
KAREN E. SMITH, Urban Institute
TOM W. SMITH, NORC at the University of Chicago
JAMES P. ZILIAK, Department of Economics and Center for Poverty Research, University of Kentucky
CAROL C. HOUSE, Study Director
ADRIANNE BRADFORD, Research Associate
AGNES GASKIN, Administrative Assistant
COMMITTEE ON NATIONAL STATISTICS
ROBERT M. GROVES (Chair), Office of the Provost, Department of Mathematics and Statistics, and Department of Sociology, Georgetown University
FRANCINE BLAU, School of Industrial and Labor Relations, Cornell University
MARY ELLEN BOCK, Department of Statistics, Purdue University (emerita)
ANNE C. CASE, Woodrow Wilson School of Public and International Affairs, Princeton University
MICHAEL CHERNEW, Department of Health Care Policy, Harvard Medical School
JANET CURRIE, Woodrow Wilson School of Public and International Affairs, Princeton University
DONALD DILLMAN, Social and Economic Sciences Research Center, Washington State University
CONSTANTINE GATSONIS, Center for Statistical Sciences, Brown University
JAMES HOUSE, Survey Research Center, Institute for Social Research, University of Michigan
THOMAS MESENBOURG, U.S. Census Bureau (retired)
SARAH NUSSER, Office of the Vice President for Research and Department of Statistics, Iowa State University
COLM O’MUIRCHEARTAIGH, Harris School of Public Policy Studies, University of Chicago
JEROME P. REITER, Department of Statistical Science, Duke University
ROBERTO RIGOBON, Sloan School of Management, Massachusetts Institute of Technology
JUDITH A. SELTZER, Department of Sociology, University of California, Los Angeles
EDWARD SHORTLIFFE, Department of Biomedical Informatics, Columbia University/Arizona State University
BRIAN A. HARRIS-KOJETIN, Director
CONSTANCE F. CITRO, Senior Scholar
Acknowledgments
The Panel on the Review and Evaluation of the 2014 Survey of Income and Program Participation: Content and Design, wishes to acknowledge the support, effort, and expertise of the many individuals who contributed to this report and the deliberations behind it. As chair of the panel, I want to begin by thanking my fellow panel members for the extraordinary efforts they have made over the past 3-1/2 years and for the commitment each gave to this team. The panel members brought a wide range of needed expertise to the table, along with a dedication to good science and to the importance of the data that the Survey of Income and Program Participation (SIPP) provides in support of research into, and evaluation of, the social programs of the United States.
The empirical analysis presented in this report was based on a substantial number of independent tabulations of the 2014 SIPP wave 1 (internal) data and multiple waves of the 2008 SIPP public use data. SIPP is a very complex survey, and the reengineered study database was not fully documented as we did our work. Three individuals deserve special commendation for their extensive work to produce these tabulations. Nathanial (Sasha) Ruby did much of the initial programming of our 2014 panel tables. His work was provided through a contract with the Urban Institute. We express our thanks to both Sasha and the Urban Institute for his contributions. The second individual is Adrianne Bradford, a doctoral student at the Joint Program for Survey Methodology, University of Maryland. Adrianne was hired as a research assistant to listen to recorded interviews and examine potential contributing factors to seam effects in SIPP. When Sasha left the Urban Institute, Adrianne stepped into the role of program-
ming the 2014 tabulations. Adrianne also drafted Chapter 4’s “Handling of Preloaded Data” and Chapter 5’s “Computer Audio-Recorded Interviews.” This report would not have been possible without Adrianne’s skill and perseverance. We are very grateful. The third individual is Karen Smith, one of our panel members. Karen is a long-time technical user of SIPP data. She designed all of the tabulations and produced the voluminous estimates from the 2008 panel and other sources that were based on public use data. She also led the effort to examine the new data structures and variables, decided how to formulate the estimators, oversaw the work done by Sasha and Adrianne, and performed subsequent validation checks with the 2014 public use data. Karen also produced extensive graphics that facilitated the panel’s review of the tabulations.
Panel reports reflect the efforts of all their members, but I want to acknowledge the contributions of those members who authored significant sections of this volume. Michael Elliott drafted “Imputation Methodology” in Chapter 5 and Marianne Bitler drafted “Incentive Experiments in the 2014 Redesigned SIPP” in the same chapter. Andrew Houtenville drafted Chapter 6 on SIPP content, and Tom Smith and Daniel Kasprzyk drafted Chapter 8 on respondent burden. Daniel also made substantial contributions to the panel’s formal response to the report’s independent reviewers (see below), and Arloc Sherman provided several sets of benchmark statistics used in Chapter 7.
The panel wants to express its appreciation to our sponsor, the Census Bureau, and specifically to the SIPP staff at the bureau who facilitated our work. These staff members were under enormous pressure as they fielded a reengineered survey, reinvented the imputation system, and produced several iterations of internal data and a public-use data file. With all of this work, the SIPP staff made time to meet with us, answer our questions, and provide us with important paradata tabulations. They provided workspace for Sasha Ruby and Adrianne Bradford within the secure data facility at Suitland, Maryland, assisted in our access to internal files, and performed disclosure review of our tabulations. We wish to thank in particular Gary Benedetto, Rebecca Chenevert, Cindy Easton, Jason Fields, Graton Gathright, John Hisnanick, Denise Lewis, Matthew Marlay, Anne Ross, Martha Stinson, and Mahdi Sundukchi.
We talked with SIPP data users for both their feedback on how they used SIPP and their initial thoughts on how the new design of SIPP might fit their data needs. These individuals include Chris Chapman (National Center for Education Statistics); Karen Cunnyngham, Randy Rosso, and Bruce Schechter (Mathematica Policy Research); Bob Dalrymple (Food and Nutrition Service); Erin Godtland and Rhiannon Patterson (U.S. Government Accountability Office); Howard Iams (U.S. Social Security Administration); Don Oellerich and Susan Hauan (U.S. Department of Health and Human Services); Erik Scherpf and Constance Newman (Eco-
nomic Research Service); and Stephen Tordella and Nancy Wemmerus (Decision Demographics).
The panel would also like to thank Fred Conrad, research professor at the Institute for Survey Research, University of Michigan. The study of seam effects in surveys is one of his academic interests. He joined the study panel team as a consultant and recommended Adrianne Bradford (one of his students) to us. He met with the panel and helped devise a plan to use recorded interviews to examine possible causes of the seam effects found in SIPP. Unfortunately, the interview recording system used on SIPP was inadequate for carrying out this plan. We want to thank Fred for his input, and particularly, we thank him for recommending Adrianne Bradford to work with the panel.
This Consensus Study Report was reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that will assist the National Academies of Sciences, Engineering, and Medicine in making each published report as sound as possible and to ensure that it meets the institutional standards for quality, objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.
We thank the following individuals for their review of this report: Robert F. Belli, Department of Psychology, University of Nebraska–Lincoln; David M. Betson, Department of Economics, University of Notre Dame; Dan A. Black, Harris School of Public Policy, University of Chicago; John M. Fitzgerald, Department of Economics, Bowdoin College, Brunswick Maine; Randall J. Olsen, Department of Economics, Center for Resource Research, Ohio State University; Andy Peytchev, Center for Survey Methodology, Research Triangle Institute, Research Triangle Park, North Carolina, and Institute for Social Research, University of Michigan; Stanley Presser, Department of Sociology, University of Maryland, College Park; David Ribar, Applied Economic and Social Research, University of Melbourne, Australia; and Daniel H. Weinberg, DHW Consulting, Alexandria, Virginia.
Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations of this report nor did they see the final draft before its release. The review of this report was overseen by Robert A. Moffitt, Department of Economics, Johns Hopkins University, and Charles Manski, Department of Economics, Northwestern University. They were responsible for making certain that an independent examination of this report was carried out in accordance with the standards of the National Academies and that all review comments were carefully considered. Responsibility for the final content rests entirely with the authoring panel and the National Academies.
The panel would like to express its appreciation to the many individuals within the National Academies who provided support and assistance through this process. There were many. First, we give special thanks and commend our study director, Carol House, for her exemplary role in managing the panel’s activities, pulling together this report, and shepherding it through the review process. Carol drafted major portions of the report, contributing to every chapter. She worked with panel members individually to make certain that the final product represented the best of our collective efforts. Carol’s vision and her broad experience with survey design and implementation were absolutely critical to what the panel was able to accomplish. Connie Citro provided support to the panel throughout the entire process, attending panel meetings and providing comments and insight. Brian Harris-Kojetin also provided support as he succeeded Connie as director of the National Academies Committee on National Statistics. The panel would like to thank the staff at the National Academies Research Center who developed and ran a literature search on the uses of SIPP in academic research. Agnes Gaskin provided the panel with excellent support in all administrative areas, and Michael Siri and Eileen LeFurgy each stepped in to assist the panel after Agnes retired. The panel is grateful to our editor, Robert Katt, who did an excellent job in reviewing the report, making it more understandable and flow more easily. Kirsten Sampson-Snyder, Eugenia Grohman, and Yvonne Wise provided great professional services to help us with a peer review, editor, and processing of the report. We appreciate the assistance of Julia Kisa and Elizabeth Moylé in helping with contract issues.
Finally, we recognize the many federal agencies that support the Committee on National Statistics directly and through a grant from the National Science Foundation. Without their support for and commitment to improving the national statistical system, the work of this panel (and similar volunteer committees), which is the basis of this report, would not have been possible.
John L. Czajka, Chair
Panel on the Review and Evaluation of the 2014 Survey of Income and Program Participation: Content and Design
Acronyms
AFDC |
Aid to Families with Dependent Children |
ASEC |
Annual Social and Economic Supplement [to the Current Population Survey] |
BLS |
U.S. Bureau of Labor Statistics |
CAPI |
computer assisted personal interview |
CARI |
computer audio-recorded interviewing |
CBO |
Congressional Budget Office |
CBOLT |
CBO Long-Term microsimulation model |
CE |
Consumer Expenditure Survey |
CHIP |
Children’s Health Insurance Program |
CPS |
Current Population Survey |
CPS ASEC |
Current Population Survey Annual Social and Economic Supplement |
DEWS |
Dynamics of Economic Well-Being System |
DI |
disability insurance |
EHC |
event history calendar |
EITC |
Earned Income Tax Credit |
ERS |
Economic Research Service |
ESI |
employer-sponsored insurance |
FEM |
Financial Eligibility Model |
FICA |
Federal Insurance Contributions Act |
FNS |
Food and Nutrition Service |
FoodAPS |
National Household Food Acquisition and Purchase Survey |
GAO |
U.S. Government Accountability Office |
HISIM |
Health Insurance Simulation Model |
MCMC |
Markov chain Monte Carlo |
MEPS |
Medical Expenditure Panel Survey |
MINT |
Modeling Income in the Near Term |
NCES |
National Center for Education Statistics |
NHANES |
National Health and Nutritional Examination Survey |
NIPA |
National Income and Product Accounts |
NRC |
National Research Council |
NSDUH |
National Survey of Drug Use and Health |
NSFG |
National Survey of Family Growth |
OASDI |
Social Security Old-Age, Survivors, and Disability Insurance |
QC |
quality control |
SCF |
Survey of Consumer Finances |
SCHIP |
State Children’s Health Insurance Program |
SER |
summary earnings record |
SIPP |
Survey of Income and Program Participation |
SNAP |
Supplemental Nutrition Assistance Program |
SRMI |
sequential regression multiple imputation |
SSA |
U.S. Social Security Administration |
SSI |
Supplemental Security Income |
SSN |
Social Security number |
SSR |
Supplemental Security Record |
TANF |
Temporary Assistance for Needy Families |
TM |
topical module |
UI |
unemployment insurance |
WIC |
Special Supplemental Nutrition Program for Women, Infants, and Children |
Boxes, Tables, and Figures
BOXES
2-1 Topical Modules in SIPP (1984-2004)
2-2 Strengths and Weaknesses of the SIPP Data
2-3 Selected Recommendations from Reengineering the Survey of Income and Program Participation
4-1 Questionnaire Sections Incorporated into the EHC
4-2 Questionnaire Sections on Benefits/Income Not Incorporated into the EHC
4-3 Questionnaire Sections on Assets and Debts
6-1 Core Sections of the 2008 SIPP
6-2 2014 SIPP Questionnaire Sections Categorized by Groups
TABLES
3-1 SIPP Bibliographic References 2000-2014, by Topic
5-3 Job Line 1 Topic Flag Compared to 2012 Administrative Record Employment
5-4 Incentive Treatments and Response
6-1 Topical Modules of the 2001, 2004, and 2008 Panels, with Designated Waves
7-1 SIPP Annual Income as a Proportion of NIPA Total, by Source and Year, 1990-2012
7-2 2014 and 2008 SIPP Panel Estimates for Earned Income and Asset Income, by Source, April 2013
7-3 2014 and 2008 SIPP Panel Estimates of Safety Net Program Participation, by Source, April 2013
7-4 2014 and 2008 SIPP Panel Estimates of Retirement Income, by Source, April 2013
7-5 2014 and 2008 SIPP Panel Estimates of Health Insurance Coverage, by Source, April 2013
7-6 2014 SIPP and CPS ASEC Aggregate Income as a Proportion of NIPA Accounts Total, by Income Source
7-9 2008 and 2014 SIPP Panel Estimates of Unemployment Beneficiaries, by Month, 2013
7-10 2008 and 2014 SIPP Panel Estimates of SNAP Beneficiaries, by Month, 2013
7-11 2008 and 2014 SIPP Panel Estimates of SNAP Households, by Month, 2013
7-12 2008 and 2014 SIPP Panel Estimates of Aggregate SNAP Benefits, by Month, 2013
7-13 2008 and 2014 SIPP Panel Estimates of Employment, by Month, 2013, Compared with CPS Estimates
7-20 Response Rates and Cumulative Sample Loss by Wave, 2001-2014 SIPP Panels
7-21 Response Rate to the SSA Supplement
FIGURES
4-1 Example of standard interviewing screen for SIPP
4-2 Example of an event history calendar screen for reengineered SIPP
4-3 Overlap in timeline of two interview waves from which preloaded data are pulled
4-4 Illustration of closed and open spells as they appear in the event calendar screen
This page intentionally left blank.
Contents
2 Background and Panel’s Activities
History of SIPP: Its Modifications, Strengths, and Weaknesses
SIPP and the 2014 Reengineered Panel
Contributions to the Research Literature
Selected Agency Applications of SIPP Data
Conclusion on the Uses of SIPP
4 The Questionnaire: Instrumentation and Flow
Summary of Questionnaire Design and Flow
6 Adequacy of the Content of the 2014 SIPP
Overview of Prior SIPP Content
Content Related Reengineering Challenges
Overview of the 2014 SIPP Content
Delineating Major Content Changes
Handling of Part-Year Residents
Conclusions and Recommendations Regarding Content
Quality Measurement for Income and Program Participation
Income Reporting in Prior SIPP Panels
Income and Program Participation
Summary of Findings with Recommendations
Respondent Burden and Its Components
Comparing Respondent Burden on SIPP: New Design Versus Old Design
Summary of Review of Respondent Burden
9 Additional Comments and Recommendations
Strengths, Weaknesses, and Changes
Facilitating the Use of SIPP Data
Supporting Respondents and Field Representatives