Summary
The Survey of Income and Program Participation (SIPP) is a national, longitudinal household survey conducted by the Census Bureau. SIPP serves as a tool to evaluate the effectiveness of government-sponsored social programs and to analyze the impacts of actual or proposed modifications to those programs. SIPP was designed to fill a need for data that would give policy makers and researchers a much better grasp of how effectively government programs were reaching their target populations, how participation in different programs overlapped, and to what extent and under what circumstances people transitioned into and out of these programs. SIPP was also designed to answer questions about the short-term dynamics of employment, living arrangements, and economic well-being. SIPP collects monthly data on labor force participation, income by source, social program participation and eligibility, health insurance coverage, the demographic characteristics of household members, and other items. Federal agencies and academic researchers use SIPP data to study critical social issues, such as poverty, disability, health insurance coverage, child welfare, resources for retirement, and more.
The Census Bureau has reengineered SIPP—fielding the initial redesigned survey in 2014.1 Two major changes made in the reengineering were that sampled households are contacted only once per year rather than three times and that the topical modules have been eliminated. The bureau asked the National Academies of Sciences, Engineering, and Medicine to conduct an independent evaluation of the new design compared with the old design, as
___________________
1 SIPP has been redesigned several times since its first implementation in 1983. See Chapter 2.
a complement to its own evaluations. In response, the National Academies established the Panel on the Review and Evaluation of the 2014 Survey of Income and Program Participation: Content and Design and tasked this study panel with: (1) comparing key estimates across the two designs, (2) evaluating the content of the redesigned SIPP, (3) evaluating the impact of the new design on respondent burden, and (4) considering content changes for future improvement of SIPP.2
It took 2 years to gain access to data from wave 1 of the 2014 SIPP panel (covering calendar year 2013) that were sufficiently close to “final” to be used in our evaluation. Processing of the wave 2 data (for calendar year 2014), which would have enabled the study panel to investigate additional key questions about the survey redesign, was still under way when the panel concluded its work. The panel wanted to explore more closely the many differences it observed, but a limited timeline and little flexibility on the end-date for delivery of a final product shaped the ultimate content of this report.
In this summary, the study panel organizes its results to mirror the Statement of Task, rather than following the sequence of chapters in the report. The numbering of conclusions and recommendations reflects the chapter in which they appear in the report body.
QUALITY OF KEY ESTIMATES
The primary charge to the panel was to analyze data collected under the new and old SIPP designs to determine the extent to which the new design improves upon, maintains, or underperforms the old design in terms of the quality of key estimates. The study panel presents 15 specific findings based on its independent data analyses. These analyses covered the topics of income and program participation, recall bias, measuring intrayear dynamics, and nonresponse. The study panel obtained mixed results.
CONCLUSION 7-1: The panel’s assessment of variables key to the Survey of Income and Program Participation’s (SIPP’s) purpose indicates that, when compared to the older design of SIPP, the redesigned SIPP sometimes outperforms, sometimes maintains performance, and other times underperforms the older design. Neither design is uniformly better. The new SIPP design outperforms the old design most notably with regard to reports of earnings and asset income (interest, dividends, and rent). The new SIPP design underperforms the old design in a number of areas that are particularly important to SIPP’s focus on
___________________
2 The full text of the National Academies Statement of Task to the study panel appears in Chapter 1.
poverty and within-year dynamics of low-income households. These areas include participation early in the calendar year, overall underreporting of income from programs such as Supplemental Security Income, family assistance (Temporary Assistance for Needy Families and General Assistance), unemployment insurance, and Supplemental Nutrition Assistance Program, and underreporting of transitions in program participation.
For areas in which the redesigned SIPP underperforms, the panel members make specific recommendations to the Census Bureau to focus on improvements.
RECOMMENDATION 7-1: The Census Bureau should develop a research program to examine the effects of seam bias on the 2014 Survey of Income and Program Participation and should develop ways to mitigate these effects.
RECOMMENDATION 7-2: The Census Bureau needs to establish and document procedures to identify and handle outliers in Survey of Income and Program Participation (SIPP) data. The procedures should include early identification during the interview process in order to correct erroneous data. The procedures should identify data that significantly influence key estimates and methods to address those influences. These outlier procedures should be applied to internal SIPP data, apart from any topcoding of the public use data.
RECOMMENDATION 7-3: The Census Bureau should conduct analyses to identify and address issues with recall bias in reporting income and program participation for months during the early part of the reference year. Such bias appears to be particularly evident for lower income groups.
RECOMMENDATION 7-4: The Census Bureau should conduct research to identify why the redesigned Survey of Income and Program Participation (SIPP) has difficulty relative to the earlier SIPP panels in accurately capturing income from households at the bottom of the income distribution.
RECOMMENDATION 7-5: Accurately measuring transitions is a critical function for the Survey of Income and Program Participation (SIPP). The Census Bureau should research the areas where the redesigned SIPP performs poorly relative to the 2008 SIPP in measuring these transitions, and should develop a plan to test and improve such mea-
surements. This research should include a careful assessment of field representatives’ use of the EHC (event history calendar), and whether the EHC, as implemented, can be effective in assisting respondents’ recall of spells. The testing of a more integrated EHC should be a part of this research. The research should evaluate an option for restructure that would record all income and program participation spells before asking any follow-on questions, with a mechanism to review cumulative spells before leaving the EHC. If the research to restructure the EHC does not lead to a marked improvement in the accurate collection of spells, the Census Bureau should explore the return to a shorter reference period.
RECOMMENDATION 7-6: The Census Bureau should address the “short month” discontinuity between income measurement and poverty measurement by handling “month size” similarly in both estimates.3
RECOMMENDATION 7-7: The substantial decline in response between waves 2 and 3 of the 2014 Survey of Income and Program Participation panel raises concerns about attrition bias and the adequacy of the sample for multiyear longitudinal analysis. The Census Bureau should investigate factors contributing to the decline and whether there is evidence of attrition bias. This investigation should include the adequacy of weights in compensating for attrition.
RECOMMENDATION 7-8: The poor response rate to the Social Security Administration Supplement could discourage prospective sponsors of future supplements. The Census Bureau should conduct a nonresponse bias analysis to determine if the low response rate affected the representativeness of the results and should investigate alternative approaches to collecting supplemental data that might provide better overall quality.
CONTENT EVALUATION
The panel was also directed to “evaluate the depth and breadth of the 2014 SIPP content.” The panel compared the prior SIPP content, the ongoing uses of SIPP data, and the content of the reengineered SIPP. The panel delineated major content changes in the areas of income and program participation, expenses, and other outcomes of interest and related economic activities, and drew the following two conclusions and a recommendation:
___________________
3 The short month discontinuity is related to how the Census Bureau assigns earnings to months for persons with pay periods that vary by the number of days in the month.
CONCLUSION 6-1: Overall, the subject area content of the Survey of Income and Program Participation has remained remarkably consistent, given the large changes in survey methods.
CONCLUSION 6-2: While most of the core content and much of the key content of the topical modules (TMs) of prior panels were included in the 2014 Survey of Income and Program Participation (SIPP), important variables were lost. In fact one agency—the Social Security Administration (SSA)—found it necessary to commission a supplement to meet its information needs. Existing models dependent on SIPP data, such as those produced for the SSA and the Food and Nutrition Service, will need to be modified to varying degrees to adapt to the new data availability. Important variables related to material hardship are no longer collected. Additionally, the lack of periodic TMs limits the ability to field new questions addressing timely policy issues.
RECOMMENDATION 6-1: The Census Bureau should reevaluate whether critical content from the Social Security Supplement can be effectively added to the Survey of Income and Program Participation questionnaire.
IMPACT ON RESPONDENT BURDEN
The panel was directed to evaluate the impact of the new SIPP data collection procedures on respondent burden. Respondent burden is a concept that is difficult to define and even more difficult to measure. The panel looked first at the literature that explores whether and how components of burden might be measured. It then compared the redesigned SIPP and its predecessor design regarding interview length, number of interviews, difficulty of the interview process, and nonresponse. The study panel found that the necessary data—especially for prior panels—were inadequate or unavailable, which precluded an objective and complete assessment. The information the panel was able to examine provided a mixed picture of the potential components contributing to respondent burden. As a result, the panel is unable to make a definitive statement about change in respondent burden between the 2008 and 2014 panels.
CONCLUSION 8-1: The panel could not make a determination as to whether the redesign of the Survey of Income and Program Participation affected respondent burden, either positively or negatively.
METHOLOGICAL ENHANCEMENTS IN THE REDESIGNED SIPP
In the redesign of SIPP for 2014, the Census Bureau incorporated a number of methodological enhancements. Although the study panel was not specifically charged with reviewing these enhancements, the panel discusses four of them in Chapter 5 as steps the Census Bureau has taken to improve SIPP: a redesigned imputation system, greater use of administrative data, use of a subsystem that can record interviews, and the continuation of incentive experiments to improve response and decrease perceived burden.
The panel commends the Census Bureau and SIPP staff for their work on these enhancements, and it provides several recommendations for continued improvement.
RECOMMENDATION 5-1: The Census Bureau should continue to research, evaluate, and implement modifications and further enhancements to the new imputation system. These potential enhancements include extending the model-based approach to additional variables, such as assets. Further evaluations should also include consideration of the study panel’s suggestions for improving model-based imputation. The effects of various enhancements/modifications should be quantified and provided to data users.
RECOMMENDATION 5-2: The Census Bureau should continue to investigate effective use of administrative records, such as Internal Revenue Service data mentioned in this report, to enhance the quality of data collected in the Survey of Income and Program Participation. This research should include identification and correction of both false positives and false negatives. Efforts should continue to obtain administrative data for important state-run programs such as the Supplemental Nutrition Assistance Program.
RECOMMENDATION 5-3: Because of apparent respondent confusion between Social Security (Old-Age, Survivors, and Disability Insurance) and SSI (Supplemental Security Income) payments, questions about participation in these programs should be co-located within the calendar part of the questionnaire so there can be cross-checking of participation in these two programs. Field representatives should be further trained on the differences between these two programs and on effective ways to help eliminate confusion on the part of respondents.
RECOMMENDATION 5-4: The Census Bureau should extend the collection of audio recordings so as to capture as much of the interview as possible for all respondents, including questions within the event history
calendar. These recordings should be routinely reviewed for evidence of confusing questionnaire items and factors contributing to seam bias and item nonresponse, in addition to quality control issues such as interviewer falsification, deviations from questionnaire wording, and deviations from standardized interviewing. The use of audit trails along with audio streams would be important for these types of evaluations.
RECOMMENDATION 5-5: The Census Bureau should continue to research and utilize incentive programs in the Survey of Income and Program Participation to promote response and reduce nonresponse bias with the goal of devising an effective long-term plan. The plan should specifically address the effect of incentives on nonresponse bias.
ADDITIONAL COMMENTS AND RECOMMENDATIONS
In the concluding chapter (Chapter 9) of this report, the panel broadens its look at SIPP, adding comments and recommendations in three different areas: addressing how the strengths and weakness of SIPP (as identified by the 2009 review of SIPP by the National Academies) have changed based on the redesign; facilitating the use of SIPP data; and supporting respondents and field representatives. The panel makes one conclusion and eight specific recommendations.
CONCLUSION 9-1: The 2014 Survey of Income and Program Participation (SIPP) content will allow the SIPP program to continue to measure income and program participation and their principal determinants. Our analysis of wave 1 data, however, documents significant differences in a number of estimates between the 2008 and 2014 SIPP panels. Researchers interested in using SIPP panels as repeated cross-sectional data to generate consistent time trends should be aware of how changes (to the design, sample dynamics, reference period, question wording and ordering) may impact their findings.
RECOMMENDATION 9-1: The Census Bureau should release Survey of Income and Program Participation data within 1 year of the end of data collection.
RECOMMENDATION 9-2: The Census Bureau should commit to developing a formal release process and timetable for the Survey of Income and Program Participation, including release of data and publications. The publications should contain key estimates and data characteristics, and they should be a formal and publicized part of the process. Timetables should be met.
RECOMMENDATION 9-3: The Census Bureau should continually improve the documentation available to data users. A complete crosswalk between the variables of the old and new Survey of Income and Program Participation and a comprehensive data dictionary are needed as part of this documentation. The Census Bureau should modify and enhance the documentation based on feedback from data users.
RECOMMENDATION 9-4: The Census Bureau should engage in a long-term research agenda on the Survey of Income and Program Participation (SIPP). This agenda should address the specific issues with data and estimates outlined in this report, soliciting advice from the survey research community as it has in the past. The agenda should also include an assessment of seam bias, spell length, and asset measurement, which this study panel was not able to explore. Evaluations of the reporting of levels and changes in program participation should make use of administrative records where available. To measure the effectiveness of the research and of modifications made based on that research, the Census Bureau’s research should also include an ongoing assessment of the quality of SIPP estimates such as that presented in this report.
RECOMMENDATION 9-5: The Census Bureau should formally solicit data users’ feedback, input, and ideas after those users have had time to work with wave 1 and wave 2 datasets from the redesigned Survey of Income and Program Participation (SIPP). Based on this feedback, the Census Bureau should reevaluate the inclusion of lost content that was unique to SIPP, such as material hardship.
RECOMMENDATION 9-6: Investigating the issues with measuring intrayear transitions with the redesigned Survey of Income and Program Participation (SIPP) should be a primary focus of the research agenda described in Recommendation 9-4. Until further work is done to improve understanding of these issues, the Census Bureau should alert data users that estimates of transitions derived from the 2014 SIPP panel might not compare with estimates from earlier panels.
RECOMMENDATION 9-7: The Census Bureau should develop and implement a systematic plan to measure objective respondent burden, perceived burden, and their components, following best practices as found in current literature.
RECOMMENDATION 9-8: The Census Bureau should evaluate interviewer protocols, administration, training, and variance. It should
modify its current training for field representatives, putting more emphasis on the effective use of the event history calendar and on understanding the complexity of social programs, income sources, and other elements that are the focus of the Survey of Income and Program Participation.
This page intentionally left blank.
