9

Additional Comments and Recommendations

Chapters 6 through 8 provide the study panel’s assessment of the reengineered Survey of Income and Program Participation (SIPP) in the specific areas identified in the Statement of Task (see Box 1-1 of this publication). In this chapter, the panel broadens its look at SIPP, providing additional comments and recommendations for improvement.

STRENGTHS, WEAKNESSES, AND CHANGES

The chapter begins by drawing from the most recent National Research Council report on SIPP (National Research Council, 2009) and looks at the redesigned SIPP in comparison to the strengths and weaknesses that report identified for the previous design (see Box 2-2 of this publication). Are the strengths still there? Have some of the weaknesses been mitigated with the new design?

The redesigned SIPP has retained a number of the strengths that have made SIPP an important source of data for policy makers and researchers studying social programs in the United States, but our findings are decidedly mixed—particularly in areas where SIPP has traditionally excelled. The redesign has also failed to mitigate some of the important weaknesses. On the whole, reductions in quality of measurement are more common than improvements.

• Earnings above the poverty level and asset income are better reported in the 2014 SIPP panel than in the 2008 panel. These are the major components of income. There were mixed results for

• reporting of transfer income, poorer results in reporting of pension income, and unchanged results in reporting of health insurance coverage.
• Compared to the 2008 panel, the 2014 SIPP panel captures less income at the very bottom of the income distribution, as reflected in higher estimates of the monthly percentage of the population in families below 50 percent of the federal poverty threshold. This is a weakness because these lower income families are particularly important for social program evaluation.
• On the issue of the redesigned survey moving to a longer recall period, it was unclear to the 2009 report’s authors whether the annual survey would be subject to a recall bias across the reference year. However, this study panel’s assessment found minimal evidence of recall bias for most sources in the 2014 SIPP panel. The exceptions that our analysis discovered are problems with the reporting of unemployment compensation and the Supplemental Nutrition Assistance Program.
• The 2014 SIPP shows some serious problems in estimating transitions. See further discussion below.
• The authors of the 2009 report concluded that the topical modules were a major strength of the 2008 SIPP (National Research Council, 2009, p. 30). Although topical modules have been eliminated in the redesign, our assessment of content shows that most, but not all, of the critical data collected in these areas are now part of the annual survey. However, a supplement was needed to supply additional critical information for the Social Security Administration.
• A lack of timeliness in the release of data files continues to be a major problem for SIPP, as discussed below.
• The 2009 report found that inadequate documentation to assist users was a serious weakness for SIPP (National Research Council, 2009, p. 31). This study panel believes that the SIPP 2014 Users’ Guide (U.S. Census Bureau, 2016) provides good documentation, although it was not available early enough for the panel’s work.

FACILITATING THE USE OF SIPP DATA

SIPP is a very complex survey, and the issues addressed by many users of SIPP are also complex. Thus, the Census Bureau needs to be proactive in supporting these users as they adjust to the SIPP redesign. The panel offers a conclusion and several recommendations in this section.

Changes in SIPP will necessitate careful attention by data users. Changes in the reference period and slight improvements in the phrasing of questions alone will likely produce seams in time trends that will be difficult to

disentangle from real changes in the phenomena being measured. Producing time trends, however, is only one of SIPP’s functions and arguably is not critical to many of its main objectives. And while time trends would be of interest, it is more important to maintain and improve the measurement of intrayear dynamics while addressing changing resources available to the SIPP program.

CONCLUSION 9-1: The 2014 Survey of Income and Program Participation (SIPP) content will allow the SIPP program to continue to measure income and program participation and their principal determinants. This analysis of wave 1 data, however, documents significant differences in a number of estimates between the 2008 and 2014 SIPP panels. Researchers interested in using SIPP panels as repeated cross-sectional data to generate consistent time trends should be aware of how changes (to the design, sample dynamics, reference period, question wording and ordering) may impact their findings.

The panel reiterates that SIPP’s usefulness depends on maintaining a survey process that produces high-quality information in a timely and cost-effective manner. In the initial waves of the 2014 SIPP, timeliness has been a major problem. Data collection for wave 1 was conducted in the spring of 2014, but the public-use panel dataset for wave 1 was released in March 2017. There is the expectation that wave 2 will be released more quickly. The 2009 report recommended that the “Census Bureau should release Survey of Income and Program Participation data within 1 year of data collection” (National Research Council, 2009, p. 10). This study panel reinforces this earlier recommendation.

RECOMMENDATION 9-1: The Census Bureau should release Survey of Income and Program Participation data within 1 year of the end of data collection.

Part of the barrier to timely release of SIPP data is that there do not exist a formal release process and timetable that are communicated to data users and adhered to by the Census Bureau. As pressure mounts and resources become more limited, delaying the release of SIPP data may be an easier choice than delaying other important Census Bureau releases. To deflect such decisions, this panel believes that structure, supported by senior management, is needed for the entire release process of SIPP.

RECOMMENDATION 9-2: The Census Bureau should commit to developing a formal release process and timetable for the Survey of Income and Program Participation, including release of data and pub-

lications. The publications should contain key estimates and data characteristics, and they should be a formal and publicized part of the process. Timetables should be met.

The study panel performed its analyses by developing its own programming routines and applying them to internal datasets, with only the internal SPIDER system and, later, the SIPP 2014 Users’ Guide for documentation of variables on the new system. There was no crosswalk to indicate how variables changed in content and naming convention between the old and new SIPP. Creating accurate programming routines under these conditions was both time consuming and frustrating. In the summer of 2016, the Census Bureau issued the SIPP 2014 Users’ Guide. It is well written and contains much of the needed information. It contains a partial crosswalk of variables. This guide was provided too late for most of the study panel’s work, and a number of details of how the new system works are not addressed in the guide. However, the guide is a strong step in the right direction, and data users need a continuing flow of good documentation.

RECOMMENDATION 9-3: The Census Bureau should continually improve the documentation available to data users. A complete crosswalk between the variables of the old and new Survey of Income and Program Participation and a comprehensive data dictionary are needed as part of this documentation. The Census Bureau should modify and enhance the documentation based on feedback from data users.

This panel assessed the redesigned SIPP while working under a number of constraints. The panel has reported a broad set of results but our assessment did not include areas it had hoped to evaluate, such as seam bias, spell length, impact of imputation, and asset measurement. The wave 2 data necessary for analyses on these areas did not become available in time to include in our analytical work, and this delay in data access prevented our examination of key questions regarding the viability of the survey’s estimates of short-term dynamics. More work is needed in a long-term effort to address the identified problems, to explore the areas that the panel could not, and to monitor quality over time.

RECOMMENDATION 9-4: The Census Bureau should engage in a long-term research agenda on the Survey of Income and Program Participation (SIPP). This agenda should address the specific issues with data and estimates that the study panel has outlined in this report, soliciting advice from the survey research community as it has in the past. The agenda should also include an assessment of seam bias, spell length, and asset measurement, which this study panel was not able to

explore. Evaluations of the reporting of levels and changes in program participation should make use of administrative records where available. To measure the effectiveness of the research and of modifications made based on that research, the Census Bureau’s research should also include an ongoing assessment of the quality of SIPP estimates such as that presented in this report.

An important assessment of the quality of the new SIPP data will be made by data users as they employ these data to answer existing and evolving questions. The Census Bureau needs to tap into the findings and concerns that result from the assessments made by these data users, and the bureau should allow their needs to help drive the long-term research agenda.

RECOMMENDATION 9-5: The Census Bureau should formally solicit data users’ feedback, input, and ideas after those users have had time to work with wave 1 and wave 2 datasets from the redesigned Survey of Income and Program Participation (SIPP). Based on this feedback, the Census Bureau should reevaluate the inclusion of lost content that was unique to SIPP, such as material hardship.

This panel has serious concerns about the use of the redesigned SIPP data to study short-term dynamics and intrayear transitions (Finding 7-10). Because providing the data to conduct this type of analysis is a key objective of SIPP, the panel wants to further emphasize its concern here and recommend action by the Census Bureau.

RECOMMENDATION 9-6: Investigating the issues with measuring intrayear transitions with the redesigned Survey of Income and Program Participation (SIPP) should be a primary focus of the research agenda described in Recommendation 9-4. Until further work is done to improve understanding of these issues, the Census Bureau should alert data users that estimates of transitions derived from the 2014 SIPP panel might not compare with estimates from earlier panels.

SUPPORTING RESPONDENTS AND FIELD REPRESENTATIVES

Reducing burden on respondents sampled for SIPP was an important goal of the SIPP redesign, but the panel was unable to tell whether that goal was achieved. With that said, the monitoring and reduction of respondent burden should continue to be an important goal of the SIPP program as it moves into the future. This monitoring will require better systematic measures of objective burden, perceived burden, and the components of both. These measures should draw upon data and information collected

from interviewer debriefings and interviewer notes, respondent debriefings, broader utilization of recorded interviews, and other techniques identified through the literature. Apart from helping to understand burden, these techniques and the measures of burden resulting from them may also provide an understanding of respondents’ difficulty (or hesitation) in answering certain questions and lead to improved question design.

RECOMMENDATION 9-7: The Census Bureau should develop and implement a systematic plan to measure objective respondent burden, perceived burden, and their components, following best practices as found in current literature.

The redesign of SIPP to an annual survey has changed a number of dynamics in fielding the survey. In prior years SIPP was a year-round survey; when field division and field representatives completed a wave of SIPP, they would begin the next wave. Field representatives trained on SIPP remained busy and engaged throughout the year. With the annual survey, the dynamics and timeline have changed. The survey is fielded across fewer months (approximately 5 months rather than 12), with no SIPP data collection the rest of the year. This creates a need for a more intensive retraining of returning field representatives and will likely mean a higher turnover of field representatives, with an entire cohort of new hires having to be brought on board and trained at the start of each data collection period.

The study panel used the computer audio-recorded interviewing system to listen to selected interviews. Even though panel members could not listen to interviews “at the seam” to investigate seam effects, they did learn from the recordings they were able to access. They found that the field representatives did not use the full potential of the event history calendar to identify gaps and inconsistencies in spells and facilitate accurate monthly reporting. They also found that field representatives were sometimes confused about social programs, for example Social Security versus Supplemental Security Income. When the field representative is confused, the respondent is likely to be also. Additional training and follow-through are needed.

RECOMMENDATION 9-8: The Census Bureau should evaluate interviewer protocols, administration, training, and variance. It should modify its current training for field representatives, putting more emphasis on the effective use of the event history calendar and on understanding the complexity of social programs, income sources, and other elements that are the focus of the Survey of Income and Program Participation.

The panel also reemphasizes Recommendation 5-4 (see Chapter 5), which advises a more inclusive recording of interviews and using those recordings for a broader set of evaluations than is currently in practice.

CONCLUDING COMMENTS

This panel is very supportive of SIPP. The survey has a critical role, needed now more than ever, in assisting efforts to evaluate the effectiveness of government-sponsored social programs and to analyze the impact of actual or proposed modifications to those programs. The study panel appreciates the work that the Census Bureau has done to redesign SIPP to meet changing budgets and data needs. The panel members strongly encourage a continuous-improvement approach that will make the new SIPP even better able to fulfill its objectives.

This page intentionally left blank.