The 2014 Redesign of the Survey of Income and Program Participation An Assessment (2018) / Chapter Skim
Currently Skimming:
7 Quality of Key Estimates
7 Quality of Key Estimates
Pages 107-162
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 107... ...
This chapter presents an assessment of data quality in the 2014 SIPP panel relative to the 2008 panel1 and, in some cases, independent sources. 1 The 2014 SIPP panel data and 2008 SIPP panel data, from which these comparisons were made, were generated through different survey designs, questionnaires, reference periods, and imputation methodology.
|
From page 108... ...
To provide a suitable context for our assessment, this chapter begins with the study panel's view of quality measurement for survey estimates of income and program participation, followed by a review of income reporting in prior SIPP panels. The chapter then describes the methods used in the study panel's assessment of the new SIPP data.
|
From page 109... ...
Definitional differences tend to be more of an issue with income than with program participation. For example, receipt of unemployment compensation or enrollment in Medicaid has a very clear definition in administrative data, but interest or dividend income will differ substantially depending on whether or not the amounts earned in retirement accounts are included in the definition.
|
From page 110... ...
The standard of comparison for this assessment is nonresponse in recent SIPP panels and, for item non esponse, in another Census Bureau survey whose income r estimates are widely used. INCOME REPORTING IN PRIOR SIPP PANELS The 2009 NRC report documented modest declines in the quality of SIPP's measurement of income from a variety of sources between the survey's debut in 1983 and 1996 (National Research Council, 2009, p.
|
From page 111... ...
This is a critical question in our assessment of the quality of reporting of income and program participation in the redesigned survey. METHODS The partial overlap between wave 1 of the 2014 SIPP panel and the final waves of the 2008 panel afforded the study panel an opportunity to assess how estimates from the two panels compare over the same period of time.
|
From page 112... ...
Total, by Source and Year, 1990-2012 Source of Income 1990 1991 1992 1993 1994 1995 1996 1997 Earnings 0.89 0.89 0.87 0.86 0.85 0.84 0.89 0.87 Wages and salary 0.85 0.86 0.84 0.84 0.84 0.83 0.87 0.85 Self-employment 1.19 1.19 1.06 0.98 0.96 0.93 1.06 0.97 Property 0.57 0.54 0.49 0.48 0.46 0.45 0.40 0.35 Interest 0.66 0.69 0.63 0.59 0.61 0.58 0.55 0.44 Dividends 0.40 0.35 0.35 0.39 0.34 0.35 0.28 0.26 Rent and royalties 0.59 0.48 0.41 0.41 0.40 0.36 0.34 0.32 Transfers 0.93 0.92 0.89 0.88 0.87 0.88 0.87 0.87 Social security + railroad 0.97 0.95 0.93 0.91 0.90 0.90 0.87 0.87 retirement Supplemental security income 0.84 0.87 0.85 0.84 0.85 0.88 0.90 0.95 Family assistancea 0.91 0.91 0.87 0.96 0.94 0.95 0.95 0.93 Other cash welfare 0.21 0.24 0.23 0.26 0.21 0.24 0.29 0.33 Unemployment compensation 0.70 0.77 0.77 0.78 0.70 0.68 0.59 0.56 Workers' compensation 0.65 0.60 0.58 0.54 0.56 0.51 0.72 0.66 Veterans' payments 0.78 0.76 0.78 0.74 0.73 0.66 0.73 0.69 Pensions 0.92 0.95 0.93 0.95 0.97 0.95 0.86 0.93 Private pension 0.98 0.98 0.91 0.95 0.99 0.99 0.87 0.96 Federal pension 0.78 0.83 0.83 0.84 0.87 0.91 0.77 0.79 Military retirement 0.88 0.88 0.89 0.89 0.90 0.86 1.00 1.10 State and local pensions 0.78 0.83 0.82 0.81 0.79 0.78 0.67 0.71 Total Income 0.86 0.87 0.84 0.84 0.83 0.82 0.85 0.83 NOTES: Omitted years (2000 and 2008) did not have 12 months of data collected by SIPP.
|
From page 114... ...
The estimates presented here represent a small subset of our analyses; they were selected to address what the study panel believes are the most important questions that they could answer with respect to the 2014 SIPP panel.
|
From page 115... ...
On balance, then, it did not appear likely that uncorrected differences in sample composition due to attrition would be a significant factor in the comparisons the panel conducted. INCOME AND PROGRAM PARTICIPATION Because of findings such as those reported in the previous section of this chapter showing that surveys underestimate benchmark estimates of aggregate income -- and for some sources underestimate by substantial margins -- comparative evaluations of reported income can generally rely on the principle that "more is better" to determine which survey's estimate of a given income source is the most accurate.
|
From page 116... ...
Therefore, in addition to basing our assessment on comparisons between the two panels, the study panel also looked at the consistency of 2014 panel reporting over the full 12 months. With two notable exceptions, unemployment compensation and the Supplemental Nutrition Assistance Program (SNAP)
|
From page 117... ...
SOURCE: Panel generated with data from the 2008 (public-use data file) and 2014 SIPP (iteration 12)
|
From page 118... ...
Retirement Income The study panel's retirement income comparisons include railroad retirement, the four main types of pensions (private, federal, military, and state and local) , and withdrawals from IRA, 401k, Keogh, and thrift plans.
|
From page 119... ...
SOURCE: Panel generated with data from the 2008 (public-use data file) and 2014 SIPP (iteration 12)
|
From page 120... ...
. Mean amounts of income were significantly higher in the 2014 panel estimates for railroad retirement and private pensions; (statistically)
|
From page 121... ...
SOURCE: Panel generated with data from the 2008 (public-use data file) and 2014 SIPP (iteration 12)
|
From page 122... ...
panels. Comparison with Independent Sources Table 7-6 compares annual aggregates calculated from the 2014 SIPP panel with NIPA totals and the 2014 Current Population Survey Annual Social and Economic Supplement (CPS ASEC)
|
From page 123... ...
bCPS ASEC rent value includes royalties, but royalties in the 2014 SIPP are combined with other investment income and cannot be separated for the purpose of adding to rent. The NIPA rent amount also excludes royalties.
|
From page 124... ...
Comparing Table 7-6 with Table 7-1, the results for the 2014 SIPP panel represent a substantial fall-off from the 2008 panel, which captured 75 percent of the NIPA total pension income in 2012. Income Relative to Poverty Many of the uses of SIPP reviewed in Chapter 3 focus on the lowincome population.
|
From page 125... ...
TABLE 7-7 2008 and 2014 Survey of Income and Program Participation (SIPP) Panel Estimates of the Percentage of the Population in Selected Ranges of Poverty Thresholds, by Month, 2013 Percentage Below 50% Percentage 50% to 100% Percentage Above 100% but Less of Threshold of Threshold Than 200% of Threshold 2014 2014 2014 2014 Panel 2014 Panel 2014 Panel Panel Without Panel Without Panel Without 2008 With Type Type 2 2008 With Type Type 2 2008 With Type Type 2 Month Panel 2 Income Income Panel 2 Income Income Panel 2 Income Income January 7.73 10.06 10.37 8.82 7.35 7.36 20.74 17.55 17.58 February 7.83 10.33 10.62 8.98 8.20 8.24 20.95 18.68 18.75 March 7.66 9.89 10.19 9.00 7.48 7.52 21.02 17.63 17.62 April 7.53 9.75 10.03 8.81 7.60 7.64 21.13 18.03 18.05 May 7.16 9.34 9.63 8.77 7.59 7.63 20.67 17.53 17.54 June 7.47 9.23 9.51 8.99 7.83 7.87 20.99 18.18 18.20 July 7.42 9.17 9.45 8.63 7.74 7.77 20.97 17.80 17.81 August 6.90 9.04 9.26 8.67 7.68 7.74 21.12 17.68 17.70 September 6.69 8.91 9.08 8.52 7.87 7.86 21.47 18.24 18.26 October 6.60 8.58 8.71 8.37 7.62 7.64 21.08 17.93 17.87 November 7.00 8.71 8.80 8.41 7.85 7.85 20.99 18.27 18.26 December 8.64 8.74 7.78 7.77 17.90 17.84 SOURCE: Panel generated with data from the 2008 (public-use data file)
|
From page 126... ...
The estimate of persons falling into this range of family income is higher in the 2008 panel by about 1.5 percentage points. The difference between the two panel estimates declines over time to as low as 0.6 percentage points.
|
From page 127... ...
One must ask whether the addition to resources by Type 2 persons in families above 50 percent of the poverty threshold is truly offset by their addition to the poverty threshold (level for the unit) or whether the very crude estimate of their added income obtained by the 2014 SIPP panel tends to understate their contribution.
|
From page 128... ...
Treatment of Excessive Hourly Wages The 12-month reference period allowed respondents to the 2014 SIPP to report their earnings, interest, and other sources of income as annual amounts, which could facilitate more accurate reporting for sources with 10 This issue exists only in the 2014 SIPP because in the 2008 SIPP the Census Bureau did not allocate earnings to a month based on how many days that month contained.
|
From page 129... ...
While this flexibility was expected to improve the accuracy of reporting -- and perhaps even reduce item nonresponse -- it could also lead to errors when respondents reported a correct amount but for the 11 For wage and salary income, the annual reporting period could also make it easier for r espondents to report gross rather than net income, as gross earnings rather than takehome pay are what appear on employees' W-2 forms, whereas take-home pay is much more p rominent -- and presumably more salient to recipients -- on the paychecks received throughout the year. Some respondents' reporting of net instead of gross earnings has been suggested as a possible reason for the earlier SIPP panels' substantial underreporting of annual earnings compared to the CPS ASEC (Roemer, 2000)
|
From page 130... ...
The study panel produced most of its estimates from internal data that were not topcoded. To enable us to generate more reasonable estimates of aggregate earnings, the Census Bureau identified the high and low outliers to our programmer, and the panel removed these observations -- about 40 in total -- from our estimates of monthly and annual earnings.
|
From page 131... ...
In its own evaluation, briefly described above, the panel compared the 2014 SIPP panel estimates of monthly program participation in 2013 with estimates derived from the 2008 panel over the same period. Where possible to assemble monthly data on program participation from administrative sources, the panel compared SIPP estimates to these data as well.
|
From page 132... ...
The 2008 SIPP panel captures this general trend, declining from 3.8 million in January to a minimum of 2.4 million in September and then reversing direction to rise back to 2.8 million in November. The 2008 panel estimates are 64 to 65 percent of the administrative totals from January through June but show some volatility over the remaining months as the sample gets thinner.
|
From page 133... ...
Estimates of SNAP participation in the 2008 panel were likewise fairly stable over the first half of the year, holding steady at between TABLE 7-10 2008 and 2014 Survey of Income and Program Participation (SIPP) Panel Estimates of Supplemental Nutrition Assistance Program (SNAP)
|
From page 134... ...
. The 2008 SIPP panel estimates TABLE 7-11 2008 and 2014 Survey of Income and Program Participation (SIPP)
|
From page 135... ...
Over the year as a whole, reported payments as a share of true payments fell in the 2008 panel from close to 80 percent in the first 5 months of the year to percentages in the low 70s late in the year, while rising in the 2014 panel estimates by a modest amount, with an increase in reporting from 63.5 to 73.5 percent over the course of the year. To summarize, estimates from the 2014 panel for two of the most important safety net programs exhibit trends that run counter to program data because of a substantial recall bias that contributes to a marked decrease in reporting accuracy from the end of the reference period back to the beginning.
|
From page 136... ...
All three surveys show similar upward trends, with the 2014 SIPP panel estimates running 1 to 2 percentage points higher than the CPS and the 2008 panel running 2 to 3 percentage points lower in all but one month. CPS employment increases by 3.5 million between January and July and ends the year down 0.7 million from its peak.
|
From page 137... ...
For waves 2 and later of the 2014 panel, the data collected for the early part of the reference period during the prior wave should eliminate the effects of recall bias on reported program participation during those months. Whether dependent interviewing -- reminding respondents of program participation that they reported in the previous interview -- reduces recall bias over the remaining months of the reference period remains to be seen.
|
From page 138... ...
Whereas 25 percent of transitions, on average, might be expected to fall between the fourth month of the previous wave reference period and the first month of the current wave reference period, this fraction for participation in many programs exceeded 80 percent and approached 100 percent for some programs. Table 7-14 reports the percentages of transitions into and out of selected programs, employment, and poverty that occurred at the seams between waves in the final 12 months of the 2008 SIPP panel.
|
From page 139... ...
SOURCE: Panel generated with data from the 2008 SIPP panel (public-use data file)
|
From page 140... ...
Selected states have begun to submit microdata files to the Census Bureau, but these have not been used as yet to generate statistics on transitions. The quality control sample data can be used to estimate new beneficiaries, and this report makes use of these data, but as a sample of active cases these data contain no information on program exits.
|
From page 141... ...
Table 7-9 showed that the 2014 panel understated the unemployment compensation caseload by a much greater margin early than late in the reference year, and this is reflected in Table 7-15 by transitions that start low in February and rise and flatten out over the next few months before rising more steeply at the end of the year. As a fraction of the administrative estimates, the 2014 SIPP estimates are roughly similar to the 2008 panel estimates from April through the end of that series in August.
|
From page 142... ...
be diminished in future waves, due to a combination of data collected for the first part of the reference period in the prior wave and the use of dependent interviewing. With this expectation, the findings for unemployment compensation suggest that the 2014 SIPP panel may ultimately capture transitions at least as well if not better than the 2008 panel.
|
From page 143... ...
The fact that the underlying estimates of labor force status are based on reports for the survey week makes the estimates of gross change less susceptible to recall error than those from either SIPP panel. In addition, the labor force questionnaire itself is very brief, and respondents are being asked the same questions that nearly three-quarters of them answered the previous month.
|
From page 144... ...
Table 7-17 compares 2008 and 2014 SIPP panel estimates of transitions into and out of employment to the BLS estimates based on the CPS. The 2008 panel estimates of transitions into employment range from 50 to 68 percent of the BLS estimates, with a mean of 61 percent, while the 2014 panel estimates run from 19 to 42 percent of the BLS estimates, with a mean of 29 percent.
|
From page 145... ...
bThe 2008 panel estimates have been adjusted to compensate for the absence of off-seam data from one rotation group. SOURCE: Panel generated with data from the Bureau of Labor Statistics (available: https://www.bls.gov/webapps/legacy/cpsflowstab.htm)
|
From page 146... ...
The study panel notes as well that transitions out of employment have a substantial peak in September in the 2014 SIPP panel -- consistent with students leaving summer employment to return to school -- whereas the peak in the BLS series occurs in August. As with transitions into employment, this 1-month shift in timing may reflect the differences in how employment is measured in the two surveys.
|
From page 147... ...
, by Month, 2013 2008 Panel 2014 Panel Ratio of 2014 to 2008 Transitions Transitions Transitions Transitions Transitions Transitions into out of into out of into out of Month Medicaid Medicaid Medicaid Medicaid Medicaid Medicaid January 2,221 2,440 February 2,389 2,391 224 119 0.09 0.05 March 2,196 2,333 191 104 0.09 0.04 April 2,405 2,211 137 149 0.06 0.07 May 2,308 2,385 150 146 0.06 0.06 Junea 326 271 Julyb 2,035 2,303 251 207 0.12 0.09 Augustb 2,304 2,429 417 256 0.18 0.11 Septembera 247 205 Octobera 381 240 Novembera 395 333 Decembera 318 213 Average 2,265 2,356 276 204 0.10 0.07 aSeam transitions were not collected in the 2008 panel for this month, so no estimates of transitions from that panel are reported. bThe 2008 panel estimates have been adjusted to compensate for the absence of off-seam data from one rotation group.
|
From page 148... ...
One possibility is that 2014 panel respondents reported their Medicaid enrollment at the time of their interviews to the same degree of accuracy as 2008 panel respondents but rarely reported any change in that enrollment between then and the previous January. In our examination of monthly reporting of program participation, the panel found little change over the year in most programs, including Medicaid.
|
From page 149... ...
, by Month, 2013 2008 Panel 2014 Panel Ratio of 2014 to 2008 Entrances Entrances Entrances into Exits from into Exits from into Exits from Month Poverty Poverty Poverty Poverty Poverty Poverty January 4,175 4,752 February 4,849 4,254 5,057 1,661 1.04 0.39 March 3,750 4,142 1,079 4,782 0.29 1.15 April 3,733 4,483 1,945 2,065 0.52 0.46 May 3,315 4,530 1,569 2,957 0.47 0.65 Junea 2,899 2,597 Julyb 4,350 4,680 2,184 2,736 0.50 0.58 Augustb 4,098 4,709 2,090 2,809 0.51 0.60 Septembera 3,174 3,044 Octobera 1,606 3,475 Novembera 2,976 2,024 Decembera 2,352 2,841 Average 4,039 4,507 2,448 2,817 0.56 0.64 aSeam transitions were not collected in the 2008 panel for this month, so no estimates of transitions from that panel are reported. bThe 2008 panel estimates have been adjusted to compensate for the absence of off-seam data from one rotation group.
|
From page 150... ...
, there is no alternative source that can be consulted to determine which SIPP panel is more correct in its estimates of poverty transitions. The panel has noted concerns that the frequent interviews in the older SIPP could produce overstated transitions.
|
From page 151... ...
The wave 2 response rate -- among households that responded to the first wave and remained eligible -- was 74.2 percent or nearly 18 percentage points lower than the wave 2 response rates for the previous three SIPP panels. The wave 3 response rate of 59.4 percent represented a wave-to-wave plunge unlike any seen in a recent SIPP panel, where the largest previous decline between consecutive waves was 4.4 percentage points, recorded between waves
|
From page 152... ...
152 THE 2014 REDESIGN OF SIPP TABLE 7-20 Response Rates and Cumulative Sample Loss by Wave, 2001-2014 Survey of Income and Program Participation Panels Nonresponse Rates Eligible Interviewed Response Cumulative Wave Holds Holds Rate Type A Type D Sample Loss 2014 Panel 1 42,348 29,685 70.1 29.9 -- 29.9 2 31,085 23,060 74.2 21.9 3.9 48.0 3 32,119 19,090 59.4 33.7 6.9 58.3 2008 Panel 1 52,031 42,032 80.8 19.2 -- 19.2 2 42,481 39,000 91.8 6.9 1.3 25.9 3 42,779 37,651 88.0 9.7 2.3 29.0 4 43,176 36,195 83.8 13.2 2.9 32.4 5 43,422 35,873 82.6 14.0 3.3 33.3 6 43,544 34,891 80.1 15.9 4.0 35.5 7 43,619 33,827 77.6 18.2 4.2 37.5 8 43,609 33,417 76.6 19.0 4.3 38.2 9 43,621 32,567 74.7 20.4 4.7 39.7 10 43,690 31,445 72.0 22.7 5.1 41.9 11 43,720 31,007 70.9 23.5 5.3 42.7 12 43,678 30,716 70.3 24.0 5.6 43.4 13 43,654 30,213 69.2 25.2 5.6 44.4 14 43,600 29,810 68.4 26.0 5.5 44.9 15 43,653 28,885 66.2 27.5 5.8 46.5 16 32,566 20,135 61.8 31.4 6.1 53.0 2004 Panel 1 51,363 43,711 85.1 14.9 -- 14.9 2 44,150 40,587 91.9 6.6 1.4 21.9 3 44,614 39,117 87.7 9.9 2.5 25.5 4 44,930 38,309 85.3 11.6 3.1 27.6 5 45,350 37,446 82.6 13.7 3.7 29.8 6 45,638 36,931 80.9 15.0 4.1 31.2 7 45,688 36,289 79.4 16.1 4.5 32.5 8 45,684 35,966 78.7 16.1 5.2 33.1 9 21,296 16,587 77.9 16.9 5.2 34.0 10 21,342 16,235 76.1 18.5 5.3 35.5 11 21,347 15,894 74.5 19.7 5.7 36.9 12 21,332 15,952 74.8 18.9 6.4 36.6 2001 Panel 1 40,489 35,102 86.7 13.3 -- 13.3 2 30,514 28,086 92.0 6.4 1.7 21.9 3 30,899 27,453 88.8 8.6 2.7 24.7 4 31,111 27,179 87.4 9.5 3.2 25.9 5 31,300 26,775 85.5 10.9 3.6 27.5
|
From page 153... ...
Fields, U.S. Census Bureau.
|
From page 154... ...
The supplement targeted all respondents from wave 1 and was implemented as a telephone survey with a fairly limited nonresponse follow-up. A telephone survey is generally expected to have lower response rates than a faceto-face survey, and staff from the Census Bureau and the SSA discussed those expectations.
|
From page 155... ...
With its longer reference period, the 2014 panel collects the most recent monthly amounts and then asks only about changes to these amounts. Because of these different approaches, imputation rates cannot be compared between the 2014 and earlier SIPP panels for many sources of income.
|
From page 156... ...
However, in comparison with the CPS ASEC, the 2014 SIPP panel has lower imputation rates for 16 of 17 sources. For many items the imputation rates in the CPS ASEC -- the official source for estimates of household income and poverty statistics in the United States -- are nearly twice as high as they are in the 2014 SIPP panel.
|
From page 157... ...
Finding 7-5: Reported health insurance coverage is virtually unchanged between the 2008 and 2014 SIPP panels. Finding 7-6: Compared to the 2008 panel, the 2014 SIPP panel captures less income at the very bottom of the income distribution, as reflected in higher estimates of the monthly percentage of the population in families below 50 percent of the federal poverty threshold.
|
From page 158... ...
Poverty spikes in February, with 5.1 million persons entering poverty in that month and 4.8 million exiting the next month. Finding 7-12: Reflecting a secular trend of declining survey response rates, the wave 1 response rate for the 2014 SIPP panel was more than 10 percentage points below the wave 1 response rate of the 2008 panel.
|
From page 159... ...
However, imputation rates for 16 of 17 sources are lower in the 2014 SIPP panel than in the 2014 CPS ASEC, with the rates in the latter being nearly twice as high in many cases. Finding 7-14: Misreporting of the unit of measure caused outliers in the 2014 SIPP wage estimates.
|
From page 160... ...
RECOMMENDATION 7-4: The Census Bureau should conduct research to identify why the redesigned Survey of Income and Program Participation (SIPP) has difficulty relative to the earlier SIPP panels in accurately capturing income from households at the bottom of the income distribution.
|
From page 161... ...
The research should evaluate an option for restructure that would record all income and program participation spells before asking any follow-on questions, with a mechanism to review cumula tive spells before leaving the EHC. If the research to restructure the EHC does not lead to a marked improvement in the accurate collec tion of spells, the Census Bureau should explore the return to a shorter reference period.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.