The Future of the Survey of Income and Program Participation (1993) / Chapter Skim
Currently Skimming:
4 SURVEY DESIGN
4 SURVEY DESIGN
Pages 91-130
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 91... ...
In this section we consider the following major design elements: · the number of interviews or waves in each panel; · the length of the reference period covered by each interview; · the length of each panel (a function of the number of interviews and the reference period length) ; · the frequency with which new panels are introduced; and · the total initial sample size for each panel.
|
From page 92... ...
Given a fixed budget that puts a ceiling on the number of interviews that can be fielded each year, a change in one of the design elements will generally necessitate an offsetting change elsewhere. For example, an increase in panel length must be offset by one or more of the following changes: a reduction in the frequency with which new panels are introduced, a reduction in the sample size per panel, or an increase in the reference period length for each interview wave.
|
From page 93... ...
There is currently no oversampling of specific population groups in SIPP, with one exception: the 1990 panel includes about 3,800 extra households continued from the 1989 panel, selected because they were headed by blacks, Hispanics, or female single parents at the first wave of the 1989 panel. The initial sample size for the first 1984 SIPP panel was about 21,000 eligible households, with the expectation that, by combining two panels of that size, users would be able to obtain a total sample size of about 37,00038,000 households.2 However, budget cuts necessitated an 18 percent reduction in the sample size midway through the 1984 panel (beginning with wave 5~.
|
From page 94... ...
Users most often suggest extending SIPP panels to 5 years, although some users would be satisfied with extending them to 4 years; at least one user has suggested lengthening SIPP panels to 10 years to permit the data to be used to study welfare dependency and persistent poverty (Manski, 1991~. In order to increase sample size and panel length, many users of the longitudinal data say they are willing to live with longer reference periods for each interview, thereby decreasing the number of interviews per year, typically from three 4-month to two 6-month waves.
|
From page 95... ...
. Similarly, Bureau staff made the original decision to have reference periods of 4 months, instead of 6 or 3 months, as a compromise between the need for accurate monthly data and reduced cost of field operations.
|
From page 96... ...
5However, in the section on sample design considerations, we discuss extending the length of SIPP panels for a longer period than whatever is the standard length for the full samplefor subgroups of interest as a means of adding sample size and longitudinal information for the subsampled groups. 6Attrition will reduce the number of required interviews: eligible households that do not respond in the first wave are dropped from the sample; eligible households that subsequently fail to respond are pursued for one more interview before being dropped.
|
From page 97... ...
(Two interviews times two panels times 40,000 equals 160,000 interviews per year.) We initially considered another very different design that strives to reconcile the widely voiced desire for larger sample size with the view that cross-sectional uses require short reference periods and frequently refreshed samples (Doyle, 1992~.
|
From page 98... ...
Design B retains the 4-month reference period of the current design, but provides fewer additional sample cases than the other designs. Our challenge was to assess the implications of these design choices for the "bottom line": the ability of SIPP to provide high-quality, relevant data for research and policy analysis related to income and program participation.
|
From page 99... ...
We also looked at the negative effects of more frequent panels, one of which is a reduction in sample size available for longitudinal analysis of single panels. Negative effects can also stem from what we term the "complexity factor": specifically, having multiple panels in progress at the same time can increase the burden on interviewers and data processing operations, which, in turn, can introduce errors and reduce timeliness of data products.
|
From page 100... ...
. Panel surveys are also subject to wave nonresponse, or attrition, at each successive interview.9 9More precisely, total sample loss at each interview, or total wave nonresponse, includes attrition per se, that is, nonresponse by households that are never brought back into the survey, plus nonresponse of households that miss a wave but are successfully interviewed at the next wave.
|
From page 101... ...
Evidence on Attrition To date, the wave nonresponse rates from SIPP show a definite pattern (see Table 4-1~. Total sample loss in the 1984-1988 and 1990 panels is highest at the first and second interviews 5-8 percent of eligible households at wave 1 and an additional 4-6 percent of eligible households at wave 2.
|
From page 102... ...
The 1979 ISDP research panel experienced a total sample loss of 18.1 percent after 6 waves (with 3-month reference periods) , compared with sample loss rates for the SIPP 1984-1988 panels of 18-20 percent after 6 waves (with 4-month reference periods)
|
From page 103... ...
Several studies of the PSID in the 1980s found that, although cumulative sample loss by that time was over 55 percent, there was no evidence that attrition correlated with individual characteristics in a way that would produce biased estimates.~3 For example, Becketti et al.
|
From page 104... ...
It is important to note that current cross-sectional nonresponse adjustments in SIPP make only minimal use of the information that is available from previous waves for many current nonrespondents. Also, in constructing longitudinal files from SIPP panels, the Census Bureau assigns zero weights to original sample members who missed only one or a few waves in addition to those who missed all or most waves.
|
From page 105... ...
Lepkowski, Kalton, and Kasprzyk (1990) compared responses from wave 4 of the 1984 panel with wave 1 of the 1985 panel for original sample
|
From page 106... ...
compared quarterly estimates of earnings, labor force activity, poverty, and program participation for 19851987, using data from the 1984-1987 SIPP panels.~4 In general, they found little evidence of time-in-sample effects. They did find significant differences occurring across panels when comparing estimates for the first quarter of each year, indicating that there may be systematic differences between wave ~ and subsequent interviews.
|
From page 107... ...
, as were 27 percent of spells without health insurance (McBride and Swartz, l990:Table 1~.is Extending the panel length from 32 to 48 months would not only enlarge the sample size of spells of program participation, poverty, and other states, but de 15Note that these figures include spells that were nght-censored because of sample reduction in the 1984 panel, as well as those right-censored because the panel ended. The duration of another 15 percent of AFDC spells and 12 percent of spells without health insurance that began in the 1984 SIPP panel was not observed because the respondents dropped out of the survey.
|
From page 108... ...
Recall Error and Seam Effects One of the concerns of the ISDP that laid the groundwork for SIPP was the appropriate recall or reference period for each interview. The goal for SIPP was to obtain improved estimates of annual income, compared with the March CPS, and to obtain estimates of subannual income and program participation that could be related to administrative data.
|
From page 109... ...
The seam phenomenon affects most variables for which monthly data are collected in SIPP-often strongly. For example, in the first year of the 1984 SIPP panel, four times as many social security participants reported exiting the program between months that spanned interviews as between months within the reference period of a single interview.
|
From page 110... ...
Overlapping Panels One of the major features of the current SIPP design the yearly overlapping of panels was adopted with the goal of maximizing sample size and minimizing the effects of attrition and time-in-sample biases for cross-sectional estimates. Introducing new SIPP panels annually afforded users the
|
From page 111... ...
More recently, the Bureau has issued reports on income and program participation, using complete panel files, that have focused on such longitudinal issues as duration of participation and year-to-year change in economic status. The Bureau's future plans for a regular report series from the core data include cross-sectional as well as longitudinal statistics (see Chapter 6~.
|
From page 112... ...
Sample Size A pervasive complaint about SIPP panels is that they provide too few sample cases for observation of subgroups of policy interest for example, recipients of food stamps and AFDC. We made rough estimates of the number of cases for these two populations that would be available for analysis by the end of waves 1, 4, 8, and 12 from SIPP panels of different size initial samples (without any oversampling of low-income groups or combimng of panels)
|
From page 113... ...
and assume that: food stamp recipient units are 10 percent of total households and AFDC recipient units are 4 percent of total households and attrition is a function of the number of waves: cumulative attrition is 7 percent of the initial sample size at wave 1, 15 percent at wave 4, 21 percent at wave 8, and 25 percent at wave 12. Results are rounded to the nearest 10 and are labeled as "minimum sample sizes" because no account is taken of the increase in sample cases that is likely to occur due to household formation by original sample members or that could be obtained by combining panels or by oversampling low-income people.
|
From page 114... ...
Designs A, C, and D also have this problem for the last year of each panel.21 These missing data will impair the ability to construct reliable estimates for the calendar year involved or for year-to-year compansor~s.22 20Given monthly rotation groups and 4-month interview waves, the recall length for calendar-period cross-sectional estimates will vary across the sample: for example, for a calendarmonth estimate, the recall period will vary from 1 to 4 months, with an average recall length of 2-1/2 months. In contrast, if all interviews were conducted at the end of each 4-month reference period, different calendar months would have different recall lengths (e.g., 4 months for January, May, September; 3 for February, June, October, etc.)
|
From page 115... ...
SURVEY DESIGN TABLE 4-3 Reference Penods for Rotation Groups for SIPP Redesign 115 Begin in February Begin in March Reference Rotation Group Rotation Group Month and Year 1 2 3 4 1 2 3 4 Year0 October 1 November 1 1 1 December 1 1 1 1 1 Year 1 January 1 1 1 1 1 1 1 1 February 2 1 1 1 1 1 1- 1 March 2 2 1 1 2 1 1 1 Apnl 2 2 2 1 2 2 1 1 May 2 2 2 2 2 2 2 1 June 3 2 2 2 2 2 2 2 July 3 3 2 2 3 2 2 2 August 3 3 3 2 3 3 2 2 September 3 3 3 3 3 3 3 2 October 4 3 3 3 3 3 3 3 November 4 4 3 3 4 3 3 3 December 4 4 4 3 4 4 3 3 Year2 January 4 4 4 4 4 4 4 3 February 5 4 4 4 4 4 4 4 March 5 5 4 4 5 4 4 4 Apnl 5 5 5 4 5 5 4 4 May , 5 5 5 5 5 5 5 4 June 6 5 5 5 5 ~5 5 July 6 6 5 5 6 5 5 5 August 6 6 6 ~6 6 5 5 September 6 6 6 6 6 6 6 5 October 7 6 6 6 6 6 6 6 November 7 7 6 6 7 6 6 6 December 7 7 ~6 7 7 6 6 Year 3 January 7 7 7 7 7 7 7 6 February 8 7 7 7 7 7 7 7 March 8 8 7 7 8 7 7 7 April 8 8 8 7 8 8 7 7 May 8 8 8 8 8 8 8 7 June 9 8 8 8 8 8 8 8 July 9 9 8 8 9 8 8 8 August 9 9 9 8 9 9 8 8 September 9 9 9 9 9 9 9 8 October 10 9 9 9 9 9 9 9 November 10 10 9 9 10 9 9 9 December 10 10 10 9 10 10 9 9 Year 4 January 10 10 10 10 10 10 10 9 February 11 10 10 10 10 10 10 10 March 11 11 10 10 11 10 10 10 Apnl 11 11 11 10 11 11 10 10 May 11 11 11 11 11 11 11 10 continued on next page
|
From page 116... ...
(In the current design, the seventh and eighth interviews, which result in three panels in the field for most of each year, serve, respectively, to complete the monthly data for the second calendar year of a panel and to obtain the annual income and tax information for that year.) The annual income roundup provides useful information with which to validate the monthly amounts but is not used directly in estimates, so that its omission for the last year is not serious.
|
From page 117... ...
The sample size for each panel should be increased over that for the current design. 24The complication arising from having different reference period lengths for different rotation groups should be manageable with a CAPI mode of data collection, as is planned for SIPP.
|
From page 118... ...
;25 · reduces the frequency of introduction of new panels to every 2 years, which relieves the pressures on the Census Bureau of dealing with three panels each year-one of them new without, we believe, adding matenally to the biases in cross-sectional estimates; and · increases the sample size for cross-sectional and longitudinal analysis: the initial sample of 26,700 households per panel is double the size of the 1985-1989 SIPP panels and one-third more than the 20,000-household size originally planned for SIPP. We heard strong arguments for extending the panel length and increasing the sample size even more than in design B
|
From page 119... ...
However, the Census Bureau will need to consider the pros and cons of alternative ways to obtain complete data for the fourth calendar year of each panel for the affected rotation groups. It will be important to determine a cost-effective solution in order to enable SIPP to provide a reliable time series of cross-sectional estimates of income and program participation.
|
From page 120... ...
The 1970 and 1980 censuses formed the basis of the sample design and selection of census addresses for the 1984 and 1985-1994 SIPP panels, respectively (see Jabine, King, and Petroni [l990:Ch.
|
From page 121... ...
When the new sample design is introduced in 1995, it is expected that the census address portion of the sample will constitute about 70 percent of the total and the area frame portion about 20 percent. The remaining 10 percent will represent addresses of new construction, for which no oversampling will be performed; obviously, over the course of a 10-year period, this category will grow as a proportion of the total.
|
From page 122... ...
For example, it could be useful to conduct research on the extent to which the household poverty classification of addresses in the 1985 SIPP panel corresponds to the 1980 census classification.28 There is also no opportunity to change any aspect of the design because the Census Bureau plans to draw 10 years' worth of sample for SIPP (and other household surveys) at the same time.
|
From page 123... ...
For the redesign, the Census Bureau is essentially defining a cohort of low-income people on the basis of their previous year's household income-to-poverty ratio. However, many people with low incomes at wave 1 will move into a higher income category over the life of a SIPP panel and vice versa (see Short and Littman, 1990; Short and Shea, 1991~.
|
From page 124... ...
There are also ways to reduce costs. It may be possible to conduct much of the screening using a centralized CATI system that eliminates interviewer travel costs.29 Another way to reduce costs is to treat the screening interview as wave 1 of a SIPP panel instead of as an added interview.
|
From page 125... ...
Another possibility is to extend the length of one or more SIPP panels for subgroups of interest. This strategy both provides additional longitudinal information for the subsampled cases and makes it possible to treat them as an addition to the sample for the next panel (see David, 1985a)
|
From page 126... ...
Recommendation 4-~: The Census Bureau should take steps to ensure that it will be possible to extend the length of SIPP panels for selected subgroups of interest or to follow them up at a later date, should such options be desired to obtain increased sample size and longitudinal information. Multiple-Frame Samples Yet another way to obtain an additional sample for subgroups of interest in SIPP is to develop multiple-frame samples, that is, samples of households together with cases that are drawn from one or more types of administrative records for example, program records, tax records, or employer records.30 Augmenting a household sample with cases from administrative records can offer considerable benefits.
|
From page 127... ...
(For example, in the case of a dual-frame sample including SSI cases drawn the August before the start of a SIPP panel, the questionnaire would ask about receipt of SSI in the preceding August.) A possibly more reliable approach is to match the household sample members with the administrative frame.
|
From page 128... ...
We suggest that a decision to adopt this means of oversampling, particularly in light of the operational and technical difficulties it would pose, should be contingent on the support and cooperation of an interested agency. We encourage the Census Bureau staff to keep up to date on the methodology of multiple-frame samples, so that SIPP can be responsive to requests from agencies that want to obtain a larger sample size and information for a particular population by adding a component to the SIPP sample that is drawn from their records.32 FOLLOWING RULES At present, SIPP follows original sample adults that is, all people aged 15 and older who resided in an interviewed household in wave 1 for the life of a panel or until they leave the universe or drop out of the survey.
|
From page 129... ...
We urge the Census Bureau to treat all children present in interviewed households at wave 1 together with children born subsequently to original sample mothers as original sample members who are followed throughout the life of a panel. When original sample children move into a household of nonsample members, information would be obtained about them and about other members of their new household.
|
From page 130... ...
Recommendation 4-6: SIPP panels should treat all children who reside in interviewed households at the first wave and also children born during the course of a panel to original sample mothers as original sample members, who are followed if they move into households without an original sample adult. SIPP panels should also continue to follow and collect data for both original sample adults and children if they move into institutions.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.