The Future of the Survey of Income and Program Participation (1993) / Chapter Skim
Currently Skimming:
7 METHODOLOGICAL RESEARCH AND EVALUATION
7 METHODOLOGICAL RESEARCH AND EVALUATION
Pages 202-226
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 202... ...
Looking to the future, we believe that SIPP will continue to require methodological research on many aspects of the program.2 The Census Bureau will need information in the near teens for many details of the proposed redesign. Subsequently, the Bureau will need information on the impact of the redesign to guide research and experimentation directed to further improvements in the survey.
|
From page 203... ...
We conclude with a detailed discussion of a recently inaugurated program of cognitive research on the SIPP questionnaire. This innovative work shows great promise to improve data quality although it presents difficult questions of implementation and integration with other planned improvements for SIPP, such as computerassisted personal interviewing (CAPI)
|
From page 204... ...
Methods and Results The Census Bureau has used a range of techniques for methodological research and evaluation, including: · small-scale and large-scale field experiments with changes in procedures or question wording (experiments were conducted with telephone interviewing to reduce costs, gifts to reduce attrition, collecting data on employer-provided benefits, different procedures to reduce the seam problem,
|
From page 205... ...
Whatever the outcome for SIPP operations, in almost all instances re 5The missing wave module was designed to fill in information for people who had missed the preceding wave but were interviewed at the wave prior to that one. The module, which was administered beginning in wave 4 of the 1984 panel and discontinued midway through the 1986 panel, asked an abbreviated set of questions on labor force status, program participation, income receipt, and asset ownership for the reference period covered by the preceding wave.
|
From page 206... ...
In this section we discuss the components of a program of methodological research that we believe the Census Bureau should put in place to inform and evaluate the SIPP redesign. These components include: research to improve the format and wording of the questionnaire; research targeted to other aspects of the redesign (e.g., implementation of CAPI)
|
From page 207... ...
We are sure that the community of users will have suggestions for the questionnaire as well. We understand that the Census Bureau hopes by the end of the year to decide on the content changes in the core questionnaire that will be implemented as part of the redesign It appears quite reasonable to adopt this schedule for determining topic areas and the general level of detail in the core, particularly given that the staff who are working to design the CAPI and database management systems for SIPP need this information.
|
From page 208... ...
Another advantage of forward record checks for questionnaire research purposes is that, for many research objectives, it is not necessary to obtain a nationally representative sample. lIence, in the case of state-administered programs, for which there is wide variation in the quality and accessibility of administrative records, the samples could be selected from those states with the best systems.7 Finally, forward record checks can be implemented on a timely basis because there is no need for an after-the-fact match of the administrative and survey data.
|
From page 209... ...
Other Aspects of the Redesign Other aspects of the SIPP redesign for which it would be useful to conduct methodological research (including design changes that could be appropriate to implement somewhat later on) are the length of the recall period, oversampling based on screening, implementation of CAPI, and telephone interviewing.
|
From page 210... ...
As part of the plan to convert to CAPI, the Census Bureau will need to consider the appropriate role of personal versus telephone interviewing in SIPP. Research is needed on the implications for costs and data quality of continuing the current mode of maximum telephone interviewing in waves 3-5 and 7-8 versus reverting to heavier use of in-person interviews.
|
From page 211... ...
One avenue to pursue is research on the benefits of more extensive use of wave 1 income and program participation variables to adjust for attrition in subsequent waves. The need for further research on longitudinal weighting and imputation is even more critical.
|
From page 212... ...
The use of imputation seems particularly promising for cases with only one or two missing waves, in which the missing waves are bounded by interview data for the preceding and succeeding waves. For another longitudinal estimation issue, we urge the Census Bureau to conduct research on appropriate weighting for analyses of spell duration (spells of low income or program participation)
|
From page 213... ...
To support research and development work in this area, the database management system chosen for SIPP (as noted in Chapter 5) should permit ready implementation of alternative imputation procedures.
|
From page 214... ...
Household surveys other than SIPP experience substantial undercoverage for some population groups, but undercoverage may be particularly important for SIPP with its two main goals of improving information on income and programs. We urge the Census Bureau to conduct research on population undercoverage in SIPP, including simulation studies to assess the sensitivity of SIPP estimates to alternative procedures for adjusting for undercoverage The goal of such research should be to develop improved 14Clogg, Massagli, and Eliason (1986)
|
From page 215... ...
Full record checks that match SIPP sample cases, including reporters and nonreporters, with administrative records to assess net reporting error, including both underreports and overreports, would also be useful to conduct periodically.ig The proposed redesign entails a significant extension in the length of SIPP panels from 32 to 48 months. We do not expect that cumulative attrition will increase very much because the available evidence is that most attrition occurs in the first few waves of a panel.
|
From page 216... ...
Hence, it is only prudent for the Census Bureau to have contingency plans to permit timely assessment and corrective action in the event that a serious problem occurs with the redesign. Recommendation We believe that methodological research and evaluation is needed to inform and assess the SIPP redesign.
|
From page 217... ...
, including the length of the recall period, screening techniques to obtain larger sample sizes for subgroups of interest, and data collection modes (the best combination of computer-assisted personal and telephone interviewing and the possible role of centralized telephoning) ; · research on issues of estimation and data use, taking into account the features of the redesign and including ways to improve cross-sectional and longitudinal weights, imputation procedures, and population coverage; · research to evaluate the success of major elements of the redesign (e.g., the attrition effects of longer panels)
|
From page 218... ...
Also, it would be very useful to periodically conduct reverse or full record-check studies based on matching SIPP panels with administrative records (e.g., tax returns or program case records)
|
From page 219... ...
As in the past, these documents should include error analyses that originate from both Census Bureau staff and outside users. Recommendation 7-2: The Census Bureau should undertake continuous monitoring of error levels in present and future SIPP panels and regularly provide information on errors to users, in periodic updates of the SIPP Quality Profile and other publications.
|
From page 220... ...
Federal statistical agencies, including the National Center for Health Statistics, the Bureau of Labor Statistics, and the Census Bureau, are now making considerable use of cognitive techniques for questionnaire research and experimentation, such as one-on-one sessions in which a researcher probes the respondent after each question to ask what he or she had in mind in answering. Results The Census Bureau recently began applying cognitive techniques to the task of improving SIPP measures of income and program participation (see Marquis, Moore, and Bogen, 1991~.
|
From page 221... ...
has begun to produce findings of relevance to understanding response errors in SIPP (see Cantor et al., 1991~. The research involved recruiting 125 respondents, about one-half of whom were participants in some type of government program, to receive wave 1 and wave 2 SIPP interviews.
|
From page 222... ...
Many respondents in the Westat study made errors in reporting earningsbased on comparing responses to the SIPP questions with an additional detailed recall most commonly because of misdating or miscounting paychecks. Most respondents find it fairly easy to remember amounts for program participation, but very difficult to retrieve amounts for assets.
|
From page 223... ...
Income sources are grouped in ways that seem likely to make sense to respondents: for example, asking about money from the military, including veterans' payments, military retirement, National Guard pay, and GI bill benefits; or asking about "surprises," such as an inheritance, lottery win nings, profits from gambling, insurance settlements, and work-related bonuses or awards. At subsequent interviews, to help reduce response errors, dependent interviewing techniques (reminding respondents of prior wave responses)
|
From page 224... ...
The promise for a very different way of relating to respondents that obtains high-quality responses seems strong. We were not in a position to comment on specific details of the current program of field testing and experimentation with alternative questionnaires and interviewing procedures because the program was in the very early stages at the time of our deliberations.
|
From page 225... ...
Should the technique of free recall of income sources, using worksheets to record payment streams, prove effective in the field, its use will have major implications for SIPP data processing. It seems possible that the equivalent of worksheets could be built into a CAPI system and that the necessary computations to produce monthly incomes from the individual payment records could be made within a database management system.
|
From page 226... ...
Finally, as planned for the upcoming evaluation study in Milwaukee, all tests should build in the means to evaluate fully the effects of the new procedures on costs and data quality. Recommendation 7-3: We strongly support the Census Bureau's program of cognitively based research and experimentation with the SIPP questionnaire, which could contribute to questionnaire improvements for the current redesign and perhaps, in the future, to a major revision of the questionnaire and interviewing procedures.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.