Click for next page ( 203


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 202
7 Methodological Research and Evaluation One of the undoubted strengths of the SIPP program, including its forerun- ner, the Income Survey development Program (ISDP), has been the extent of research on the quality of the data and ways to enhance quality (and efficiency) through improving the design and operation of the survey. In- deed, a strong research and evaluation component was essential dunug the planning and start-up phases because of SIPP's complex nature and broad scope. Looking to the future, we believe that SIPP will continue to require methodological research on many aspects of the program.2 The Census Bureau will need information in the near teens for many details of the pro- posed redesign. Subsequently, the Bureau will need information on the impact of the redesign to guide research and experimentation directed to further improvements in the survey. In addition, users will need continuing information on data quality to make the most appropriate use of the survey information. 1To assist it in determining priorities for research and evaluation and in designing specific projects, the Census Bureau has consulted with the members of the Working Group on Techni- cal Aspects of SIPP, sponsored by the Survey Research Methods (SUM) Section of the Amer- ican Statistical Association (ASA). 2We note also the importance of research on analytical measurement issues, such as those discussed in Chapter 6 (e.g., the definition of spell length and whether to include all spells observed in a panel in an analysis of duration), and we believe strongly (see Chapter 8) that the Census Bureau should give increased attention to analytical issues for SIPP. 202

OCR for page 202
METHODOLOGICAL RESEARCH AND EVALUATION 203 In this chapter we first briefly review the scope and accomplishments to date of the SIPP methodological research program at the Census Bureau.3 We then outline research strategies and summarize priority topics (many of which we discuss in other chapters) for which further methodological work is indicated, under two main headings: research to inform and evaluate the redesign and continuous monitoring of error levels for the benefit of ana- lysts both inside and outside the Bureau. We conclude with a detailed discussion of a recently inaugurated program of cognitive research on the SIPP questionnaire. This innovative work shows great promise to improve data quality although it presents difficult questions of implementation and integration with other planned improvements for SIPP, such as computer- assisted personal interviewing (CAPI). RESEARCH TO DATE Topics From the very early days of designing a new income survey, the Census Bureau has been concerned with identifying and conducting research on a wide range of methodological issues that were critical for SIPP to achieve its goals of providing improved data on income and program participation.4 The ISDP investigated a number of important topics, including: the length of the recall period; respondent rules (self- versus proxy reporting); alternative questionnaire designs (e.g., short versus long forms of the income receipt and amounts questions; household screening format versus person-by-person format); mover follow-up rules; and the definition of longitudinal units. The abrupt termination of the ISDP program in 1981 left many issues in venous stages of resolution. Initial methodological research and evaluation at the Census Bureau during the first 2 years of SIPP concentrated on the following topics: 3We note that important methodological research and evaluation on SIPP has been conduct- ed by analysts outside the Census Bureau; examples are Curiin, Juster, and Morgan (1989); Doyle and Dalrymple (1987); and Vaughan (1988). 4This section draws heavily on Committee on National Statistics (1989:Ch. 7), which, in turn, benefited greatly from a presentation by Daniel Kasprzyk to the Committee on National Statistics subcommittee on SIPP. See the SIPP Quality Profile (Jabine, King, and Petroni, 1990) for references and summaries of findings for methodological studies conducted on SIPP; see also Kasprzyk (1988b) and Petroni, Huggins, and Carmody (1989). David (1983) provides references to methodological studies conducted on the ISDP.

OCR for page 202
204 THE SURVEY OF INCOME AND PROGRAM PARTICIPATION wave nonresponse and its treatment; field considerations, including the impact of respondent rules in gen- eral, respondent rules for students, and mover follow-up rules; the "seam problem" (i.e., the observed tendency to report more tran- sitions between months marking the boundary between two interviews than between months included within a single interview); comparisons of survey responses with administrative records data; and issues involved in developing and analyzing longitudinal data from SIPP, including imputation, estimation, and household and family-concepts. As more SIPP data became available for analysis, other issues pertinent to the quality and utility of the information emerged and gained in impor tance: the pervasiveness of the seam problem; sample attrition; problems with the independent collection of industry and occupation data from one wave to the next; the lack of baseline data (e.g., the lack of information in the first interview for program participants on when the participation spell began); the lack of data on employer-provided benefits; and the need to provide users with measures of quality through such means as comparisons of SIPP cross-sectional estimates with independent sources. Still later, yet additional methodological concerns surfaced, including: problems with constricting program eligibility measures given that key data were scattered across topical modules; and the possibility of time-in-sample bias (i.e., systematic differences in responses by individuals between later and earlier interviews). At the same time, budget cutbacks, which necessitated restricting sample size and the number of interviews, motivated research on ways to compen- sate for diminished sample size and to control costs. Methods and Results The Census Bureau has used a range of techniques for methodological re- search and evaluation, including: small-scale and large-scale field experiments with changes in proce- dures or question wording (experiments were conducted with telephone in- terviewing to reduce costs, gifts to reduce attrition, collecting data on em- ployer-provided benefits, different procedures to reduce the seam problem,

OCR for page 202
METHODOLOGICAL RES~RCH AND EVALUATION 205 and providing respondents with prior-year asset responses to improve re- porting of changes in asset holdings); comparisons of SIPP responses with administrative records on an individual match basis (most notably, the record-check study [see Marquis and Moore, 1989, 1990a, l990b]), comparisons of aggregates from SIPP with those from other surveys and administrative records; internal analysis of SIPP data (e.g., an evaluation of the effective- ness of the cross-sectional weights in compensating for attrition by compar- ing estimates at wave 2 for all wave 2 respondents and only those respon- dents who remained as of wave 6 [see Petroni and King, 1988; King et al., 19903~; analysis of data from reinterviews of SIPP respondents (e.g., Hill, 1989); simulation studies (e.g., simulating alternative schemes for o~rersampling subgroups of policy interest); and most recently, application of cognitive research techniques (e.g., one- on-one sessions in which the interviewer asks the respondent to think aloud in answering each question) to understand respondents' perceptions of the questionnaire (see below). As a result of the research and evaluation program, the Census Bureau has instituted some changes in procedures and questionnaire content for SIPP. For example, a second administration in each panel of the assets and liabilities module was dropped, as was the missing wave module,5 because of adverse research findings about the quality of the data. Research results also contributed to the recent decision to switch from maximum personal to maximum telephone interviewing for SIPP, as well as the strategy adopted for oversampling low-income households beginning in the 1995 panel. In other instances, research has not produced clear findings and hence is con- tinuing: for example, various steps to reduce the seam problem have thus far had little effect. In still other cases, resources have not been available to implement findings: for example, no changes have yet been made to impu- tation and weighting procedures to adjust for biases found in research. Whatever the outcome for SIPP operations, in almost all instances re 5The missing wave module was designed to fill in information for people who had missed the preceding wave but were interviewed at the wave prior to that one. The module, which was administered beginning in wave 4 of the 1984 panel and discontinued midway through the 1986 panel, asked an abbreviated set of questions on labor force status, program participation, income receipt, and asset ownership for the reference period covered by the preceding wave. Evaluation determined that the number of transitions reported was much smaller than predicted and that the additional information obtained did not appear to justify the respondent burden and cost of collection (Jabine, King, and Petroni, 1990:40).

OCR for page 202
206 THE SURVEY OF INCOME AND PROGMM PARTICIPATION search has generated valuable information for users on the possibilities and limitations in the data. Research results have been disseminated through the SIPP Working Paper series. Also, to bring together available informa- tion on data quality, the Census Bureau put a substantial effort into develop- ing the SIPP Quality Profile. (The ASA/SRM working group was a prime mover behind this project.) The first edition of the profile was published as a SIPP Working Paper (King, Petroni, and Singh, 1987~; the second edition (labine, King, and Petroni, 1990) represents a reorganization and significant expansion of the material. The SIPP Qualify Profile brings together what is known about method- ological problems with SIPP at each stage from data collection through data dissemination and also includes aggregate comparisons with other data sources. The profile is of great value for data users, for survey methodologists, and for the Census Bureau in determining the agenda for further research and evaluation. Indeed, the profile sets a standard for the field that other sur- veys would do well to emulate. REDESIGN OF SIPP SIPP is scheduled to undergo a major redesign in the mid-199Os. The Census Bureau expects to decide on the basic elements of the redesign by the end of 1992, but many details will be worked out later. It will be important to have a targeted research program to provide inflation to help resolve outstanding issues and to ensure the smoothest implementation possible of the new design. When it is in place, the redesign should achieve important improvements in many aspects of survey operations and in the quality and utility of the data; however, it may also have untoward effects. Again, research will be needed to identify the successes of the redesign and to suggest ways to handle any problems that anse. In this section we discuss the components of a program of methodologi- cal research that we believe the Census Bureau should put in place to in- form and evaluate the SIPP redesign. These components include: research to improve the format and wording of the questionnaire; research targeted to other aspects of the redesign (e.g., implementation of CAPI); research on issues of estimation and data use (e.g., weighting and imputation) in light of the redesign; research to evaluate the success of the redesign; and, finally, a quick-response capability to address unanticipated problems in and after implementation. Questionnaire Content In Chapter 3 we propose a number of content changes to SIPP for ex- ample, adding a few questions about the respondent's family of origin,

OCR for page 202
METHODOLOGICAL RESEARCH AND EVALUATION 207 obtaining additional detail about household relationships, and ascertaining information needed to determine eligibility for major assistance progrmns on a more frequent basis. (Correspondingly, we identify some topics that might be scaled back in detail.) We also recommend experimental work to develop measures of protection against economic risk (e.g., access to credit). In addition, we strongly urge a total overhaul of the questionnaire content related to measurement of assets in SIPP. We are sure that the community of users will have suggestions for the questionnaire as well. We understand that the Census Bureau hopes by the end of the year to decide on the content changes in the core questionnaire that will be imple- mented as part of the redesign It appears quite reasonable to adopt this schedule for determining topic areas and the general level of detail in the core, particularly given that the staff who are working to design the CAPI and database management systems for SIPP need this information. How- ever, we believe that the Census Bureau should not try to lock in the precise format and question wording before thorough testing of proposed changes. Such testing is critical in view of the inevitable tendency in any ongo- ing survey program to resist frequent questionnaire changes. Although we hope and expect that the conversion of SIPP to CAPI and database manage- ment system technology will make it easier to modify content and format as needed, it is still true that questionnaire changes will not and should not be made lightly. The occasion of the redesign offers the opportunity to add, delete, or modify a large number of questions. Such an opportunity will not likely occur again for many years. Hence, it is incumbent upon the Census Bureau to evaluate proposed content changes as thoroughly as possible be- fore determining the final format and question wording for the redesign. We realize that time is short, particularly given the need to implement the questionnaire in CAPI and a new database management system. We suggest that, in addition to standard pretests, the Census Bureau make use Of twn means of questionnaire testing and evaluation that we believe could ~ ^ ~, ~ = _ be implemented rapidly. First, we suggest that the SIPP staff who use the core data and who will redesign the questionnaire work closely with the researchers who are in charge of the Census Bureau's program to apply the results of cognitive research to developing improved methods for collecting higher quality data in SIPP (see below). That program is testing a very different set of inter- viewing procedures (e.g., more use of records by respondents, conducting the interview for a household on a group basis) and a very different ques- tionnaire format, in which many of the questions are free-form (e.g., the respondent is asked to name income sources in any order and supply amounts as received rather than as fixed monthly totals). As discussed below, we are highly supportive of this program but skeptical that it can be carried out on a sufficiently rapid time schedule or with sufficient evaluation to permit the

OCR for page 202
208 THE SURVEY OF INCOME AND PROGRAM PARTICIPATION new procedures and format to be incorporated for the redesign. Recogniz- ing that it may be difficult to obtain the full benefits from this program unless it is adopted as a package, we nonetheless believe that the format and wording of a more standard SIPP questionnaire could likely be improved by the results of the cognitive research program.6 We urge the SIPP design and analysis staff to make as full use as possible of the findings of this program in determining the final form of the questionnaire. Second, we suggest that the Census Bureau use small-scale record- check studies, particularly forward record-check studies, as a vehicle for questionnaire testing and evaluation. Forward record checks would involve drawing samples of a few hundred cases each from relevant administrative record sources (e.g., program records, employer records, or tax records) and administering the SIPP interviews to each case. There are several advan- tages of this approach for questionnaire research. The samples can target population groups of special relevance for SIPP, such as recipients of such programs as AFDC. The availability of administrative data for the samples provides the opportunity to evaluate the quality of the responses. For ex- ample, cognitive research has demonstrated that program recipients may fail to report their benefits or may describe them as something else because they do not recognize the name of the program as it is listed in the questionnaire. A forward record check would provide a ready means to evaluate the effec- tiveness of alternative ways of identifying programs for respondents. Another advantage of forward record checks for questionnaire research purposes is that, for many research objectives, it is not necessary to obtain a nationally representative sample. lIence, in the case of state-administered programs, for which there is wide variation in the quality and accessibility of administrative records, the samples could be selected from those states with the best systems.7 Finally, forward record checks can be implemented on a timely basis because there is no need for an after-the-fact match of the administrative and survey data. We note that the Census Bureau's cognitive research program is itself using a forward record-check approach, drawing small samples from records for four programs and one employer in Milwaukee County, Wisconsin, which will be the site of a major evaluation study of the new interviewing proce- dures and questionnaire format (Bureau of the Census, 1992b; see also below). We note further that the ISDP experienced considerable success 6For example, it might be possible to allow respondents to supply information about amounts in a somewhat freer format once recipiency has been ascertained in the standard fashion (e.g., specifying the date and amount of each payment, which could then be converted by the data processing system to monthly values). 7However, for programs for which eligibility rules, regulations, and administrative practices vary widely across states (e.g., AFDC), record checks that are limited to a few states may resolve reporting issues only for those states and not others.

OCR for page 202
METHODOLOGICAL RESEARCH AND EVALUATION 209 with forward record checks for evaluating the ability of the survey ques- tionnaire to elicit accurate reports of program participation and benefits (see Kasprzyk, 1983~. We urge the Census Bureau to give high priority to using forward record-check studies for refining the questionnaire that is imple- mented as part of the SIPP redesign. However, forward record-check studies cannot assess the effects of false reports of program participation (or other behaviors), since this would re- quire drawing samples of known nonparticipants of the program under study. Since false positive errors are rare in SIPP (see Marquis and Moore, 1989, 1990a), large samples would be required to obtain a sufficient number of false positive responses to study the causes and remedies for overreporting in SIPP. More research is needed on the design of these types of record- check studies. For example, sampling efficiency could be improved if the sample could target groups of nonparticipants who are more prone to misreport program participation, such as people who experience frequent transitions into and out of programs. Other Aspects of the Redesign Other aspects of the SIPP redesign for which it would be useful to conduct methodological research (including design changes that could be appropri- ate to implement somewhat later on) are the length of the recall period, oversampling based on screening, implementation of CAPI, and telephone interviewing. We note that forward record-check studies, in addition to supporting questionnaire research, could well be used to evaluate other as- pects of the redesign. For length of recall period, we decided not to recommend a change in the current 4-month recall period for SIPP because of the possible adverse effects on the quality of the monthly information, which is critical for so marry policy and research uses of SIPP. However, a move to 6-month recall could permit an increase in both the sample size and length of each panel and might reduce the effects of attrition for longer panels. The literature does not provide clear guidance on the pros and cons of 4-month versus 6- month recall for SIPP. Hence, we urge the Census Bureau to give priority to research on this issue.8 Research is also needed to investigate recall period effects for cognitively designed interviews (see below), since memory effects for these types of interviews may be quite different from those for the traditional SIPP interview. We do not consider it appropriate to change the recall period for the redesign. However, if the research results indicate that a 6-month recall Further analysis of the SIPP record-check study could perhaps contribute to understanding of recall effects on the quality of the monthly data.

OCR for page 202
210 THE SURVEY OF INCOME AND PROGRAM PARTICIPATION period would provide data of acceptable quality, then the Census Bureau should give serious consideration to changing the survey design to some- what longer and larger panels without waiting until the next major sched- uled redesign. We suggest that the efficiency of the proposed oversampling of the low- income population in SIPP, which the Census Bureau plans to implement by using information from the 1990 census, might be improved by conducting a screening interview close to the time of wave 1 for each panel. Research needs to be conducted on ways to implement the screening approach, to minimize the costs and possible difficulties of this approach for field opera- tions, and to develop better estimates of the reductions in sampling errors, compared with those in the Census Bureau's proposal. As the Census Bureau moves toward CAPI for SIPP, methodological research is needed to evaluate and understand CAPI's effects on the inter- view, the respondent, the interviewer-respondent interaction, and ultimately, the distribution of measurement error for SIPP items. In addition, more should be learned about the cost-error tradeoffs of using CAPI for SIPP. Also, research is needed to develop methods that take full advantage of the capabilities of CAPI (e.g., on-line editing, computer-directed probing, and interviewer help screens) that have the potential for reducing measurement errors. As the development of a CAPI system for SIPP progresses, research should proceed simultaneously on the quality differential between CAPI and paper-and-pencil methods for SIPP. As part of the plan to convert to CAPI, the Census Bureau will need to consider the appropriate role of personal versus telephone interviewing in SIPP. Research is needed on the implications for costs and data quality of continuing the current mode of maximum telephone interviewing in waves 3-5 and 7-8 versus reverting to heavier use of in-person interviews. A particular concern is the feasibility of using recently developed cognitively based interviewing procedures and questionnaire formats over the telephone. The Census Bureau's evaluation of telephone versus in-person inter- viewing to date (from a 1985-1986 telephone experiment) does not provide sufficient information for an informed decision. For example, sizable dif- ferences for many items were not statistically significant, indicating inad- equate power for the comparisons. The telephone experiment also did not obtain adequate information to assess the cost implications of different in- terview modes (because only the designated interview mode not the actual mode was recorded).9 More needs to be learned in order to evaluate the most cost-effective allocation of personal visits and use of the telephone in an environment of computer-assisted interviewing. Also, work should be awe presume the Census Bureau is obtaining cost information from the use of maximum telephone interviewing for the current SIPP panels.

OCR for page 202
METHODOLOGICAL RESEARCH AND EVALUATION 211 done to determine the possible savings in cost and improvements in quality that could be effected by using centralized computer-assisted telephoning for some types of SIPP interviews. Estimation and Data Use Much of the SIPP methodological research program has focused on under- standing and improving data quality at the source, for example, by means of changes in questionnaire wording and interviewing procedures. Although this focus is appropriate, research is also needed on how best to treat the quality problems that remain at the points of data analysis and use. In this section we discuss needed research on weighting and imputation that is particularly important given the changes that we propose in the design of SIPP namely, longer panels and less frequent introduction of new panels. We also discuss the problem that undercoverage in SIPP varies across popu- lation groups, which has implications for weighting adjustments as well as for improved field procedures. Population undercoverage affects household surveys generally, but it may be particularly troublesome for SIPP given the evidence that it is low-income people who are most likely to be missed. Weighting and Imputation To date, Census Bureau staff and outside analysts have examined weighting procedures to compensate for sample selection and attrition and imputation procedures to adjust for item nonresponse. Both cross-sectional and longi- tudinal procedures have been assessed. It is clear that the current cross- sectional weights do not adequately adjust for differential attrition by such characteristics as income level (Petroni and King, 1988; King et al., 1990~. It appears likely as well that the weights do not adequately compensate for differential undercoverage of population groups in the survey (see below). There is also evidence that the current cross-sectional imputations do not adequately reproduce known relationships between income, assets, and pro- gram participation (Doyle and Dalrymple, 1987; Allin and Doyle, 1990~. However, the research in these areas has not been carried to the point of identifying optimal revisions to the Census Bureau's weighting and imputa- tion programs. We urge that such research be conducted and that appropri- ate changes in the Bureau's procedures be implemented on the basis of the results. One avenue to pursue is research on the benefits of more extensive use of wave 1 income and program participation variables to adjust for attrition in subsequent waves. The need for further research on longitudinal weighting and imputation is even more critical. The current longitudinal weights reduce the available sample size for analysis because only people with data from all waves of an

OCR for page 202
212 THE SURVEY OF INCOME AND PROGRAM PAR TICIPATIOlI entire panel (or a calendar year) are given positive weights. The current longitudinal imputations are implemented only for selected variables and only after all waves of a panel are complete.~ Further investigation of improved weighting and imputation procedures, both cross-sectional and longitudinal, is especially important in the context of the proposed redesign. The less frequent introduction of new panels will require research on more effective cross-sectional weighting adjustments to compensate for attrition bias in estimates for the "off,' years (every other year), when a new panel is not in the field. Research will be needed as well on the extent of bias in the off-year estimates (and ways to compensate for the bias) that results from loss of coverage for people who enter the SIPP universe (e.g., by leaving institutions) and have no antecedents in the previous on-year population. The increase in panel length will also require research on more effective longitudinal weighting procedures to minimize the loss of sample cases. The development of imputation procedures to supply data for waves that are missing in their entirety could help with this problem. The use of imputation seems particularly promising for cases with only one or two missing waves, in which the missing waves are bounded by interview data for the preceding and succeeding waves. For another longitudinal estimation issue, we urge the Census Bureau to conduct research on appropriate weighting for analyses of spell duration (spells of low income or program participation) that use such standard sur- vival analysis procedures as the Kaplan-Meier and proportional hazards modeling approaches. Analysts using these methods often assume that the survival probability of people who remain in the sample is the same as those who do not (within subgroups or given a common set of covariates). Armed with this assumption, analysts proceed to make use of all sample cases up to the point of attntion. They typically ignore the survey weights because there is no appropriate set of weights available. An alternative is to restrict the analysis to cases with complete data. Then the analyst can readily adapt the Kaplan-Meier and proportional hazards procedures to in- corporate survey weights that adjust for nonresponse. However, this ap- proach fails to use all of the cases. Research is needed into alternative imputation and weighting strategies that make fuller use of the cases with 10These imputations replace previous cross-sectional imputations for item nonresponse; they are not used to supply data for waves that are missing in their entirety. 1 1Under the proposed design with 4-year panels introduced every other year, cross-sectional estimates with maximum sample size for "on" years can be based on the first year of a new panel and the third year of the previous panel; cross-sectional estimates with maximum sample size in "off" years can be based only on the second year of the most recent panel and the fourth year of the previous panel. 12Work is currently under way on this topic through a joint statistical agreement between the Census Bureau and the University of Michigan.

OCR for page 202
216 THE SURVEY OF INCOME AND PROGRAM PARTICIPATION Should attrition at early waves increase more than expected, the Census Bureau should have resources set aside to permit timely experimentation with ways to overcome the problem. Finally, as noted above, an important part of the research that is devoted to estimation issues will need to be the development of more effective weighting and imputation procedures to minimize the effects of differential attrition from the longer panels. It is unlikely that sufficient information to determine the optimum re- call period for SIPP will be obtained prior to the redesign; hence, the Cen- sus Bureau will need to continue research on this topic. Again, forward record-check samples can provide a vehicle for experimentation-on recall effects. The Census Bureau will also need to plan a research program to assess the effects of the CAPI system that is implemented for SIPP versus paper-and-pencil techniques on the quality and comparability of the data across SIPP panels. Finally, the Census Bureau should plan studies of th quality of the data collected in response to new or modified questions in SIPP. Comparisons with other data sources should be helpful for this pur- pose, as should analysis of the information available in the forward record- check studies. Quick-Response Capability As we have noted, it will not be possible to anticipate every data quality problem or user need that may arise in SIPP after the redesign. It is impor- tant for the Census Bureau to have some reserve capability to conduct research and evaluation on new problems and concerns as they arise and to determine the best ways to respond. For example, evaluation of question- naire changes introduced as past of the redesign may reveal unexpectedly serious problems that require timely experimentation with revised content, edits, or other procedures. Or shifting policy interests may require more detailed coverage of a topic in SIPP, and the Bureau will need the capacity for timely field testing of new or alternative questions. Or it may be neces- sary to take steps to deal with unexpectedly high rates of attrition. Hence, it is only prudent for the Census Bureau to have contingency plans to permit timely assessment and corrective action in the event that a serious problem occurs with the redesign. Recommendation We believe that methodological research and evaluation is needed to inform and assess the SIPP redesign. The magnitude of the changes that are pro- posed for the redesign together with the relative newness of SIPP and its undeniable scope and complexity argue for a strong, multifaceted program.

OCR for page 202
METHODOLOGICAL RESEARCH AND EVALUATION Recommendation 7-1: The Census Bureau should support meth- odological research and evaluation for SIPP leading up to and following the survey redesign. The research program should include the following components: research to improve the format and wording of the ques- tionnaire, making use of record-check studies and, to the extent possible, of findings from the current program of cognitively based questionnaire experimentation; research targeted to other aspects of the current redesign (and to possible design changes later on), including the length of the recall period, screening techniques to obtain larger sample sizes for subgroups of interest, and data collection modes (the best combination of computer-assisted personal and telephone interviewing and the possible role of centralized telephoning); research on issues of estimation and data use, taking into account the features of the redesign and including ways to im- prove cross-sectional and longitudinal weights, imputation pro- cedures, and population coverage; research to evaluate the success of major elements of the redesign (e.g., the attrition effects of longer panels); and a quick-response capability to address unanticipated prob- lems with the implementation of the redesign. CONTINUOUS ERROR MONITORING 217 With the focus on the upcoming redesign of SIPP, it is important that the Census Bureau not lose sight of the need for continuous monitoring of error levels in SIPP. Users need a constant flow of information about data quality as they seek to work with later panels and topical modules and to compare their results with work on earlier panels and other data sources. In this regard, we wholeheartedly support the plan of the Housing and Household Economic Statistics Division to compare income data from the March 1991 CPS and the 1990 SIPP panel. Many users will want to work with the 1990 SIPP panel because it offers the largest sample size of any SIPP panel yet fielded, and an in-depth comparison of SIPP and CPS data for 1990 will greatly benefit users. Continuous monitoring, which is a vital source of information for the SIPP staff at the Census Bureau as well as outside users, covers a range of topics and methods. For example, internal analysis of SIPP data generates rates of attrition and of person and item nonresponse. Internal analysis also generates estimates of the extent to which transitions are reported between pairs of months on and off the seam. Analysis of reinterviews of subsamples

OCR for page 202
218 THE SURVEY OF INCOME AND PROGRAM PARTICIPATION of SIPP respondents may also provide useful information on questions that are not well understood or reported. Comparison of SIPP aggregates with aggregates from other sources is another form of monitoring, which may reveal differences that raise warn- ing flags for users and SIPP staff. To be meaningful, such comparisons require detailed understanding both of SIPP and the alternative source. For example, aggregates from the National Income and Product Accounts (NIPA) often serve as a basis of comparison for income totals from surveys, but extensive manipulation is usually required to achieve a valid comparison. Substantial revisions to the NIPA are under way or being planned-, and it is important that Census Bureau analysts keep abreast of these changes to the NIPA and their implications for evaluation of income data from SIPP and the March CPS. Record-check studies can also make a significant contribution to as- sessing error levels in SIPP. The small-scale forward record checks that we recommend as part of the research leading up to and following the redesign should be helpful in this regard. Also, it would be very useful to periodi- cally conduct reverse or full record-check studies based on matching SIPP panels with administrative records (e.g., tax returns or program case records). Reverse record checks that are based on probability samples of the entire SIPP universe are more expensive and time consuming to carry out, but they are the only way to obtain a full assessment of reporting errors, includ- ing false positives and false negatives. We note that the Census Bureau now has under way a match of the 1990 SIPP panel with an extract of the Individual Master File of tax returns from the Internal Revenue Service (IRS). Although the extract file contains a limited set of items, it can help evaluate certain types of income reports. We also encourage the Census Bureau to further exploit an existing resource: the reverse record-check study that involved matching records for a number of federal and state assistance programs in four states with SIPP cases in the first two waves of the 1984 panel. The analyses of the matched file conducted to date have been useful (see, e.g., Marquis and Moore, 1989, 1990a), but they did not go far enough. Very little marginal cost would be required to carry out additional analyses, such as investigating biases in duration of spells and reports of benefit amounts, and we urge that these be done at an early date. Such studies would benefit users and could also contribute useful information for consideration of possible improve- ments to the questionnaire.~9 |9The staff who worked on the SIPP record-check study are now engaged with other projects. If Census Bureau staff are not available for further analysis of the data, the Bureau should consider inviting an outside researcher to carry out the work through a fellowship or other on- site arrangement that permits access to the data.

OCR for page 202
METHODOLOGICAL RESEARCH AND EVALUATION 219 Another aspect of error monitoring relates to sampling error or van- ance. The Census Bureau needs to document for users the estimated sam- pling errors for estimates of subgroups from SIPP panels of different sizes and also consider innovative ways to reduce sampling error.20 The redesign of SIPP will entail important changes in the sampling scheme namely, an increase in overall panel size and the introduction of oversampling of lower income groups. The Census Bureau will need to thoroughly investigate the variance effects of the new design on estimates developed for subgroups of the oversampled, undersampled, and total population. Indeed, we urge the Census Bureau to plan an in-depth technical report for users on the variance implications of the new sample design. Looking to the future, we expect that the introduction of new data collection and processing technology (namely, CAPI and a new database management system) should improve the efficiency of the monitoring func- tion by making it possible for analysts to have hands-on access to the data at an earlier stage in the processing. Such access should make it possible for many of the results from monitoring to have an immediate, beneficial impact on survey operations (e.g., leading to changes in edit or imputation routines). We urge the Census Bureau to make improved access to SIPP data on the part of its analysts an important goal of its investment in new technology. Finally,- a critical part of the monitoring function is the provision of information to all users. For this purpose, it is important that the Census Bureau support such means of communication as the SIPP Working Papers and, most especially, regular updates to the SIPP Quality Profile. As in the past, these documents should include error analyses that originate from both Census Bureau staff and outside users. Recommendation 7-2: The Census Bureau should undertake continuous monitoring of error levels in present and future SIPP panels and regularly provide information on errors to users, in periodic updates of the SIPP Quality Profile and other publica- tions. 200ne avenue for reducing variance that the Census Bureau is considering is to adjust the SIPP sample weights, using some type of raking or iterative proportional fitting procedure, so that weighted estimates match control totals from administrative records. Muggins and Pay (1988) conducted preliminary research on the feasibility of this approach to reduce the vari- ance of income estimates from SIPP. In their study, the sample weights on a 12-month research file from the 1984 panel were adjusted to control totals derived from a 1 percent sample of individual income tax returns from the IRS. (Only the weights of SIPP cases that were successfully matched to a full IRS file were adjusted.) The results showed reduction of variance for such estimates as mean and median income.

OCR for page 202
220 THE SURVEY OF INCOME AND PROGRAM PARTICIPATION COGNITIVE RESEARCH Innovative data collection and processing technologies such as CAPI and a database management system can benefit SIPP (and other surveys) in many ways, including making possible substantial improvements in data quality. However, technology cannot substitute for appropriate understanding and motivation on the part of the survey respondents. To the extent that respon- dents do not understand the questions to mean what the survey designers intended or are not motivated to search their memories or consult records to the extent necessary to provide an accurate response, then the quality of the data will necessarily suffer. A recent development in survey research has been the introduction of approaches from cognitive psychology to study in greater depth the ways in which respondents react to and interpret specific question wording. The results have often shown startling differences in perceptions between re- spondents and survey personnel (see Jabine et al., 1984~. Federal statistical agencies, including the National Center for Health Statistics, the Bureau of Labor Statistics, and the Census Bureau, are now making considerable use of cognitive techniques for questionnaire research and experimentation, such as one-on-one sessions in which a researcher probes the respondent after each question to ask what he or she had in mind in answering. Results The Census Bureau recently began applying cognitive techniques to the task of improving SIPP measures of income and program participation (see Mar- quis, Moore, and Bogen, 1991~. This work had its origins in the SIPP record-check study when analysis of the data did not produce clear findings for such phenomena as the seam problem or other response errors (although the analysis was limited). Specifically, there was little support for the forgetting theory of memory: that is, underreports of participation for most programs were no more likely to occur for 4 months prior to the interview than only 1 month. Exploratory observational research conducted in fall 1989 suggested that respondents often use simple heuristic devices, combined with a few recalled facts, to construct 4-month income streams instead of making the effort to develop a detailed recall or to check their records (e.g., they may derive monthly values from annual amounts). This research also suggested that misunderstandings about the intent of particular questions often occur because respondents do not understand the goals of particular sections of the questionnaire (Marquis, 1 990~.2i 21For this research, Census Bureau headquarters staff, who received special Gaining in think-aloud and direct questioning techniques, accompanied SIPP interviewers, took notes on

OCR for page 202
METHODOLOGICAL RESEARCH AND EVALUATION 221 Formal cognitive laboratory research conducted by Westat, Inc. (under contract to the Census Bureau) has begun to produce findings of relevance to understanding response errors in SIPP (see Cantor et al., 1991~. The research involved recruiting 125 respondents, about one-half of whom were participants in some type of government program, to receive wave 1 and wave 2 SIPP interviews. At wave 1, half the respondents were administered the regular interview, and the other half were administered that interview and additional procedures. These procedures included asking the respon- dents to think-aloud during the interview, repeat questions back in their own words, and answer additional detailed questions designed to obtain a second measure of recipiency and amounts. At wave 2, the additional procedures were used for both groups. Cantor et al. (1991:4-15) report some very provocative findings from the wave 1 interview on respondents' motivation, information storage, com- prehension, and information retrieval and formulation. On motivation, the questionnaire, as currently structured, does not encourage respondents to be active participants in the interview or to do their best to provide accurate information. For example, the questionnaire does not allow acceptance of information that respondents volunteer out of sequence, and it bores respon- dents with long lists of income sources of which only a few are usually relevant. Errors in reporting program information often result from the fact that participants do not know the name of the program or know it by another name than that used in the questionnaire. For example, elderly respondents in the Westat study often could not differentiate between social security, Medicare, and "medical assistance," which is the local Medicaid name in Maryland and the District of Columbia. In reporting earnings, many re- spondents think of net rather than gross pay, and they often have trouble aggregating weekly or biweekly paychecks to monthly units. Also, respon- dents often do not commit to memory the particular types of assets they hold, for example, whether they have a money market deposit account or a money market fund. Information on asset income amounts is even less frequently stored in memory. Comprehension is clearly a problem: respondents find the labor force questions complex, with many qualifying phrases that make them hard to answer. Respondents often do not understand the distinction between earned or accrued and received income. (SIPP wants the latter amounts, including, for example, a current paycheck for work performed some time back, but the interviews, interrupted at various points to ask respondents how they interpreted and an- swered questions, and prepared written summaries of their impressions and experiences. The interview sessions were also taped.

OCR for page 202
222 THE SURVEY OF INCOME AND PROGRAM PARTICIPATION excluding, for example, interest on a certificate of deposit that has not yet matured.) A common mistake is for respondents to think about the reference pe- riod up to the date of the interview, which leads to a number of problems with respect to anchoring time points and calculating total income. To recall earnings, respondents tend to use a simple heuristic strategy, such as retrieving paycheck amounts and calculating the number of paychecks in a month, or multiplying the number of hours worked by an hourly rate, or dividing annual salary by 12. Once they have calculated the amount for the most recent month, respondents tend to apply this figure to the other months. Many respondents in the Westat study made errors in reporting earnings- based on comparing responses to the SIPP questions with an additional detailed recall most commonly because of misdating or miscounting pay- checks. Most respondents find it fairly easy to remember amounts for program participation, but very difficult to retrieve amounts for assets. They often guess at asset income-for example, calculating interest on the basis of the average balance in a savings account. Also, respondents find it too difficult and beyond their patience to develop accurate asset income amounts in response to the SIPP questions that require them to distinguish between separate and joint accounts, to aggregate across months, and to aggregate across asset types. Preliminary findings from the wave 2 interviews confirm the findings from wave 1 about the kinds of errors typically made by respondents.22 The results show evidence of the seam problem, particularly for amounts of asset income and wages. In many instances in which respondents report changes in amounts at the seam between the two interviews, it is because they make a fresh calculation at the wave 2 interview for the month prior to that interview and generalize the result to the other 3 months of the refer- ence period. Alternative Questionnaires and Interviewing Procedures It must be kept in mind that the findings from both the exploratory observa- tional research and the more formal cognitive procedures applied by Westat are based on limited samples. Nonetheless, there are strong indications that the current SIPP questionnaire may contribute to inaccurate responses for many income and program-related items. 220f the 125 respondents at wave 1, 76 completed a wave 2 interview (personal communica- tion, David Cantor, Westat, Inc.). The high nonresponse rate was due primarily to respondents' not returning phone calls or to their moving to a new address. Also, some of the wave 1 respondents were temporary residents of a homeless shelter and were no longer there at wave 2.

OCR for page 202
hIETHODOLOGICAL RESEARCH AND EVALUATION 223 Staff of the Census Bureau's Center for Survey Methods Research are currently experimenting with an alternative set of questionnaires and inter- viewing procedures (Marquis, Moore, and Bogen, 1991; Bureau of the Cen- sus, 1992b). In brief, the alternative being tested relies much less on re- spondent recall and much more heavily on the use of records. At the first interview, the goal of the core questionnaire, namely to obtain complete and accurate information on all sources of family income during the reference period, is explained and respondents are asked to bring out payment records. Respondents who receive but do not ordinarily save records are given a folder in which to keep them for future interviews. Using records-whenever possible, respondents are asked to help the interviewer fill out a worksheet for each income source that provides the amount and date of each payment. They can provide payment information for their income sources in any order (e.g., bank account interest before wages), and they do not have to compute monthly amounts. In other words, the respondents have the initia- tive in this portion of the interview, although the interviewers are trained to press respondents to think hard and not adopt a simple heuristic (e.g., re- porting all paychecks from their job as the same). To help ensure that no sources of income are overlooked, the inter- viewer next refers to flashcards that show about 50 sources of income and asks a short set of questions that are designed to jog respondents' memo- ries. Income sources are grouped in ways that seem likely to make sense to respondents: for example, asking about money from the military, including veterans' payments, military retirement, National Guard pay, and GI bill benefits; or asking about "surprises," such as an inheritance, lottery win nings, profits from gambling, insurance settlements, and work-related bo- nuses or awards. At subsequent interviews, to help reduce response errors, dependent interviewing techniques (reminding respondents of prior wave responses) are used. However, they are introduced late in the interview, in order to assist but not unduly influence respondents' recall. Also, the reference periods for pairs of interviews partly overlap, and differences in reported income sources and amounts are reconciled.23 Finally, all of the respon- dents in a household are urged to answer the questions in a group setting, and (at least during the testing period) they are asked for permission to record the interview. Preliminary results from initial small-scale field tests of these new methods are encouraging on some dimensions, although less so on others (Bureau of the Census, 1992b). On the positive side, much higher proportions of re- spondents used records than in regular SIPP interviews: for example, 65-80 23Specifically, the reference period for the first of two interviews extends up to the time of the interview, instead of, as now, stopping at the end of the month prior to the interview.

OCR for page 202
224 THE SURVEY OF INCOME AND PROGRAM PARTICIPATION percent of income sources were supported by at least one record in the tests, compared with 20 percent in the regular SIPP interviews. Also, there was essentially no seam bias in reports of program participation: the ratio of transitions on or off all programs reported between waves 1 and 2 to transi- tions reported within a wave was about 1, compared with ratios of 2 to 6 for various programs in the regular SIPP interviews. Finally, most respondents were willing to have the interviews recorded, and most adult household members participated in a group interview. On the negative side, the wave 1 interviews took longer- and hence were more costly to complete, as respondents sought out their records. Also, the response rates were unacceptably low (64% after two waves), although the reasons for low response do not appear related to the new procedures, as most refusals occurred before the procedures were intro- duced. Rather, the causes appear to include the traditional problems of infrequent and inefficient call-backs to follow up nonresponse (e.g., too many daytime and too few evening calls to households). However, there remains the concern that the new interviewing procedures, by placing new demands on the interviewers, will require field personnel who are better trained and more highly skilled than is the case with the current procedures. Thus, there is the potential for interviewer variance and bias as well as increased costs of recruiting, training, and retaining interviewers who are able to consistently and accurately apply the new procedures. The SIPP cognitive research staff are currently planning a larger evalu- ation of the new procedures and questionnaire format in Milwaukee, Wis- consin, to include two waves administered in fall 1992-winter 1993. A sample of 700 households will be obtained from administrative records for four programs and the records of a local employer, permitting evaluation of the quality of many of the survey items through a forward record-check approach. Half the sample will be interviewed using the regular SIPP for- mat and half using the new format. The staff have also proposed adding a third wave, conducted by telephone, and a study to assess the ability to program the new questionnaire format into CAPI. Looking to the Future We are impressed with the effort that the Census Bureau has made to use cognitive research methods to understand and seek to improve the quality of SIPP responses. The promise for a very different way of relating to respon- dents that obtains high-quality responses seems strong. We were not in a position to comment on specific details of the current program of field testing and experimentation with alternative questionnaires and interview- ing procedures because the program was in the very early stages at the time of our deliberations. We urge the Census Bureau to seek continued review

OCR for page 202
METHODOLOGICAL RESEARCH AND EVALUATION 225 and guidance on this work from experts in the field (e.g., members of the ASAISRM Working Group on Technical Aspects of SIPP). Our major concern is how the research on alternative questionnaire design and interviewing fits in with the other planned improvements in SIPP data collection and processing technology and with the overall goal of implementing major changes in the 1995 or 1996 panel. Should the tech- nique of free recall of income sources, using worksheets to record payment streams, prove effective in the field, its use will have major implications for SIPP data processing. It seems possible that the equivalent of worksheets could be built into a CAPI system and that the necessary computations to produce monthly incomes from the individual payment records could be made within a database management system. However, an extensive amount of reprogramming would be required, given that work is already in progress to develop CAPI and database management systems for SIPP that assume that a questionnaire close to the current fixed-format document will be in use.24 It seems unlikely to us that such reprogramming could be accom- plished and fully tested in time for the redesign. In addition to the question of integration with the CAPI and database management system development, we are concerned about the time that will be required for rigorous evaluation of the new procedures, which differ greatly from current survey practice. We believe that the Census Bureau should have results from more than one full-scale test (as is planned for Milwaukee) in order to develop sufficiently reliable assessments of the cost, feasibility, and data quality implications before making these kinds of changes. Overall, it seems very ambitious to attempt to carry out the necessary testing and evaluation of the new procedures and achieve a seamless inte- gration with the CAPI and database management system development work prior to the SIPP redesign. But if the cognitive questionnaire research is treated entirely as an experimental program, the likely result is that it will be starved for resources and that any positive findings will have little im- pact on SIPP until at least the next major redesign in the year 2005. There is no easy way out of this dilemma. We urge the Census Bureau to consider an approach whereby a team of systems and research staff work on a prototype of an integrated system that includes the CAPI and database management system programs that will be required for a new questionnaire. A firm schedule should be developed for design and testing of the prototype so that positive outcomes can lead to changes in SIPP without waiting until the next major redesign. (Also, as we urge above, the findings from the 24The SIPP CAPI staff' in discussions with the study panel, indicated that they are making changes to the current questionnaire that reflect some of the cognitive research findings (e.g., reordering and clarifying questions and simplifying skip patterns). However, they are not currently planning for any type of free format.

OCR for page 202
226 THE SURVEY OF INCOME AND PROGRAM PARTICIPATION cognitive research program should be used to the extent possible to improve the standard questionnaire.) Furthermore, we encourage the Census Bureau to devote adequate resources to the field test program so that results can be obtained on a timely basis from reasonably large samples of respondents in more than one site. Finally, as planned for the upcoming evaluation study in Milwaukee, all tests should build in the means to evaluate fully the effects of the new procedures on costs and data quality. Recommendation 7-3: We strongly support the Census Bureau's program of cognitively based research and experimentation with the SIPP questionnaire, which could contribute to questionnaire improvements for the current redesign and perhaps, in the fu- ture, to a major revision of the questionnaire and interviewing procedures. The Bureau should subject the cognitive work to rigorous evaluation, including record- check studies to evaluate data quality.