Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Summary The Survey of Income and Program Participation (SIPP) is a major continu- ing household survey, conducted by the Bureau of the Census, on people's economic well-being and their receipt of assistance from a wide range of government programs. Beginning in 1983, new samples of households have been selected for the survey every year. The adults in these households become panel members who are interviewed at 4-month intervals over a 32- month period. The questionnaire for each interview wave includes a core questionnaire about income, employment, and program participation and one or more supplemental modules on related topics. Funding cutbacks hindered the development of this highly complex sur- vey in its early years, but SIPP is now clearly established as an important source of information for federal policy making and social science research. The survey has a growing community of users in federal agencies, academic institutions, and other organizations. Analysts have used the data for new knowledge about such topics as part-year poverty and program participa- tion, multiple program participation, the effect of asset holdings on program eligibility and poverty, patterns of health insurance coverage, and the short- term behavioral dynamics of individuals and families. Despite its suc- cesses, however, SIPP has experienced problems that have kept it from being as useful as it could have been in the past and that, if not adequately addressed, could affect its usefulness in the future. After nearly 9 years of operation, the Census Bureau has under way a comprehensive reassessment of SIPP A new sample design, using informa
THE SURVEY OF INCOME AND PROGRAM PARTICIPATION lion from the 1990 census, will be implemented for SIPP beginning with the 1995 panel. At that time, the Census Bureau will make other changes to enhance the utility and cost-effectiveness of the program. As part of the evaluation and redesign effort, the Bureau asked the Committee on National Statistics (CNSTAT) to convene a study panel to conduct an independent review of SIPP. The panel drew on the work of an interim assessment of SIPP, performed by CNSTAT in 1989, which focused on federal agency uses of the data; consulted widely with users both inside and outside federal agencies; and conducted its own assessments of SIPP. Our report covers the following aspects of SIPP: the survey's goals and their implications for content and the relationship of SIPP to other surveys and administrative record data sources; survey and sample design; data col- leciion and processing; publications and other data products; analytical methods for using the complex longitudinal data from SIPP; methodological research and evaluation needed to plan and evaluate the SIPP redesign; and the management and oversight of the SIPP program. The first part of the Sum- mary presents our overall conclusions and recommendations on those as- pects; the second part presents all the panel's detailed recommendations. GOALS Over the course of SIPP's history, many people involved with the survey have wanted to expand it in one or another way to provide detailed informa- tion for their fields of concern. To satisfy these varied interests, SIPP would need to be an all-encompassing survey in the area of social welfare policy. We believe that SIPP cannot and should not be viewed as such. Rather, it is essential for the cost-effective operation of the program that it focus on a core set of major goals. The two primary goals for SIPP should be, as its name implies, to provide detailed information on the distribution of income and other eco- nomic resources and on eligibility for and participation in government as- sistance programs. Within these two goals, the survey should pay most attention to improving information for people who are economically at risk: poor people and near-poor and middle-income people who, if they experi- enced an event such as loss of a spouse or parent or job, would be at risk of economic deprivation and in need of assistance. As an added but secondary goal, SIPP should continue and strengthen its capability to respond to cur- rent policy needs for data in topical areas that are related to its core sub- jects, such as support for children and use of health care. We have identified several ways in which the data from SIPP should be enhanced to better serve the two primary goals. In the area of income- related measures, the priorities for improving information are: enhancing the quality of those measures that are relevant to program eligibility and
SUMMARY 3 participation; developing measures of taxes and after-tax income; and de- veloping measures that take account of in-kind benefits and that reflect changing family characteristics. In the area of program-related measures, the priorities for improving information are obtaining more complete and frequent data with which to determine program eligibility and developing adequate measures of periods or spells of both eligibility and participation. Finally, it is important that SIPP keep up to date with respect to new and changing sources of income and types of programs. How SIPP best achieves its goals raises the issue of its relationship to other surveys and administrative records. SIPP was developed-to provide added information and remedy deficiencies in the March income supple- ment to the Current Population Survey (CPS), which for decades has been the primary source of the nation's income and poverty statistics. SIPP's design enables it to collect more detailed information than is possible in the March CPS (e.g., intrayear and cross-year in addition to annual measures). Also, SIPP has achieved improvements in data quality (e.g., less item nonresponse) that would be difficult to match in the March CPS. However, to date, such problems as small sample size and lack of timeliness have limited SIPP's ability to provide regular income statistics. Changes that are implemented as part of the redesign should alleviate these problems. We urge the Census Bureau to set a target date by which time SIPP will be able to serve as the primary source of annual and other measures of income and poverty. (Some information on income should of course continue to be collected in the CPS for use in analyses of the labor force data that are the prime focus of that survey.) The use of administrative records (e.g., program case records and tax returns) can be helpful to SIPP in many ways. These records can provide additional information on sample persons, furnish the means to obtain addi- tional samples for groups of policy interest, and provide the basis for evalu- ating and improving the quality of the survey responses. However, the use of administrative records poses technical and operational problems that will need to be addressed. Also, some uses raise concerns about the confidenti- ality of the information, which must be adequately protected. SURVEY DESIGN We evaluated several alternative designs for SIPP-varying in panel length, frequency of introduction of new panels, length of recall period, and total sample size all of which were constrained to have the same number of annual interviews as provided for by the current SIPP budget. Each design has its own strengths and weaknesses relative to the current design and the other alternatives. We conclude that the current design is not optimal to the needs of SIPP
4 THE SURVEY OF INCOME AND PROGRAM PARTICIPATION users for timely, high-quality, and relevant data for cross-sectional and lon- gitudinal applications. We recommend that the length of each SIPP panel be increased from 32 to 48 months, a change that will make SIPP more suitable for analysis of spells of poverty and program participation and the dynamics of poverty and program entrances and exits. We also recommend that panels be introduced every 2 years rather than annually, a change that should reduce the operational complexity of the survey and facilitate timely data processing without compromising data quality. We recommend retain- ing the 4-month recall length, which means that each panel under the new design will have 12 interviews. However, we urge the Census Bureau to conduct research on 6-month versus 4-month recall periods, since an in- crease in recall length-if there were no adverse effects on the quality of the intrayear information- would permit longer and larger panels. Under the design that we propose, the total sample size of each panel would in- crease from 20,000 to 27,000 households, and it could increase further if savings are achieved through the introduction of new data collection and processing technology. DATA COLLECTION AND PROCESSING The Census Bureau is planning to convert SIPP data collection from paper- and-pencil methods to computer-assisted personal interviewing (CAPI). The Bureau is also planning to install a modern relational or other database management system for processing SIPP. We support both of these initia- tives, which should lead to improvements in data quality and timeliness of data products. We encourage the Census Bureau to identify CAPI and database management systems that incorporate all of the features necessary to handle the complex SIPP data. For example, the systems chosen should permit data from a previous interview to be used in the current interview and in the imputation of missing items. The data processing performed by the database management system should be fully and carefully integrated with the SIPP CAPI system. Because of the necessity for thorough testing of CAPI and database management technology in conjunction with other proposed changes in SIPP, we urge the Census Bureau to consider fielding a somewhat smaller dress rehearsal panel in 1995 as a means of working out operational problems. Under this scheme, full implementation of the new design would occur in 1996. DATA PRODUCTS AND THEIR USE To earn a high return on the investment in a survey as rich and complex as SIPP, it is critical that the responsible agency have an active dissemination
SUMMARY s program that includes timely and accessible published reports, computer- readable data products, and associated explanatory materials. In the past, there have been weaknesses in the data dissemination effort for SIPP, in- cluding the absence of a regular publication series for the core content on income and program participation. We support the Census Bureau's plans to publish regular, comprehen- sive descriptive reports from the SIPP core data. These reports should include longitudinal statistics on the dynamics of income, poverty, and pro- gram status and cross-sectional measures and should be issued on a frequent schedule. In addition, we encourage the Bureau to establish-a research report series to include in-depth analytical studies of topics related to in- come and programs. The Bureau should also continue publications from the topical modules and should produce reports from SIPP on the dynamics of major demographic and employment transitions. Development of statistics from the SIPP core data gives rise to difficult conceptual and measurement issues, particularly in the case of longitudinal measures. These issues include developing statistics that appropriately re- late to policy needs, specifying analysis units (e.g., household or person), characterizing change in contextual variables (e.g., marital status changes), constructing equivalence scales (i.e., income measures that are adjusted for household characteristics), measuring the duration of spells of poverty and program participation, and treating missing interviews. Because the issues involved in developing appropriate statistics from SIPP are so complex, we urge the Census Bureau to ensure that the analysis staff are able not only to prepare publications, but also to undertake an ongoing research program and keep up to date with regard to relevant analytical techniques and policy concerns. In addition to published reports, the Census Bureau should develop improved microdata products from SIPP to support policy analysis and so- cial science research. Priorities for improvement include moving toward a goal of releasing core data files within 6 months after the end of data collection and producing calendar-year files that combine two panels. Im- provements in documentation and related user information services are also necessary and important for SIPP. METHODOLOGICAL RESEARCH AND EVALUATION One strength of SIPP, going back to the days of its predecessor, the Income Survey Development Program, has been the extent of research on data qual- ity and ways to improve both the quality and the efficiency of the survey. SIPP should continue that focus on methodological research and evaluation. In the immediate future, the Census Bureau will need the results of method- ological studies to develop the details of the proposed redesign. Research
6 THE SURVEY OF INCOME AND PROGRAM PARTICIPATION should be conducted on proposed questionnaire changes, using small-scale administrative record-check studies and taking full advantage of the find- ings of current experiments with cognitively based interviewing procedures and questionnaire formats. Research should also be conducted on other aspects of the redesign, including alternative methods of oversampling the low-income population and the best combination of personal and telephone interviews in an environment of computer-assisted interviewing. Research should be carried out as well on issues of estimation and data use in light of the redesign, including ways to improve weighting and imputation. In the longer term, the Census Bureau will need information on the impact of the redesign to identify successes and to respond to problems. Finally, it will be important for the Bureau to monitor error levels in present and future SIPP panels on a continuing basis and to provide regularly updated infor- mation to users. MANAGEMENT AND OVERSIGHT An effective management structure is a key component to a survey's suc- cess. In the case of SIPP, the Census Bureau has leadership, analysis, and dissemination responsibilities that are not typical of the household surveys that it conducts for other agencies. These responsibilities, together with SIPP's complexity, argue for a management structure for SIPP that is stron- ger and more focused than the Bureau's structure for its other household surveys. We recommend that the Bureau establish a high-level position of project director with full responsibility for its income statistics program, including both SIPP and the March CPS income supplement. The project director needs to combine relevant substantive expertise with strong survey management skills and have a sufficiently large staff to guide the program, prepare reports and analyses, and address analytical concepts and methods. Finally, the Census Bureau has made commendable efforts to seek out- side advice about SIPP, which is essential to keeping the survey oriented to user needs and up to date with improvements in survey design and analysis. We urge the Bureau to strengthen its advisory mechanisms, particularly in seeking input from users in the academic community, and in obtaining ex- pert advice on analytical methods that is comparable to the advice it cur- rently obtains on survey methodology and data products. RECOMMENDATIONS The text of all of the panel's recommendations are presented here, keyed to the chapters in which they appear in the report.
SUMMARY Goals for SIPP Recommendation 2-1: The two primary goals of SIPP should be to provide improved information on the distribution of income and other economic resources for people and families and on eligibility for and participation in government assistance programs. Within these two goals, most attention should be paid to improving the information for people who are economi- cally at risk. A third important but subordinate goal is for SIPP to have a capability to respond to current policy needs for data in topical areas that are related to the core subjects of SIPP. Achieving SIPP's Goals Recommendation 3-1: Priorities for improved income and related measures from SIPP should include: · enhancing the quality of income and related measures that are rel- evant to program eligibility and participation; · developing measures of taxes and after-tax income; · valuing (or otherwise taking account of) in-kind benefits; and · constructing income and related measures that take account of fam- ily characteristics and changes in families over time. SIPP should also keep up to date with respect to new sources of income and other economic resources. Recommendation 3-2: The measurement of asset-related information in SIPP should be reassessed in light of SIPP's focus on programs and the economically at-risk population. The collection of asset items should be redesigned and simplified, if possible, to reduce respondent burden and improve the quality of the data needed to serve SIPP's primary goals. Recommendation 3-3: SIPP should develop, on an experimental basis, selected measures of economic security against risk, such as access to credit. Recommendation 3-4: Priorities for improved measures of program par- ticipation and eligibility from SIPP should include improving the range and frequency of information needed to determine eligibility for major assis- tance programs and providing adequate measures of spells of both eligibil- ity and participation. SIPP should also keep up to date with respect to newly important programs and program changes. Recommendation 3-5: The topical module component of SIPP should con- tinue and be strengthened by: · obtaining input from both government agencies and the social sci- ence research community about topics related to SIPP's core goals to con- sider for modules;
8 THE SURVEY OF INCOME AND PROGRAM PARTICIPATION · streamlining the content development process so that timely infor- mation can be collected on emerging policy and research issues; and . using some topical modules as a means for the Census Bureau's analysis staff to conduct research on expanded and alternative measures of income and programs. Recommendation 3-6: SIPP should become, over time, the primary source of the nation's income statistics in place of the March CPS income supple- ment. SIPP should receive priority for major investments to develop im- proved income measures. As there will necessarily be a transition period during which SIPP and CPS income statistics are both published, every effort should be made to increase user understanding of differences and similarities and to effect incremental improvements as appropriate in both surveys. Survey Design Recommendation 4-~: SIPP should be redesigned as an ongoing panel survey in which each panel lasts for 4 years and has 12 4-month interviews, with a new panel introduced every 2 years. The sample size for each panel should be increased over that for the current design. Recommendation 4-2: The Census Bureau should conduct research on the data quality effects of 6-month versus 4-month reference periods in SIPP, so that information is available to consider other possible design changes at a later date, including the possibility of further extending the length of SIPP panels beyond 4 years. Recommendation 4-3: The use of a monthly rotation group structure should be retained for SIPP. The Census Bureau should consider cost-effective means to obtain the core data for the last calendar year of each panel that will otherwise be missing for some months for some groups. The Census Bureau should also investigate ways to minimize the loss of mover house- holds that may result in part from the closeout of follow-up at the end of each month. Recommendation 4-4: The Census Bureau should investigate alternative methods of oversampling the low-income population in SIPP, including the use of screening interviews as a possible complement to or substitute for an approach based on using information from the 1990 census. Recommendation 4-5: The Census Bureau should take steps to ensure that it will be possible to extend the length of SIPP panels for selected sub- groups of interest or to follow them up at a later date, should such options be desired to obtain additional sample size and longitudinal information
SUMMARY 9 Recommendation 4-6: SIPP panels should treat all children who reside in interviewed households at the first wave and also children born during the course of a panel to original sample mothers as original sample members, who are followed if they move into households without an original sample adult. SIPP panels should also continue to follow and collect data for both original sample adults and children if they move into institutions. Data Collection and Processing Recommendation 5-1: We strongly support the Census Bureau's goal to convert SIPP to computer-assisted personal interviewing (CAPI). Since the Bureau's current CAPI software system (QUISC) does not appear to meet the data collection requirements for SIPP, the Census Bureau should give high priority to investigating other available CAPI systems and determine the most appropriate system for SIPP. Recommendation 5-2: We strongly support the Census Bureau's plans to adopt a new database management system for SIPP. The Census Bureau should use the capabilities of a DBMS to the fullest in seeking to make improvements in all aspects of processing, analyzing, and documenting the data from SIPP. The processing performed by the database management system should be fully integrated with the SIPP CAPI system. Recommendation 5-3: In view of the major advances that continue to occur in computing hardware and software, the Census Bureau should de- vote significant resources to continued education and training of its data processing staff. In particular, the SIPP processing staff should take advan- tage of the experience of other data processing facilities outside the Census Bureau that deal with longitudinal surveys. Recommendation 5-4: The Census Bureau should make every effort to ensure smooth implementation of CAPI and an improved database manage- ment system for SIPP under the new design of 4-year panels introduced every 2 years. One option that the Census Bureau should consider is to field a large-scale dress rehearsal panel in 1995 as a way to work out any operational problems. Under this scheme, full implementation of the SIPP survey redesign would occur in 1996. Data Products and Their Use Recommendation 6-1: The Census Bureau should move forward with its plans for regular, comprehensive series of descriptive reports on income, programs, and related topics from the core data in SIPP. Longitudinal statistics (e.g., on the dynamics and correlates of transitions in income,
10 THE SURVEY OF INCOME AND PROGRAM PARTICIPATION poverty, and program status) should be published; cross-sectional statistics should also be issued on a frequent schedule. The Census Bureau should also establish a research report series to include in-depth analytical and methodological studies of special topics re- lated to income and program participation. Data sources for these studies could include in addition to SIPP the March CPS income supplement and other surveys and administrative records. Finally, the Census Bureau should continue publications from the topi- cal modules and also establish a regular series of summary and in-depth reports from SIPP on the dynamics and correlates of major demographic and employment transitions (e.g., marriage, retirement). Recommendation 6-2: The Census Bureau should ensure that its analysis staff, in addition to preparing the regular publications from SIPP, are able to undertake an ongoing program of research and development into effective means of analyzing and presenting SIPP statistics and are able to stay well- versed in relevant policy issues arid analytical techniques. Recommendation 6-3: The Census Bureau should continue to develop im- proved microdata products from SIPP to support policy analysis and social science research. Priority improvements include: · moving toward a goal of releasing core data files within 6 months after the end of data collection; . producing calendar-year files that combine panels, in addition to wave and panel files; . determining, in consultation with users, changes and additions to the file contents that would assist their analyses; and · developing additional ways of delivering SIPP microdata products to users, such as by means of high-storage capacity compact disks (CD-ROM) and an improved on-line data extraction system. Recommendation 6-4: The Census Bureau should work to improve docu- mentation and related user information services for SIPP. Priority improve- ments include: · making use of CAPI and database management system technology to fully integrate documentation (including frequency counts for variables) and data; · developing documentation for recoded variables and the types of imputations that are performed for missing data in SIPP; . developing means to update key explanatory documents, such as the SIPP Users' Guide, on a more frequent basis; · restoring and expanding information and training programs, such as training sessions, working papers, and compilations of professional society presentations; and
SUMMARY 1 1 · maintaining effective channels of communication for users to feed back problems and suggestions and learn of the Bureau's response, and for users to be informed of new developments in the survey and its data prod- ucts. Methodological Research and Evaluation Recommendation 7-1: The Census Bureau should support methodological research and evaluation for SIPP leading up to and following the survey redesign. The research program should include the following components: · research to improve the format and wording of the questionnaire, making use of record-check studies and, to the extent possible, of findings from the current program of cognitively based questionnaire expenmenta- tion; . research targeted to other aspects of the current redesign (and to possible design changes later on), including the length of the recall period, screening techniques to obtain larger sample sizes for subgroups of interest, and data collection modes (the best combination of computer-assisted per- sonal and telephone interviewing and the possible role of centralized tele- phoning); · research on issues of estimation and data use, taking into account the features of the redesign and including ways to improve cross-sectional and longitudinal weights, imputation procedures, and population coverage; · research to evaluate the success of major elements of the redesign (e.g., the attrition effects of longer panels); and · a quick-response capability to address unanticipated problems with the implementation of the redesign. Recommendation 7-2: The Census Bureau should undertake continuous monitoring of error levels in present and future SIPP panels and regularly provide information on errors to users, in periodic updates of the SIPP Quality Profile and other publications. Recommendation 7-3: We strongly support the Census Bureau's program of cognitively based research and experimentation with the SIPP questionnaire, which could contribute to questionnaire improvements for the current redesign and perhaps, in the future, to a major revision of the questionnaire and inter- viewing procedures. The Bureau should subject the cognitive work to rigor- ous evaluation, including record-check studies to evaluate data quality. Management and Oversight Recommendation 8-1: To be as effective as possible in carrying out its responsibilities to produce timely, comprehensive, relevant, high-quality,
2 THE SURVEY OF INCOME AND PROGRAM PARTICIPATION and analytically appropriate statistics on income and program participation, the Census Bureau should establish a senior-level position of project direc- tor for the Bureau's income surveys, SIPP and the March CPS income supplement. That position should include full management and budgetary authority for the income statistics program and sufficient resources to ob- tain the level of analysis staff that is needed to provide substantive guidance to the program, prepare reports, conduct analyses, and evaluate analytical concepts and methods. The person who fills this position should have recognized substantive expertise in topics related to income, poverty, and assistance programs, combined with strong survey management skills. Recommendation 8-2: We support the Census Bureau's efforts to obtain outside advice about the SIPP program and encourage the Bureau to further strengthen its advisory mechanisms. The Bureau should regularly seek advice about the content, overall design, and goals of SIPP from federal agency users and from other users, including academic researchers. The Bureau should also regularly seek advice about technical matters from ex- perts in the field. Working groups should be formed or continued in three main areas: (1) survey methods and evaluation of ways to improve data quality; (2) conceptual and analytical issues in the development of appropri- ate income and program statistics from complex longitudinal data; and (3) microdata products, documentation, and means of data access.