6

Adequacy of the Content of the 2014 SIPP

Reengineering the Survey of Income and Program Participation (SIPP) represented a tremendous opportunity to strengthen the survey’s ability to address its main objectives while tackling changes in available resources. The purpose of this chapter is to assess the changes made to the content subject matter covered by SIPP and whether these changes are likely to affect SIPP’s ability to adequately achieve its main objectives. These objectives include providing accurate measures of income and program participation and comprehensive information about the principal determinants of income and program participation. Furthermore, as a longitudinal survey of moderate length, SIPP focuses on short-term (intrayear) changes in income and program participation and contemporaneous changes in the principal determinants of income and program participation. This focus distinguishes SIPP’s objectives from the objectives of cross-sectional surveys, such as the Current Population Survey Annual Social and Economic Supplement (CPS ASEC) and long-term longitudinal surveys such as the Panel Study of Income Dynamics.

The chapter provides (a) an overview of the content of prior SIPP panels, (b) a discussion of the content-related challenges faced during the reengineering process, (c) an overview of the 2014 SIPP content, (d) a delineation of major changes, (e) the handling of part-year residents, (f) a discussion of the potential impact of these changes on the ability of the SIPP program to achieve its main objectives, and (g) the study panel’s conclusions and recommendations regarding content.

OVERVIEW OF PRIOR SIPP CONTENT

The comprehensiveness of previous SIPP panels, both in breadth and depth, was perhaps unrivaled among nationally representative socioeconomic surveys. As described in Chapter 2, SIPP was composed of a core survey questionnaire plus topical modules (TMs). Box 6-1 lists the sections of the core survey questionnaire. The core survey itself surpassed the CPS ASEC with respect to data collection on demographic characteristics, employment, health insurance, expenses, sources of income, and program participation. The demographic characteristics collected included age; gender; race and Hispanic origin; language spoken and linguistic isolation; citizenship; Armed Forces and veteran status; educational attainment and extent of current enrollment; marital status; and household, family, and subfamily size and composition, including recent changes in household composition.

The income information collected in each 4-month period was obtained from an extensive set of questions:

• Employment income—gross earnings for each of two jobs plus total earnings from all other jobs, salary drawn from up to two businesses owned, casual earnings, and severance pay;
• Business income—net profit/loss from each business owned (up to two);
• Retirement income—federal civil service pensions, Railroad Retirement pensions, state government pensions, local government pensions, company pensions, union pensions, veteran’s pensions, U.S.

• military retirement, income from paid-up life insurance, draws from a qualified retirement savings account or plan (e.g., Individual Retirement Account, Keogh Plan, Section 401K defined contribution plan, or thrift savings plan), Social Security retirement benefits, and lump sum pension/retirement payments;

• Asset income—income from savings accounts, checking accounts, certificates of deposit, mutual funds, U.S. government securities, municipal or corporate bonds, stocks, margin accounts, rent, royalties, and other nonretirement financial investments;
• Program income—state and supplemental unemployment benefits, Temporary Assistance for Needy Families (TANF), foster child care payments, federal and state Supplemental Security Income (SSI), Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) payments, Supplemental Nutrition Assistance Program (SNAP) (i.e., food stamps), general assistance/relief, short-term cash assistance, (household level) energy assistance, clothing assistance income, food assistance income, transportation assistance in the form of gas vouchers and subway tokens, and other government income;
• Disability-related income—Social Security disability benefits, employer disability payments, veteran’s disability compensation, workers’ compensation, and sickness, accident, and disability insurance payments; and
• Other income—alimony, child support payments, survivor’s income, money from relatives or friends, and miscellaneous income.

Data on the receipt, but not the amount, of child care assistance, housing assistance, and education-related income were also collected, including grants, scholarships, and tuition remission from school; teaching/research assistantships from the school; grants or scholarships from the state; grants or scholarships from other sources; employer-provided educational assistance; and other financial aid. The core survey also collected data on the start of spells of participation (and reason(s) for receiving and reason(s) for stopping) for TANF, general assistance, other welfare programs, SNAP, WIC, and SSI.

The TMs were a major strength of SIPP. They typically collected detailed information about the determinants of program participation that are (a) less subject to intrayear change and/or (b) too detailed to fit into the core.

Table 6-1 shows the titles of TMs from the 2001, 2004, and 2008 panels and the waves in which they were fielded for each panel. The TMs of the first two waves captured historical details on key economic and demographic variables, including program receipt, employment, work dis-

TABLE 6-1 Topical Modules of the 2001, 2004, and 2008 Panels, with Designated Waves

NOTE: Asterisks indicate topical modules that only appear in the 2008 panel.

SOURCE: Panel generated with data from the Survey of Income and Program Participation history provided (see https://www.census.gov/).

ability, education and training, marriage, migration, fertility, and household relationships. Many TMs were repeated over multiple waves, collecting greater detail than the core questionnaire on income and expenses, including work-related expenses, taxes, assets and liabilities, retirement plan coverage and accounts, other assets, medical expenses/utilization, child support paid, and dependent care. Other TMs collected information on other measures of economic well-being for children and adults, as well as other determinants of economic well-being and program participation, including disability, informal caregiving, and employer health benefits. The TMs also served as a valuable resource to the policy and research community because they allowed for fielding questions on timely policy issues, such as economic stimulus programs, and for testing new questions on topics such as disability, educational certificates, and professional licenses.

CONTENT-RELATED REENGINEERING CHALLENGES

Converting SIPP’s core survey and interspersed TMs to an annual survey was quite a challenge.¹ It also presented a tremendous opportunity to streamline, innovate, and improve the data collected and thus advance the study of short-term dynamics of program participation and income. Questions from the prior SIPP panels could be kept, expanded upon, or dropped. New questions could be added. Many decisions had to be made.

The 2014 SIPP survey and Social Security Administration (SSA) Supplement have their roots in the planning for the Dynamics of Economic Well-Being System (DEWS).² While designing the DEWS, the Census Bureau held numerous stakeholder meetings in 2006 (as an example, see Brookings Institution, 2006) and sought stakeholder³ input on content by circulating, collecting, and analyzing a content matrix that was built using the 1993 SIPP instrument. The matrix listed variable names and descriptions, and stakeholders were asked whether each variable was critical and whether it should be collected on an annual or monthly basis. The results suggested a broad need for most of the SIPP core questionnaire content and for select areas of the TMs. Content that was deemed critical by a minimum of 75 percent of stakeholders included demographics (types and relationships within family and subfamilies, marital status with spouse identifier, parent-child identifiers, age, race, gender, ethnicity, geographic location, educa-

___________________

¹ In fact, a midyear supplement was conducted between waves 1 and 2 to supply additional data needed by the SSA.

² DEWS is described in Chapter 2.

³ The Census Bureau prepared a list of critical stakeholders from federal agencies, academic institutions, and nonprofit organizations. Approximately 36 provided input, which the bureau incorporated into the content matrix. This unpublished matrix was provided to the study panel by Jason Fields, U.S. Census Bureau.

tional attainment and enrollment); program participation (TANF, SNAP, WIC, other welfare, public housing, free/reduced breakfast or lunch); labor force (employment/Armed Forces status, hours worked per job/business, earnings/pay rate per job/business, number of weeks worked, looking for work, working without pay, occupation, and industry); general income (Social Security, retirement, property, other income); asset ownership, income and value (rental property, savings, certificates of deposit, money market accounts/funds, bonds, stocks, and other financial investments); and health (employer-provided health insurance, Medicaid, Medicare, military health insurance, work-limiting disability) (U.S. Census Bureau, 2016).

OVERVIEW OF THE 2014 SIPP CONTENT

The content of the 2014 SIPP largely reflects stakeholder input and contains essentially the entire core survey and parts of many TMs from the 2008 panel. As described in Chapter 4, the questions that were kept, plus new questions, were placed in one of the four sections of the 2014 survey instrument: (a) preliminary questions using standard interviewing screens, (b) spell questions using event history calendar (EHC) screens, which are interspersed with (c) spell-specific questions using standard interviewing screens, and (d) post-EHC questions using standard interviewing screens. Chapter 4 discusses the structure of the 2014 SIPP instrument in detail and describes both the standard interviewing screens and the EHC screens. That chapter also discusses the content of the 2014 SIPP in some detail, and the reader is referred back to that chapter for specifics about content. As a summary, Box 6-2 lists the topics covered by the four sections of the 2014 SIPP.

The 2014 SIPP design allowed for client agencies to sponsor midyear telephone supplements. A supplement sponsored by the SSA was fielded in 2014; it collected information on the economic and social situation of people with disabilities and people in or approaching retirement. This is the only supplement to the reengineered SIPP that has been fielded within the time frame of this report.⁴ The SSA supplement drew content from several TMs, including the Work Disability, Marital History, Retirement Expectations and Pension Plan Coverage, Annual Income and Retirement Accounts, Functional Limitations and Disability–Adults, and Functional Limitations and Disability–Children TMs. The supplement topics are listed at the bottom of Box 6-2.

___________________

⁴ As of the time of publication of this report, the Census Bureau has made no public announcement that additional supplements are being planned.

DELINEATING MAJOR CONTENT CHANGES

Creating a variable-by-variable crosswalk between the prior SIPP panels and the 2014 panel is a difficult task, but such a crosswalk has been promised by the Census Bureau to aid data users. A crosswalk was not available for the study panel’s work, but a partial crosswalk is now available in the SIPP 2014 Panel Users’ Guide (U.S. Census Bureau, 2016). Any such crosswalk is bound to be imprecise due to changes in the reference period and potential changes in wording for various questions. Creating a crosswalk by major topic areas is feasible but would not be terribly informative, because by design the 2016 SIPP draws from all of the major topic areas covered by the prior SIPP core survey and many TMs to lesser, similar, or greater extents.

The SIPP 2014 Panel Users’ Guide (U.S. Census Bureau, 2016) documents the major changes in content between the 2008 core survey and 2014 panel, excluding the SSA Supplement, and in places it mentions content derived from the TMs. The following sections describe, by topic, the content of the 2014 SIPP and related major changes from the 2008 SIPP, based on the SIPP 2014 Panel Users’ Guide (U.S. Census Bureau, 2016) and other documentation. Most of the items discussed are brought forward from the 2008 core questionnaire, but other items came from one of the

TMs. In this discussion, the report does not always make that distinction. Because there are no TMs in the new survey, items brought forward from a TM into the 2014 instrument can be viewed as having received a “promotion” in that they have changed from a (generally) one-time collection to an annual collection.

Income and Program Participation

Earnings

Detailed information is collected for up to seven jobs, with more limited information on a collective “all other jobs.” Separate questions are asked about tips, commissions, overtime pay, and bonus pay. The questionnaire allows up to two changes in wage/salary pay rate and hours worked for each job over the reference period. (In the core survey of prior SIPP panels, up to five changes could be recorded over the course of a year: one change with each of the three interviews and two changes between the three interviews.)

SNAP, TANF, WIC, SSI, General Assistance, and Other Welfare Assistance

Respondents can report up to three spells for SNAP, TANF, WIC, SSI, General Assistance, and other welfare.^5,6 In addition, there is a continuation flag provided that indicates whether SNAP (1) ended in the last month of the reference period, (2) ended after the last month of the reference period but before the interview month, or (3) was ongoing as of the interview month. The same is true for TANF, WIC, and General Assistance. However, an option to report other welfare assistance is no longer included. For SSI, a single amount variable has replaced separate variables for federal SSI payments and supplemental state SSI payments. In addition, a proxy respondent’s report about child SSI receipt is recorded on the child’s record, whereas the receipt of SSI for a child was previously recorded on a parent’s record.

Social Security Income

No major changes have been made in the redesigned SIPP.

___________________

⁵ Other welfare programs are increasingly important among the poor single or unemployed parent community as states move people off TANF.

⁶ While this is correct in principle, the panel found (while participating in mock interviews) that there is no easy way for an interviewer to record a 1-month drop-out in SNAP. The panel also learned that once the instrument’s memory was filling, opening a new window to begin a new spell takes so long that it is problematic in practice.

Unemployment Insurance, Worker’s Compensation, and Veteran’s Benefits

No change has been made other than adding the collection of service-connected disability ratings.

Energy Assistance

The amount of energy assistance is no longer collected.

Other Assistance Income

The redesigned SIPP collects information on whether other governmental and nongovernmental assistance income, such as money and/or vouchers for food and clothing and transportation assistance (e.g., gas vouchers and subway passes) is being received. However, the amounts of these forms of assistance are no longer collected.

Retirement, Disability, and Survivor’s Income

Respondents are no longer asked whether they receive certain types of income (such as from a local government pension), followed by a question that asks whether the receipt was due to retirement, disability, and/or survivorship. Instead respondents are asked about the receipt of retirement income and if so, the type(s), months(s), and amount(s). Similar questions are asked separately about disability and survivor’s income.

Retirement Income from Life Insurance

Life insurance is now collected as an annual amount with no information about the dates of receipt.

Lump Sum Severance Pay/Retirement Plan Income

Respondents continue to be asked about the receipt of lump sum severance pay and retirement income payments, but now they are also asked about the receipt of lump sum severance pay and retirement income that has been rolled into other retirement plans.

Child, Foster Child, and Spousal Support Received

No major change was made with respect to income from these sources. However information previously collected on the Child Support TM is not collected. This includes information about custody, health care, paternity, and other child support agreement–specific information.

Miscellaneous Income

No major changes have been made.

Assets and Debts

The collection of asset information has been expanded to include annuities and trusts, educational savings accounts, businesses owned as an investment only, and student loans or education-related expenses. The number of businesses owned, as a job, is expanded from two to seven businesses. Separate items are collected for the value of savings accounts, money market deposit accounts, certificates of deposit, interest-earning checking accounts, non-interest-earning checking accounts, municipal bonds, and other financial investments. The available response options for asset values are expanded to include predefined brackets when the specific amount is not available. Person-level recodes are generated. Joint ownership is identified, even for nonpartner relationships. Vehicle values are provided using data from the National Automotive Dealers Association. These values are now generated anew in each wave rather than obtained from a single TM and then depreciated. In contrast to the earlier SIPP panels that collected monthly interest, dividend, rental income, royalties, and other financial investment income, the reengineered SIPP only collects annual amounts for these sources with no information on the timing of the receipts.

Tax Data

Dependent status of persons between ages 15 and 25 is now collected. However, detailed tax return information—previously collected in a TM—is no longer collected (including the number of exemptions, gross income reported, net tax liability, tax deductions, Earned Income Tax Credit (EITC) amount, capital gain and loss, and paid property taxes). Information is still collected on the filing of a federal tax return, filing status (single, joint, etc.), EITC receipt, and whether specific children are claimed as dependents. An important loss of information, compared with earlier panels, is that tax information is not collected from tax filers with whom the (household) parent does not live—for example, children who live elsewhere.

School Meals

Participation in free or reduced cost school lunch programs is reframed from whether “any” of a child’s lunches are part of the program to whether the child “usually” gets lunch at school with a response option on how it is paid for. The same is true for school breakfast programs.

Child Care

Respondents continue to be asked about Head Start enrollment, and new follow-on questions were added to improve the quality of Head Start utilization estimates. However, the number of hours in child care and the location are not collected. Child care payment information is collected based on all arrangements used for all children. In prior SIPP panels, child care payment information was collected by the type of child care arrangement.

Scholarships

Respondents are no longer asked whether they were awarded grant, scholarship, tuition remission, or other forms of educational support.

Expenses

Child, Foster Child, and Spousal Support Paid

No major changes have been made.

Dependent Care Expenses

The amount paid is now collected, consistent with the Dependent Care TM in the 2008 panel.

Health Care Utilization and Medical Expenditures

Over-the-counter and non-over-the-counter expenses are collected separately rather than combined. The use of, rather than the employer’s offer of, a Flexible Spending Account is collected. A question about drug/alcohol treatment is no longer asked, nor is a question regarding dental sealant.

Other Outcomes of Interest

Adult Well-Being

The scope of adult well-being information decreased from more than 100 items in the 2008 wave 6 Adult Well-Being TM to about 15 items, which are located in the post-EHC portion of the 2014 SIPP.

Child Well-Being

This section was expanded to contain a series of four questions (child works hard in school, does enough to get by, always did homework, and cares about school) similar to an established measure of school engagement. However, overall the number of questions is reduced from approximately 60 items in the Child Well-Being TM to approximately 20 questions.

Other Related Economic Activities

Employment

As mentioned earlier with regard to earnings, the redesigned SIPP collects detailed information for up to seven jobs or businesses in the reference year compared to two jobs and two businesses in each 4-month reference period. The redesigned SIPP continues to collect aggregate information on incidental and casual earnings in the reference period.

Commuting and Work Schedule

Information is collected for up to seven jobs during the year, and commuting and work schedule information are collected concurrently. In the 2008 panel, data on work schedule were collected in TMs of waves 5 and 8 and information on work-related expenses was collected in TMs of waves 4, 7, and 10. Response options were expanded from 5 to 10 (drove own vehicle; rode in someone else’s vehicle/van pool; used bus, rail, or other public transportation; walked; bicycled; drove company vehicle; worked at home; and other).

Health Insurance

In prior SIPP panels, whether respondents had health insurance coverage and type of coverage were the only health insurance questions asked on a monthly basis (via the core survey). All other questions were asked on an annual basis, such as premium cost information. In the 2014 SIPP, all health insurance–related questions are collected on a monthly basis. Furthermore, detailed reasons for not taking up health insurance (such as “too expensive”) were asked for all types of health insurance, not only for employer-sponsored health insurance as on previous panels. Both children (via adult proxy) and adults received the same set of health insurance questions, as opposed to different questions as in previous panels. Military coverage is now asked separately rather than as part of private coverage, as in previous panels.

As an aside, wave 2 of the 2014 panel wave added questions on coverage by Marketplace Exchange plans and whether premiums for such plans were subsidized. This demonstrates some flexibility to add new content on timely issues from wave to wave—something threatened with the loss of the TMs.

Determinants of Income and Program Participation

Household and Family Relations

The relationship of everyone to everyone else, including persons who lived in the household during the reference period but were no longer residing in the household at the time of the interview (“Type 2” persons), is now generated for each month of the reference period rather than just once—in the wave 2 Household Relationships TM. Parent pointers are now gender-neutral to facilitate the reporting of same-sex parents. The response options for relationship to household reference person have been refined to distinguish between same-sex and opposite-sex couples and have been increased to include (consistent with the 2008 wave 2 TM) mother-, daughter-, father-, son-, brother-, and sister-in-law, plus aunt, uncle, niece, and nephew. On the 2008 SIPP, the relationships of household members were defined with respect to the “household reference person.” However, in the 2014 SIPP the identification of who is the household reference person is not carried forward on data for each month; that information is only available for the interview date. The relationships of individual household members are established to all other household members each month.

Fertility

Shared parentage by persons within the household is identified even if the child was not present in the household. More information is provided on men’s parentage. However, questions from the Fertility History TM that focused on a woman’s first-born child are not included (e.g., employment and leave before and after birth, where child lives, and whether child is deceased).

Marital Status

Monthly marital status information on the 2014 survey is collected in the preliminary section and the EHC portion of the questionnaire. Information similar to the 2008 wave 2 Marital History TM was collected in the SSA supplement. Same-sex and opposite-sex married partners are coded using relationship information. Additional questions are asked of

unmarried partners to distinguish registered domestic partnerships from civil unions. It is possible to connect unmarried partners using an unmarried partner pointer.

Education

Questions related to licenses and certificates are included, similar to the questions on the 2008 wave 13 Professional Certifications and Educational Certificates TM. The current questions were developed by the Federal Interagency Working Group on Expanded Measures of Enrollment and Attainment. Enrollment information for individuals under age 15 is now collected at the same time as other household members, as opposed to being collected in a periodic Child Well-Being TM.

Disability

A six-question sequence that is used in the American Community Survey and the CPS (incoming rotation groups) to address limitations (with hearing, seeing, cognitive activities, ambulatory activities, self-care activities, and independent living activities) is included in the redesigned SIPP. Three questions from the Child Disability TM are included as well; these address limitations in playing with other children of the same age, doing regular school work, and doing ordinary activities. The SSA Supplement included all questions from the Adult Disability and Child Disability TMs, with some minor revisions.

Language

The basic question remains the same; however, the number of categories for language spoken by an individual and in the home has been increased, and the categories reported in the public release file have been changed (e.g., Arabic was made a separate category and non-Russian Slavic languages were moved into the “other language” category).

Nativity

The basic questions remain the same; however, for non-U.S. nativity the number of response categories available in the public release file has decreased (e.g., the Caribbean, Central America, and South America are now combined into one category). Former wave 2 TM questions on the state and country of birth, immigration status on arrival (whether lawful permanent resident or other), year of entry, and biological parents’ place of birth are now included in the annual questionnaire.

Residence

The questionnaire allows up to five residential spells within the reference year. Spells are collected for time spent in group quarters, medical institutions, emergency or transitional shelters, tent or trailer sites, cars or vans, and living abroad (although this information is not available in the public release file). A new question was added about the reason for moving. The metropolitan principal city is coded for large metropolitan areas.

Parent Mortality

New questions are included about the month and year of birth of the respondent’s biological mother and biological father, whether deceased, and if so when.

HANDLING OF PART-YEAR RESIDENTS

Under both the old and new SIPP designs, persons who joined a SIPP household after the wave 1 interview are eligible to be interviewed as sample members for any wave in which they remain household members at the time of the interview. However, they are not considered panel members themselves and are no longer eligible to be interviewed once they leave the SIPP household. If they join a SIPP household during a wave in which they are eligible to be interviewed, their interview covers the entire year, as per sample members. If they are unable to be interviewed, their data are collected by proxy or imputed, yielding complete data for the months they were present.

Under the old design, for any wave after the first, persons who were present for at least part of the reference period but not the interview would have been subject to full data collection for the months that they were present—again, by proxy if possible or imputed if not. The goal was to have complete data for everyone who shared a household with a SIPP panel member for the months that they were present. Such persons have records and weights for all of the months that they resided with panel members. This enabled the calculation of income and poverty statistics for the full family and household in every month of every wave.

Under the redesign, the longer reference period led to a change in the amount of data collected for part-year residents who are not eligible to be interviewed. Very little information is collected for such persons, and they are not given person records or weights. For example, the only income data collected for these persons are estimates of annual income from all sources. Some data users are concerned about a general loss of information, including tax data, for these individuals. Because of the limited data obtained for

these Type 2 persons, the Census Bureau calculates income and poverty statistics with and without such persons, so that users of the data can choose whether or not to account for these persons in their analyses.⁷ However, the relative paucity of data collected for Type 2 persons on the 2014 SIPP represents a loss of information relative to prior SIPP panels.

Under the old design, wave 1 was handled differently from later waves. The same is true of the new design, but the two approaches differ. Under the old design, household composition was frozen at the time of the wave 1 interview. All panel members who were living together at the time of the interview were treated as having lived together for the entire 4-month reference period except for children born or adopted into the household during the reference period. No information was collected on persons who lived with a panel member for any part of the reference period but were not in the sample household at the time of the interview.

Under the redesign, the 12-month reference period makes it more problematic to ignore changes in household composition that may have occurred during the period. Panel members who lived for part or even all of the reference period in a household other than the one from which they were sampled are now identified as having done so. Any persons with whom they lived during their period away from the sample household are treated as Type 2 persons—just like persons who lived with the sample household for part or all of the reference period but are no longer present at the time of the interview. The data collected for these persons are the same limited data described above for Type 2 persons. In the next chapter, the study panel explores the impact of including or excluding Type 2 persons from wave 1 poverty statistics.

The household dynamics captured in the wave 1 data for the 2014 panel have implications beyond the introduction of Type 2 persons into the data for that wave. In prior SIPP panels, cross-sectional weights in waves after the first were adjusted whenever new persons entered SIPP households—including situations in which panel members left their original households to form new households with other persons. The weighting adjustments—based on the concept of multiplicity weighting—ensured that when household and family weights were assigned (equaling the weights of household and family reference persons), the weighted numbers of households and families over the whole sample reflected the household and family dynamics in the population. For example, if a panel household combined with another household, this represented a reduction in the number of households in the population; this would be reflected in a reduction in the

___________________

⁷ The data for these persons appear on the records of all other household members.

person and household weights for the panel household.⁸ If a panel member split off from a panel household to join another household, representing no net change in the number of households in the population, the panel member’s weight would be adjusted downward (and the new housemates would receive suitably adjusted person weights as well), yielding a lower household weight than if the panel member had formed a one-person household (and, in so doing, had increased the number of households in the population).

The household dynamics of the wave 1 sample for the 2014 panel are like the dynamics of a wave 2 sample in reverse. The number of sample households across which the panel sample members are distributed grows between the end of the year and the beginning of the year. But with Type 2 persons receiving no weights, there are no multiplicity adjustments to the cross-sectional weights of the panel members. Weights change only to match the population totals to which the sample weights are post-stratified. If each unique sample household was assigned the weight of a panel member, the weighted number of households in January would be higher than in December rather than the reverse. If the actual householder among the panel members and Type 2 people could be determined (the survey does not collect this information) and that person’s weight assigned to the household, a reasonably correct household total could be obtained. However, this would entail households with Type 2 persons as heads receiving weights of zero, which is not especially satisfying. This same situation will apply to waves 2 and later, where the growth in sample households occurs in the forward direction. A difference between wave 1 and later waves of the new design is that nonsample persons who join panel households and remain long enough to be interviewed will receive weights. How such persons compare in number to Type 2 persons will not be known until the data collected in waves 2 and later can be examined.

The Census Bureau’s decision to assign no household or family weights to the 2014 panel reflects an appreciation of the underlying problem created by the treatment of Type 2 persons, but this approach will present challenges to users wishing to conduct analyses at the household and family levels.

___________________

⁸ With multiplicity weighting, the adjustment is not proportionate to the change in households at the level of the individual sample household but over the whole sample. Consider that if a panel household with two members merges with a nonpanel household consisting of one member, the nonpanel household, in effect, is represented by another panel household with one member merging with a nonpanel household with two members.

POTENTIAL IMPACT OF CHANGES

Changes in the content of SIPP based on the redesign, particularly the reduction of information collected previously on TMs, will necessarily result in loss of, or major changes in, certain uses of SIPP in the future, including the need to rebuild long-standing models. Below, the report provides some examples.

Loss of Variables That Are No Longer Collected

The SSA found that the redesigned questionnaire did not contain all of the variables that it needed in support of ongoing SSA programs. These variables used data from the previous TMs for Work Disability, Marital History, Retirement Expectations and Pension Plan Coverage, Annual Income and Retirement Accounts, Functional Limitations and Disability-Adults, and Functional Limitations and Disability-Children. As part of the overall redesigned program, the use of supplements was envisioned to address such specific needs. The SSA was able to sponsor a supplement to obtain the additional variables they needed. This supplement was fielded between wave 1 and wave 2 as a separate follow-on telephone survey of all household members responding on wave 1. This first fielding of a supplement suffered from a low response rate (52.2%, see Table 7-21 in Chapter 7) that triggered concerns from the SSA. The report discusses this in more detail in Chapter 7.

The panel has heard concerns about the impact of some other missing variables, the uses of which are unlikely to support the cost of a full supplement. For example, the redesign kept the food security questions but dropped other important material hardship variables that formerly appeared in the TMs such as evictions and phone shutoffs. The wording was modified for related variables such as “falling behind on rent/mortgage, heat/power bills, and utility shutoffs.” The redesign also dropped questions on medical/dental hardships. Some researchers are concerned that these variables were important to validating that income-poor people suffer in real-world ways.

A Case Study: Microsimulation Modeling of SNAP

Chapter 3 described the Food and Nutrition Service’s use of a SIPP-based microsimulation model to assess the impact of prospective policy changes to SNAP. The current SIPP-based model uses data from wave 10 of the 2008 SIPP panel to generate baseline simulations of SNAP that are designed to represent the 2015 fiscal year. Household demographic and economic data from August 2011 are supplemented by a variety of

additional data drawn from several TMs. The next SIPP-based model will rely on data from the 2014 panel. Changes to SIPP data as a result of the redesign will require the construction of a modified model and will affect its performance. Some of the changes to the data will enhance the model’s capabilities while others will present challenges. Drawing on two reviews by Mathematica and Decision Demographics, the panel describes key implications of the redesign.

Mathematica’s MATH SIPP+ microsimulation model uses more than 600 variables from the 2008 SIPP panel. Mathematica and Decision Demographics reviewed how the reengineering of SIPP will affect not only the microsimulation model but also other SIPP-based research on SNAP. Of the 600 variables, 38 core variables and 50 TM variables are not available in the 2014 panel.⁹ Furthermore, of those that will be available, not all are perfect matches to their 2008 counterparts: 116 are described as matching “well but with qualifications” and 45 are described as matching “but with differences” with respect to question wording, universe, or variable construction. About a third of the variables that will remain available are derived from other variables, so their exact construction is not entirely clear as yet.

A key element of the SNAP simulation involves the formation of SNAP eligibility units within each sample household (Czajka et al., 2015). A SNAP eligibility unit is the set of persons whose eligibility is jointly determined, based on their composition, income, resources, and selected other factors. The U.S. Department of Agriculture defines a SNAP unit as those “individuals who share a residential unit and customarily purchase and prepare food together” (Gray and Genser, 2014). Household members who do not purchase and prepare food together may be able to apply as separate SNAP units, although there are restrictions based on relationships. The improved relationship data in the redesigned SIPP are expected to enhance the simulation of SNAP units, but the elimination of questions on the sharing of food expenses will weaken the unit simulations. Wemmerus and colleagues noted that the redesigned SIPP contains indicators of shared ownership, payment of expenses, program coverage, and shared resources that could collectively provide a proxy for the missing information.¹⁰ The limited data collected on Type 2 persons could make it difficult to determine their most appropriate unit assignment, although the fraction of households that include such persons is small.

___________________

⁹ Memorandum from Mathematica Policy Research (K. Cunnyngham, B. Schechter, N. Wemmerus, C. Foran, S. Tordella, and P. Foran) to Jenny Genser, Food and Nutrition Service, U.S. Department of Agriculture, June 30, 2015.

¹⁰ Memorandum from Mathematica Policy Research (N. Wemmerus, T. Godfrey, and S. Tordella) to Jenny Genser, Food and Nutrition Service, U.S. Department of Agriculture, 2017.

The current SIPP-based model incorporates TM variables collected in several different waves, which means that they have (1) different reference periods and (2) will not be available for some respondents in the August 2011 base month. For TM variables that are retained in the redesigned SIPP, having them collected in the same interview as the household demographic and economic data used by the simulation model represents an improvement that will simplify the construction and enhance the quality of the model. Furthermore, the TM variables retained in the redesigned SIPP will be collected annually, which means that they will be contemporaneous with whatever wave is chosen to serve as the base for the simulation model. In the 2008 panel, by contrast, several variables applicable to immigrants were collected only in wave 2, so they were not available for immigrants who moved into sample households after wave 2. These variables will be collected annually in the redesigned SIPP.

Some of the relevant core and TM variables have been enhanced in ways that will improve their utility for SNAP simulation modeling. These enhancements include the collection of dependent care expenses; more extensive measures of functional disability; the resumption of collection of monthly income data for several sources that had been relegated to less frequent collection with imputation of intervening months; and improved valuation of vehicles, which constitute an important asset in determining SNAP eligibility in states that continue to apply a resource test.

At the same time, several variables used in the simulation model have been eliminated from the redesigned SIPP.¹¹ These include

• Amounts received in energy, educational, transportation, short-term case, or private housing assistance; other welfare; WIC; and other miscellaneous income sources;
• Reason for not starting a job, missed weeks of work, and weeks not worked due to health issues;
• Reasons for applying for or stopping other welfare;
• Reasons for government pension receipt;
• Reasons for receipt of workers’ compensation or employer disability payments;
• Share of rental property held with another party;
• For participants in training programs, source of sponsorship; and
• Whether and in what year immigration status changed to lawful permanent resident.

___________________

¹¹ Memorandum from Mathematica Policy Research (K. Cunnyngham, B. Schechter, N. Wemmerus, C. Foran, S. Tordella, and P. Foran) to Jenny Genser, Food and Nutrition Service, U.S. Department of Agriculture, June 30, 2015.

In addition to adjusting for these changes in variables, the modelers will likely have to reconsider their choice of base month. December offers advantages in terms of its proximity to reported assets, but December is also an unusual month because of its bumps in short-term employment, spending, school and work breaks, and travel—all associated with the holidays.

CONCLUSIONS AND RECOMMENDATIONS REGARDING CONTENT

CONCLUSION 6-1: Overall, the subject area content of the Survey of Income and Program Participation has remained remarkably consistent, given the large changes in survey methods.

Perhaps the greatest improvement in SIPP content from the redesign is the collection of detailed information on up to seven jobs, which will provide an unprecedented window into intrayear dynamics and the holding of multiple jobs simultaneously and over the course of a year (however, see our findings in Chapter 7 on employment transitions). This information will be particularly important when investigating the labor market activities and wages/salaries of persons in vulnerable populations and in transitionary situations, such as job loss, divorce, and graduation from a degree program.

The most significant contraction of SIPP content may be in the collection of tax information. The 2014 SIPP protocols did not ask respondents to refer to their federal tax returns for amounts reported. While this may have reduced respondent burden, the lack of detailed tax information will likely increase the variability of estimated tax liability and will require the estimation of EITC amounts. The arrival of a tax refund check/direct deposit represents a potential intrayear change in income and may influence other income-related decisions. The federal tax code is an important lever in welfare policy, as evidenced by the EITCs role in welfare reform. Measuring its impact on household, family, and personal income is essential for understanding how government policy influences the well-being of adults and children.

Variables related to material hardship such as evictions and phone shutoffs form a second area of significant contraction. The loss of data on these topics was discussed in the previous section of this chapter. A third area of significant contraction required the SSA to sponsor a supplement to meet its minimum data needs. A fourth area of contraction saw the loss of variables currently used in important microsimulation models that depend on SIPP data.

CONCLUSION 6-2: While most of the core content and much of the key content of the topical modules (TMs) of prior panels were included in the 2014 Survey of Income and Program Participation (SIPP), important variables were lost. In fact one agency—the Social Security Administration (SSA)—found it necessary to commission a supplement to meet its information needs. Existing models dependent on SIPP data, such as those produced for the SSA and the Food and Nutrition Service, will need to be modified to varying degrees to adapt to the new data availability. Important variables related to material hardship are no longer collected. Additionally, the lack of periodic TMs limits the ability to field new questions addressing timely policy issues.

For instance, a question from the Medical Expense/Utilization TM about the receipt of treatment for a drug/alcohol problem was not included in the 2014 SIPP. This and additional questions about substance abuse, had they been included, may have allowed the 2014 SIPP to investigate opiate abuse and its impact on families—issues that were not as critical when the content of the 2014 SIPP was designed.

RECOMMENDATION 6-1: The Census Bureau should reevaluate whether critical content from the Social Security Supplement can be effectively added to the Survey of Income and Program Participation questionnaire.

Whether the contents of the 2014 SIPP will adequately address the main objectives of the SIPP program will be revealed in the years to come, as the data are analyzed and findings enter the literature and policy discussions.

This page intentionally left blank.