National Academies Press: OpenBook

Review of WIC Food Packages: Improving Balance and Choice: Final Report (2017)

Chapter: Appendix K: Study Design Strategies for Reducing the Effects of Selection Bias in Studies Comparing WIC Participants to Others

« Previous: Appendix J: Nutrient and Food Intake of WIC Subgroups: Analytical Methods and Results
Suggested Citation:"Appendix K: Study Design Strategies for Reducing the Effects of Selection Bias in Studies Comparing WIC Participants to Others." National Academies of Sciences, Engineering, and Medicine. 2017. Review of WIC Food Packages: Improving Balance and Choice: Final Report. Washington, DC: The National Academies Press. doi: 10.17226/23655.
×

Appendix K

Study Design Strategies for Reducing the Effects of Selection Bias in Studies Comparing WIC Participants to Others

In Chapter 4, the committee reviewed study designs that could to some degree ameliorate the challenges to evaluating the effects of the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC). In particular, specific types of study designs that could address the problem of selection bias, or other biases due to reverse causation or measurement error, were reviewed. In this appendix, the committee defines these types of study designs, and proposes ways that the U.S. Department of Agriculture’s Food and Nutrition Service (USDA-FNS) could apply these designs to the evaluation of the WIC program.

REGRESSION DISCONTINUITY

A regression discontinuity (RD) design compares people who are just below an eligibility threshold for a program or treatment with those who are just above it. Because the likelihood of an individual falling just above or below a threshold is not affected by self-selection, this method yields high internal validation,1 provided that these individuals do not manipulate their presence below or above the threshold (the method typically includes tests for this). The types of thresholds that are relevant to WIC and can potentially be used in RD designs include (1) income eligibility, (2) age after which one can no longer participate in the program, and (3) age at which the value of the program changes (i.e., the value of the food package

__________________

1 Internal validation can be indicative of causality regarding the relationship between WIC participation and measured outcomes.

Suggested Citation:"Appendix K: Study Design Strategies for Reducing the Effects of Selection Bias in Studies Comparing WIC Participants to Others." National Academies of Sciences, Engineering, and Medicine. 2017. Review of WIC Food Packages: Improving Balance and Choice: Final Report. Washington, DC: The National Academies Press. doi: 10.17226/23655.
×

changes). Versions of this design, which rely on comparing individuals with incomes just below and just above the eligibility threshold for WIC participation, require researchers to use the measure of income that is applied by the program. Similarly, versions of the design which rely on age thresholds require researchers to identify age thresholds relevant to the program (e.g. age 5 when children are no longer eligible for WIC). RD designs yield unbiased estimates of the impact of WIC, provided that the identifying assumptions are satisfied for individuals that are right at the eligibility thresholds.2 However, RD designs are not necessarily applicable to the WIC participants with the most need, who might have lower income-to-poverty ratios and may be affected differently by WIC.

Potential USDA-FNS Applications for WIC

Research studies could target comparisons at 185 percent of poverty, at age 5 when eligibility ends, and at eligibility/takeup at 1 year when formula is no longer provided. Research examining comparisons at the income threshold would compare WIC participants near but below the top of the income threshold (170–185 percent) with non-WIC participants just above the income threshold (185–200 percent), while controlling for the income-to-poverty ratio.3 Comparisons of the two age thresholds would involve what is known as a “fuzzy RD.” At age 5, when children that meet the income criteria are no longer eligible, they would be compared to themselves at periods before and after program participation, while controlling for age. At age 1, some families may choose not to participate (particularly formula-feeding families) because the value of the package is reduced with the loss of formula. Thus, if one could follow families who use formula throughout the first year and into the second year, comparisons between families who fail to recertify before and after age 1 might be useful. The RD approach could be made more feasible if USDA-FNS encouraged states to make administrative data on health outcomes and precise income-to-poverty and age (in months or exact days) available for large numbers of potential WIC participants (including participants and nonparticipants). Another strategy could be to collect data on all Medicaid participants near

__________________

2 Other assumptions for RD are either that the potential outcomes are continuous and have a number of continuous derivatives or that within some window close to the income eligibility threshold, the side of the threshold is randomly assigned. This is easy to test for income (i.e., are there more families with income levels just below the 185 percent of poverty threshold) and less necessary for age (i.e., age is generally well-captured in administrative data).

3 The income-to-poverty ratio is the ratio of family income to the federal poverty guideline. It is this ratio which must be below 1.85 for families to be eligible for WIC.

Suggested Citation:"Appendix K: Study Design Strategies for Reducing the Effects of Selection Bias in Studies Comparing WIC Participants to Others." National Academies of Sciences, Engineering, and Medicine. 2017. Review of WIC Food Packages: Improving Balance and Choice: Final Report. Washington, DC: The National Academies Press. doi: 10.17226/23655.
×

these same WIC age and income thresholds as all Medicaid recipients are adjunctively eligible for WIC. These data will provide estimates that have high internal validity for those near the age or income thresholds (and can be indicative of causality) but may not be externally valid for those far from the age or income thresholds.

COMPARISON GROUP

In a comparison group (CG) design, WIC participants are usually compared with income-eligible nonparticipants. In contrast to RD studies, these designs can be applied to groups of the lowest-income WIC participants. However, a major drawback of CG designs is that it is not possible to know if the analysis has accounted for all unobservable differences between the two groups. The analysis can be made more robust by including an additional comparison between the treatment and control groups when program changes are taking place in what is known as a differences-in-differences or interrupted-time-series design (i.e., a series of observations over time interrupted by introduction of an intervention). Thus, studies in which program changes are implemented randomly across locations are best suited for this design.

Potential USDA-FNS Applications for WIC

Application of this approach could be optimized if the timing and scope of rule changes in WIC and other programs affecting WIC use were made available. The period of new food package implementation offers one such opportunity. Larger sample sizes and access to linked WIC administrative data on WIC eligibility and participation would allow for useful outcomes as long as nonparticipants are somehow included. It is also key to have data on individuals who are potentially eligible but not enrolling in the program. This approach relies on assumptions of common trends and the absence of changes in the composition of the comparison and treatment groups after treatment.

INSTRUMENTAL VARIABLES

The instrumental variables (IV) approach considers a population of participants and nonparticipants whose decision to participate was based on a factor that is: strongly correlated with participation, cannot be directly related to the outcome, and only affects the outcome through participation. An example is distance of residence from the WIC clinic (ideally combined with variation in locations of clinics driven by clinics closing and opening). Distance (particularly driven by openings and closings) plausibly affects

Suggested Citation:"Appendix K: Study Design Strategies for Reducing the Effects of Selection Bias in Studies Comparing WIC Participants to Others." National Academies of Sciences, Engineering, and Medicine. 2017. Review of WIC Food Packages: Improving Balance and Choice: Final Report. Washington, DC: The National Academies Press. doi: 10.17226/23655.
×

use of WIC but should not otherwise have direct effects on dietary intake. Although this is a sound method for obtaining causal estimates, it can have low external validity (i.e., results will not be generalizable). Some researchers who have tried to use IV to estimate the impact of WIC have concluded that they were unable to identify an instrument that works well to predict WIC use.

Potential USDA-FNS Applications for WIC

As an example of how IV has been applied to evaluate WIC, Rossin-Slater (2013) used, as the instrument, whether a mother resided close to a WIC clinic at the timing of conception of her oldest child. Controlling for mothers’ fixed characteristics, this instrument proved to be a strong predictor of WIC participation in Texas during a period when many clinics were opening or closing. This type of instrument (relying on distance to a location where eligibility for a program is assessed) is common in the evaluation of the effects of other programs. Another IV application would rely on state differences in assessment of eligibility by income (e.g., disregards family units, the way that eligibility is assessed for the other programs such as Temporary Assistance for Needy Families, Supplemental Nutrition Assistance Program [SNAP] and Medicaid, which confer automatic WIC eligibility). For example, Ziliak (2016) demonstrated that states where SNAP uses broad-based categorical eligibility (which raises the gross income threshold for eligibility) have higher SNAP participation. This might permit use of these SNAP rules for IV estimation.

FIXED EFFECTS

The fixed effects (FE) approach uses variation in WIC participation across time, time and place, siblings within a family, or other variables, to estimate effects. This approach has some disadvantages, including lack of control for unobserved factors associated with participation in WIC that change over time (e.g., shocks to the economy) and lack of sufficiently detailed longitudinal data on participation over time and on outcomes.

Potential USDA-FNS Applications for WIC

This approach would be made more useful by encouraging more sharing of WIC administrative data and WIC data from other data systems, including data on families with not all siblings enrolled, as well as data from locations where program changes affect eligibility and enrollment.

Suggested Citation:"Appendix K: Study Design Strategies for Reducing the Effects of Selection Bias in Studies Comparing WIC Participants to Others." National Academies of Sciences, Engineering, and Medicine. 2017. Review of WIC Food Packages: Improving Balance and Choice: Final Report. Washington, DC: The National Academies Press. doi: 10.17226/23655.
×

REFERENCES

Rossin-Slater, M. 2013. WIC in your neighborhood: New evidence on the impacts of geographic access to clinics. Journal of Public Economics 102(C):51–69.

Ziliak, J. 2016. Why Are So Many Americans on Food Stamps? The Role of the Economy, Policy, and Demographics (Chapter 1). In SNAP matters: How food stamps affect health and well-being, edited by J. Bartfeld, C. Gundersen, T. Smeeting, and J. Ziliak. Stanford, CA: Stanford University Press. Pp. 18–48.

Suggested Citation:"Appendix K: Study Design Strategies for Reducing the Effects of Selection Bias in Studies Comparing WIC Participants to Others." National Academies of Sciences, Engineering, and Medicine. 2017. Review of WIC Food Packages: Improving Balance and Choice: Final Report. Washington, DC: The National Academies Press. doi: 10.17226/23655.
×

This page intentionally left blank.

Suggested Citation:"Appendix K: Study Design Strategies for Reducing the Effects of Selection Bias in Studies Comparing WIC Participants to Others." National Academies of Sciences, Engineering, and Medicine. 2017. Review of WIC Food Packages: Improving Balance and Choice: Final Report. Washington, DC: The National Academies Press. doi: 10.17226/23655.
×
Page 675
Suggested Citation:"Appendix K: Study Design Strategies for Reducing the Effects of Selection Bias in Studies Comparing WIC Participants to Others." National Academies of Sciences, Engineering, and Medicine. 2017. Review of WIC Food Packages: Improving Balance and Choice: Final Report. Washington, DC: The National Academies Press. doi: 10.17226/23655.
×
Page 676
Suggested Citation:"Appendix K: Study Design Strategies for Reducing the Effects of Selection Bias in Studies Comparing WIC Participants to Others." National Academies of Sciences, Engineering, and Medicine. 2017. Review of WIC Food Packages: Improving Balance and Choice: Final Report. Washington, DC: The National Academies Press. doi: 10.17226/23655.
×
Page 677
Suggested Citation:"Appendix K: Study Design Strategies for Reducing the Effects of Selection Bias in Studies Comparing WIC Participants to Others." National Academies of Sciences, Engineering, and Medicine. 2017. Review of WIC Food Packages: Improving Balance and Choice: Final Report. Washington, DC: The National Academies Press. doi: 10.17226/23655.
×
Page 678
Suggested Citation:"Appendix K: Study Design Strategies for Reducing the Effects of Selection Bias in Studies Comparing WIC Participants to Others." National Academies of Sciences, Engineering, and Medicine. 2017. Review of WIC Food Packages: Improving Balance and Choice: Final Report. Washington, DC: The National Academies Press. doi: 10.17226/23655.
×
Page 679
Suggested Citation:"Appendix K: Study Design Strategies for Reducing the Effects of Selection Bias in Studies Comparing WIC Participants to Others." National Academies of Sciences, Engineering, and Medicine. 2017. Review of WIC Food Packages: Improving Balance and Choice: Final Report. Washington, DC: The National Academies Press. doi: 10.17226/23655.
×
Page 680
Next: Appendix L: Gap Analysis »
Review of WIC Food Packages: Improving Balance and Choice: Final Report Get This Book
×
Buy Paperback | $165.00 Buy Ebook | $129.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) began 40 years ago as a pilot program and has since grown to serve over 8 million pregnant women, and mothers of and their infants and young children. Today the program serves more than a quarter of the pregnant women and half of the infants in the United States, at an annual cost of about $6.2 billion. Through its contribution to the nutritional needs of pregnant, breastfeeding, and post-partum women; infants; and children under 5 years of age; this federally supported nutrition assistance program is integral to meeting national nutrition policy goals for a significant portion of the U.S. population.

To assure the continued success of the WIC, Congress mandated that the Food and Nutrition Service of the U.S. Department of Agriculture (USDA) reevaluate the program's food packages every 10 years. In 2014, the USDA asked the Institute of Medicine to undertake this reevaluation to ensure continued alignment with the goals of the Dietary Guidelines for Americans. In this third report, the committee provides its final analyses, recommendations, and the supporting rationale.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!