Click for next page ( 35


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 34
3 Accuracy and Sources of Errors Estimates of the number of people eligible and likely to participate in WIC are useful for both budgetary and evaluative needs only to the extent to which they are reasonably accurate representations of their "true" levels and trends in eligibility and participation. It is not possible to observe ac- tual eligibility for WIC in the population because eligibility is only ob- served when an individual applies for WIC benefits. However, administra- tive records can be utilized to construct a benchmark for examining the accuracy of the estimates of the number of participants. This chapter begins by accessing the accuracy of the USDA method used to predict the number of women, infants, and children who partici- pate in the program. For the overall population, USDA's methodology has led to relatively accurate forecasts of the number of participants. However, for the various subgroups of WIC eligible participants women, infants and children the accuracy of their forecasts of the total number has been poor. The chapter goes on to identify potential sources of error. ACCURACY OF THE USDA METHODOLOGY In preparing a budget request for the WIC program, USDA employs the most current survey data from the March Current Population Survey (CPS). As discussed in the previous chapter, the survey data for a given year could be used to predict eligibility and participation at least four years into the future. For example, when USDA was preparing their budget request 34

OCR for page 34
ACCURACY AND SOURCES OF ERR ORS 35 for 2002, they would have employed data covering calendar year 1998. The accuracy of this forecast of the number of participants in 2002 depends on two factors. The first is how well the methodology predicts the number of participants in the year for which there are data (in this example, 1997~. The second is the validity of the assumption that participation will be un- changed over the forecast period. To assess the level of prediction error in the estimates the error re- sulting from the use of currently available data to predicti~ture eligibility and participation Table 3-1 compares the estimated number of partici- pants by category with actual administrative counts of participants. This ratio was computed using data from 1992-2000. Estimates are first totaled over all categories and then given separately by each eligibility category.] The first line of each category contains the ratio of the estimated num- ber of WIC participants in a given year, t, to the actual number of WIC participants in the same year, t (labeled concurrent year ratio). Examining the first set of ratios, which are totaled over all categories, we see that the ratio of the estimated to actual number of participants ranges from 1.37 in 1992 to 0.86 in 2000. According to this measure of accuracy, the estimates of the number of participants were overstated by 37 percent in 1992 but understated by 14 percent in 2000. Patterns within eligibility categories vary greatly, however. The ratios for infants and breastEeeding postpartum women show a consistent underestimate of the number of WIC partici- pants compared with administrative counts of actual participants. This un- derestimate is getting worse over time. For infants, the ratios range from about 0.80 in 1992 to about 0.61 in 2000. For breastEeeding postpartum women, the ratios range from 0.99 in 1992 to about 0.49 in 2000. Ratios for children show that the estimates initially overestimate the number of participants but are very close to the actual number of participants from 1998 onward. Estimates for pregnant women were overstated in early years but in subsequent years have been slightly understated. For nonbreast- feeding postpartum women, estimates of the number of participants are significantly understated for every year except 1992 and 1993. These ratios do not really reflect the task USDA confronts each year in trying to predict eligibility and participation for the year the proposed budget is to cover. In actuality, estimates of the numbers eligible and likely Administrative records on the number of participants may include ineligible partici- pants. Estimates of eligibility may differ from administrative records because of this, but we do not know how big these differences may be.

OCR for page 34
36 ESTIMATING ELIGIBILI>~D PAR TICIPATION FOR THE ~CPROG~ TABLE 3-1 Accuracy of Current Methods Used to Predict WIC Participants: Ratios of Estimated Participants to Actual Participants, 1992-2000 1992 1993 1994 All Eligibility Categories Concurrent year ratios:a 1.365 1.300 1.111 Forecasted ratios:6 Infants Concurrent year ratios: 0.795 0.790 0.693 Forecasted ratios: Children Concurrent year ratios: 1.756 1.642 1.385 Forecasted ratios: Pregnant Women Concurrent year ratios: 1.270 1.115 Forecasted ratios: NonbreastSeeding Women Concurrent year ratios: 1.351 1.003 Forecasted ratios: BreastSeeding Women Concurrent year ratios: 0.993 0.770 Forecasted ratios: All Postpartum Women Concurrent year ratios: 1.218 0.919 Forecasted ratios: aConcurrent ratios = Estimated Participants in year t / Actual Participants in year t. Forecasted Ratios = Estimated Participants for year (t + 4) / Actual Participants year t. Note: To compute the number of eligible persons who will participate, an 80 percent . . . . participation rate IS assumec . to participate for a given year, t, are used to predict the number of partici- pants forecast for year t + 4 (e.g., data from 1992 are used to predict FY 1996 participants). To take the forecasting component of the estimation process into account, the estimated number of participants in year t are compared with the actual number of participants from administrative records from year t + 4 for each eligibility category and overall categories (labeled forecasted ratio). These ratios are given in the last line of each eligibility category.

OCR for page 34
ACCURACY AND SOURCES OF ERR ORS 37 1995 1996 1997 1998 1999 2000 1.020 0.698 1.197 0.861 0.961 1.026 0.674 0.732 1.117 1.180 1.070 1.164 0.845 0.918 0.664 0.722 0.780 0.848 0.908 1.040 0.630 0.738 1.069 1.205 0.998 1.170 0.796 0.933 0.560 0.656 0.705 0.827 0.865 0.976 0.602 0.657 1.023 1.179 0.953 1.039 0.774 0.844 0.516 0.563 0.672 0.733 0.842 0.962 0.589 0.668 0.996 1.141 0.953 1.082 0.755 0.857 0.485 0.550 0.646 0.733 0.860 0.959 0.609 0.650 1.014 1.167 0.987 1.054 0.781 0.834 0.487 0.520 0.660 0.705 Actual number of participants come from ENS administrative records. Estimated numbers of participants are the USDA estimations given to the panel by USDA. The forecasted ratios assume that estimates of participation are used to predict the actual level of participants four years into the future. Based on the forecasted ratios for total numbers of participants, it ap- pears that the current USDA method of estimating the number of partici- pants for a future budget cycle is quite accurate. The total number of pre- dicted participants matches the number of actual participants quite closely, ranging from a slight overestimate in 1996 (2.6 percent) to a slight under- estimate in 2000 (4.1 percent), indicating very small levels of errors. These smaller error levels are really just a coincidence, because the ratios for the total number of participants mask substantial over- and underestimation

OCR for page 34
38 ESTIMATING ELIGIBILI~YANDPARTICIPATIONFOR THE WICPROGRAM among the different eligibility categories. Numbers of infant participants are substantially underestimated across all years, as are estimates of the num- ber of breastSeeding postpartum women. In contrast, numbers of partici- pating children are overestimated by 14 to 21 percent compared with ad- ministrative records. Numbers of participating pregnant women are overestimated also, but the degree is not so serious in later years. Numbers of nonbreastSeeding postpartum women are consistently underestimated and become more substantially underestimated in later years, so that by 2000, estimates are understated by about 17 percent. Although USDA's current methods to estimate eligibility and partici- pation seem to accurately forecast the overall number of participants, these methods do not accurately forecast specific eligibility categories. Accurately estimating the number of participants in each category is important be- cause food package costs differ across each eligibility category. Furthermore, changes in program rules or program administration could affect eligibility and participation for each category differently. For example, increases in the Medicaid income thresholds in states for infants and children would affect eligibility of those groups but not that of pregnant and postpartum women. SOURCES OF ERROR The true number of eligible persons who are likely to participate is unknown. In making budgetary and programmatic decisions, USDA's goal is to come as close to the true number of eligibles and participants as pos- sible. There are two possible sources of error in making these estimates: (1) errors that cause a systematic bias in the estimated number of persons eli- gible or likely to participate and (2) prediction errors. Errors Causing Systematic Biases Errors may arise because data or methods used to make the estimates are not able to fully capture all the programmatic features or the realities of individuals' economic and family situations, leading to inaccurate estimates of eligibility and participation. For example, the March CPS, the data set used currently to make eligibility estimates, collects annual income instead of monthly income. As Chapter 2 explains, local WIC agencies can use weekly and monthly income, rather than annual income, to determine whether an applicant is eligible for WIC. Variation in monthly income

OCR for page 34
ACCURACY AND SOURCES OF ERR ORS 39 could mean that some people gain eligibility for WIC in some months but have annual incomes that would otherwise make them ineligible for WIC. The following possible sources of systematic error are identified and dis- cussed in this report: The undercount of infants and the overcount of children in the CPS. The use of annual rather than monthly income in estimating eligi- bility for WIC. Not fully accounting for adjunctive eligibility through means-tested programs, particularly Medicaid. The inaccuracy of adjustments to account for nutritional risk among income-eligible persons. The inaccuracy of adjustments to account for breastEeeding status . . among postpartum women. The inaccuracy of adjustments for the percentage of eligible persons who will participate in WIC. Prediction Errors The second source of errors arises because eligibility and participation must be predicted for future years from data that are probably four years old. For example, the following changes could lead to prediction errors: . Demographic changes (e.g., lower birth rates or increased immigra- tion). Changes in family structure (e.g., if the proportion of infants and children living in single-parent rather than two-parent families de- clines, then, since single-parent families are, on average, poorer than two-parent families, the proportion of those income-eligible infants and children should decline). Changes in the income distribution due to changes in wages or unemployment. Changes in program rules, including WIC, but also in other means- tested programs for which adjunctive eligibility is granted. Medic- aid rule changes in recent years are a primary example. Changes in WIC program administrative practices. Changes in participation rates of other means-tested programs. The length of time between the year that data are available for esti-

OCR for page 34
40 ESTIMATING ELIGIBILI>~D PAR TICIPATION FOR THE ~CPROGR~ mation and the year for which the estimate is being made. All else equal, a longer time period should lead to larger errors. These changes could, of course, have opposite effects on estimates of eligi- bility and participation. For example, the increase in Medicaid income eli- gibility thresholds would tend to increase the number of eligible persons. However, the economic expansion of the late 1990s should have led to a decrease in the number of persons income eligible for WIC and for Medic- aid. It is difficult to infer the extent to which these changes offset each other or not. In considering current and alternative approaches to estimating eligi- bility and participation for WIC, the panel has attempted to address these r two sources or errors. EVALUATING CURRENT AND ALTERNATIVE METHODS FOR E S T I M A T I N G E L I G I B I L I T Y A N D P A R T I C I P A T I O N In this report we evaluate methods currently used to estimate eligibil- ity and participation for WIC and examine alternatives to current meth- ods. We outline different methodological options and include recommen- dations for new approaches to the various components of the estimation methodology. The panel considered several factors for evaluating alterna- tives: The accuracy of the estimates. The feasibility of implementing an approach (e.g., the expense and burden of implementing an approach). The quality, availability, and timeliness of data used by an approach. Correspondence of the method with current WIC rules and their application at the local level. For example, adjunctive eligibility is part of the rules of the program, and methods for estimating eligi- bility should account for it. For some components of the estimation process, it is not clear how to assess the accuracy of an approach. For example, the minor adjustment currently made to account for adjunctive eligibility appears to be inad- equate (National Research Council, 20011. However, data limitations cre- ate problems in determining the true number of people who gain eligibility for WIC only through adjunctive eligibility. Estimates based on survey data

OCR for page 34
ACCURACY AND SOURCES OF ERR ORS 41 are limited because respondents underreport Medicaid and other program participation (although to what degree is not known). Administrative data on Medicaid enrollees do not identify the age of children nor the income of enrollees; therefore, no administrative data on the number of people who would gain eligibility for WIC solely through adjunctive eligibility are avail- able. The panel also based its assessments of methodologies on the premise that the methodology should reflect current rules and practices of the pro- gram. Finally, the panel recognizes that the USDA estimates serve different purposes and that different methods of estimation may be appropriate for different purposes. Evaluation of the effectiveness of the WIC program requires estimates of eligibility and participation. Because eligibility is not directly observed, estimates must be based on information reported from survey data. To make budgetary forecasts, reasonably accurate estimates might be forecasted based on administrative records from past years. Chap- ters 4-7 focus on the estimation of numbers of people eligible for WIC. Chapter 8 focuses on estimates of participation among eligible people and methods for estimating participation among them. In Chapter 9, we syn- thesize all our findings and provide different options for estimating eligibil- . . . . lty ant ~ participation. SUMMARY This chapter began with an assessment of the accuracy of the USDA estimates to predict the number of WIC participants. Using available data to examine the accuracy of participation estimates, the panel found that the total number of participants matches the reported number quite closely. However, the numbers of participating infants and breastEeeding postpar- tum women are seriously underestimated, while the numbers of participat- ing children and pregnant women are overestimated. The chapter also ex- plains the two types of errors the panel investigated in its review of the current methodology errors that cause systematic bias and prediction er- rors. Finally, in evaluating estimation methods, the panel considered accu- racy, feasibility, characteristics of data sources, and correspondence of the method with current rules and their application at the local level.