Appendix G

Causes of Systematic Differences Between American Community Survey (ACS) and Administrative Estimates

Chapter 4 describes the major causes of systematic differences between ACS estimates for percentages eligible for free, reduced-price, and full-price school meals and the data from administrative sources. This appendix provides additional background information about some of those causes, including

  • underreporting of Supplemental Nutrition Assistance Program (SNAP, formerly Food Stamp Program) benefits;
  • determining eligibility using annual rather than monthly income;
  • school choice opportunities;
  • imputation for nonresponse; and
  • certification errors.

Each of these causes is discussed in turn below.

UNDERREPORTING OF SNAP BENEFITS

As discussed in Chapter 4 and Appendix B, considerable research through the years has documented underreporting of benefits such as SNAP in household surveys. The panel conducted its own evaluation by comparing ACS estimates of SNAP reporting by households with school aged children to estimates from administrative data. The panel received a file from Mathematica Policy Research comparing counts and eligibility percentages for 2009 from two different data sources: the 2009 ACS



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 341
Appendix G Causes of Systematic Differences Between American Community Survey (ACS) and Administrative Estimates C hapter 4 describes the major causes of systematic differences between ACS estimates for percentages eligible for free, reduced- price, and full-price school meals and the data from administra- tive sources. This appendix provides additional background information about some of those causes, including underreporting of Supplemental Nutrition Assistance Program (SNAP, formerly Food Stamp Program) benefits; determining eligibility using annual rather than monthly income; school choice opportunities; imputation for nonresponse; and certification errors. Each of these causes is discussed in turn below. UNDERREPORTING OF SNAP BENEFITS As discussed in Chapter 4 and Appendix B, considerable research through the years has documented underreporting of benefits such as SNAP in household surveys. The panel conducted its own evaluation by comparing ACS estimates of SNAP reporting by households with school aged children to estimates from administrative data. The panel received a file from Mathematica Policy Research comparing counts and eligibil- ity percentages for 2009 from two different data sources: the 2009 ACS 341

OCR for page 341
342 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS Public Use Microdata Sample (PUMS) files and the 2009 SNAP Quality Control file (SNAP QC). The data support an investigation of the potential undercount of SNAP participation in student households by the ACS at the national and state levels. The SNAP QC data are sample-based administrative data that are representative at the state level and contain detailed demographic, eco- nomic, and SNAP eligibility information for an annual sample of more than 45,000 SNAP households. The data are weighted to match adminis- trative counts of individuals and households receiving benefits and the amount of benefits received (adjusted to remove ineligible households that received benefits in error and those receiving disaster assistance benefits). The SNAP QC data represent all SNAP participants regardless of where they live (so noninstitutionalized group quarter residents are included).1 The SNAP QC data do not include all individuals in households where someone receives SNAP benefits. The data include individuals in the SNAP filing unit (those covered by SNAP), and only those individu- als outside the filing unit (but in the household) whose income or assets would be counted in determining eligibility and benefits. The tables below include individuals in the filing unit as well as any other individu- als in the household that are included in the SNAP QC data. There are about 1.85 children per SNAP household in the SNAP QC data. In the ACS, SNAP participation is a household question, but it is also asked of group quarter respondents. We counted a household as having SNAP benefits if the household question was answered in the affirmative. We counted everyone living in that household as receiving SNAP benefits. According to the ACS, there were 1.89 children in each SNAP household in 2009, compared with 1.85 children per SNAP household in the SNAP QC data. Differences in household sizes across the two data sets are dis- cussed below. On the ACS, the group quarter respondents who reported SNAP participation were split approximately evenly between institu- tional and noninstitutional group quarters; only those in noninstitutional group quarters are included in the tables below. There is an additional difference in the way eligibility is determined in the two data sets. In SNAP QC, eligibility is based on income and fil- ing unit as reported on an application (and determined to be accurate). The SNAP QC file has monthly income,2 and eligibility is based on a 1There is no way to identify group quarter individuals in the SNAP QC data. 2The panel's Food and Nutrition Service (FNS) contacts told us that applications for school meals generally report monthly or more frequent income (e.g., weekly or biweekly). The same is likely to be true of SNAP applications. It is more convenient to recode income to a common monthly value in a data set such as SNAP QC.

OCR for page 341
APPENDIX G 343 comparison with the income eligibility guidelines. The SNAP QC-based eligibility is from applications made in fiscal year 2009, so the data reflect the participant's situation in that year. For the ACS data, eligibility is based on "povpip."3 Povpip is based on annual income as reported on a survey form completed during 2009, adjusted to represent income data in 2009 and compared with the 2009 poverty guidelines. A survey completed in January 2009 would reflect data on income received mainly during 2008 (representing income from the same day in January 2008 through the survey day in January 2009). 4 A survey completed in December 2009 would reflect income from the survey date in December 2008 through the survey date in December 2009. Thus there is about a half-year lag in the ACS income data relative to the SNAP QC income data.5 Additionally, as discussed later in this appendix, if monthly income is variable, using annual income smooths over periods of high and low income and may understate income eligibility for the school meals programs. Similarly, the ACS question on SNAP participation asks whether any- one in the household received food stamp benefits during the last 12 months. Individuals in a household that received SNAP benefits in 2008 could still have been eligible in 2009. However, it is also possible that their situation changed and that they were no longer eligible in 2009. Under the school meals programs, if a household is determined to be eligible for school meals because of SNAP participation or income, a student in that household remains eligible for school meals for the rest of the school year and for 1 month into the next. The tables provided to the panel compared ACS and SNAP QC esti- mates of number of households with SNAP benefits, number of house- holds with SNAP benefits with children aged 5-17, number of individuals with SNAP benefits, and number of individuals aged 5-17 with SNAP benefits. 3Povpip is the ratio of income to the poverty threshold computed by the Census Bureau and made available in its data products. For family members, it is the ratio of family in- come to the appropriate poverty threshold. For unrelated individuals, it is personal income compared with the one-person poverty threshold. It is not defined for unrelated individuals under age 15 because income data are not collected for these individuals. It is not defined for some GQ individuals. If povpip is not defined, the person is classified as eligible for free meals. 4Instructions state that the respondent is to report his or her income during the last 12 months and explains that this means "from today's date one year ago through today." However, it would be surprising if people know their income by such specific time periods. 5ACS income are adjusted using the Consumer Price Index to reflect calendar year 2009 dollars.

OCR for page 341
344 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS Comparison of ACS and SNAP QC Table G-1 shows national-level counts of households (all and those with children aged 5-17) and individuals (all and those aged 5-17) receiv- ing SNAP benefits in 2009, based on the ACS versus SNAP QC. The table presents the difference between the estimates from the two data sources, the difference expressed as a percentage of the ACS count, the standard error of the difference, and a z-statistic for testing whether the differ- ence is statistically significant. The hypothesis that the ACS and SNAP QC estimates are the same is rejected at the 5 percent significance level if z is greater than 1.96 in absolute value. All differences are statistically significant. The ACS overstates individuals receiving SNAP benefits,6 while it understates households, households with children aged 5-17, and individuals aged 5-17 receiving SNAP benefits. Table G-2 shows counts by state for our population of interest: chil- dren aged 5-17 in households receiving SNAP benefits in 2009. At the national level, the difference between the ACS and SNAP QC counts is statistically significant at the 5 percent level. ACS undercounts this population by 4.4 percent. For California, Delaware, New Mexico, and Tennessee, the difference between the ACS and SNAP QC estimates is significant at the .001 level, indicating undercounts by the ACS of 14.5 percent, 32.8 percent, 24.7 percent, and 14.6 percent, respectively. 7 In other states the differences are not statistically significant. These results demonstrate the variability among states in the tendency to underreport SNAP benefits. DETERMINING ELIGIBILITY USING ANNUAL VERSUS MONTHLY INCOME This section addresses the potential differences in eligibility per- centages due to computing eligibility for school meals based on annual income, the only option available for the ACS, and computing eligibility based on monthly income, as is done in the school meals programs. The panel based its evaluation on the 2004 SIPP, a national panel survey that collects monthly income data. 6The overstatement of individuals on SNAP by ACS may be due to the fact that receipt of SNAP is a household question and all members of the household are assumed to be on SNAP. 7The .001 significance level for each state-level test assures that the chance of rejecting the hypothesis of no difference when 51 state-level tests are conducted simultaneously has an overall significance level of .05. The Sidak multiple comparison correction selects alpha per comparison = 1 (1-alpha*)^(1/n), where alpha* is the desired overall significance level and n is the number of comparisons. If alpha* = .05, then .95^(1/51) = .999, so alpha per comparison should be .001; the critical value for a z-statistic with alpha = .001 is 3.

OCR for page 341
APPENDIX G 345 TABLE G-1 Comparison of Counts of Households and Individuals Receiving SNAP Benefits at the National Level, ACS Versus SNAP QC, 2009 ACS Diff. SNAP as ACS SNAP QC QC % of SE (thousands) (thousands) (Diff.) ACS Diff. z Diff. Households 11,718 14,981 3,263 27.8% 37 88.2 Households with 5,279 5,658 379 7.2% 48 8.0 Children Aged 5-17 Individuals 39,590 35,073 4,517 11.4% 190 23.8 Individuals Aged 5-17 10,041 10,486 446 4.4% 95 4.7 NOTES: ACS = American Community Survey; SE = standard error; SNAP = Supplemental Nutrition Assistance Program (formerly Food Stamp Program); SNAP QC = SNAP Quality Control Data File. SOURCE: Prepared by the panel using 2009 ACS and FY 2009 SNAP QC estimates provided by Mathematica Policy Research, September 2011. Survey of Income and Program Participation (SIPP)8 SIPP is a continuing program of the Census Bureau, which began inter- viewing for the survey in late 1983 and is planning to introduce a major redesign in 2013. Under the survey's current design, members of sampled households (panels) are interviewed every 4 months for 3 or 4 years. Hence, SIPP not only provides detailed annual and monthly information on income by source for a representative sample of U.S. households but also tracks changes in program eligibility and participation for the house- hold members as their incomes and other circumstances change. SIPP asks about participation of household members in SNAP, the National School Lunch Program (NSLP), the School Breakfast Program (SBP), Tem- porary Assistance for Needy Families (TANF), and other programs for low-income persons. In addition, it collects data on taxes, assets, liabilities, labor force participation, general demographic characteristics, and many special topics related to families' economic circumstances. The survey design is a series of national panels, each representing the U.S. civilian noninstitutionalized population. Over the years, panels have varied in sample size, number of interview waves, and other features. For the 1984-1993 period, a new panel of households was introduced each February. Subsequent panels have not overlapped; they include a 4-year panel beginning in 1996, a 3-year panel beginning in 2001, a 8This section draws heavily on the discussion of SIPP in Chapter 3 of the panel's interim report (National Research Council, 2010).

OCR for page 341
346 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS TABLE G-2 State-Level Counts of School-Age Children (aged 5-17) in Households Receiving SNAP Benefits, 2009 Total Individuals ACS- SNAP QC Difference ACS SNAP QC (Difference) as % of State (thousands) (thousands) (thousands) ACS z Total 10,041 10,486 446 4.4 4.7 Alabama 204 227 23 11.1 2.5 Alaska 20 20 1 3.1 0.3 Arizona 277 276 2 0.6 0.1 Arkansas 121 119 2 1.7 0.4 California 1,011 1,160 148 14.6 4.3 Colorado 111 107 5 4.1 0.7 Connecticut 71 67 4 5.0 0.8 Delaware 23 31 8 32.8 3.1 District of Columbia 26 27 0 0.7 0.1 Florida 540 550 11 2.0 0.5 Georgia 356 405 48 13.6 3.0 Hawaii 26 30 4 14.9 1.5 Idaho 47 46 1 2.4 0.3 Illinois 435 455 19 4.4 1.1 Indiana 214 210 4 1.9 0.4 Iowa 83 85 2 2.2 0.3 Kansas 73 64 9 12.1 1.6 Kentucky 190 204 14 7.2 1.5 Louisiana 234 225 9 3.8 1.0 Maine 47 52 5 9.5 0.9 Maryland 129 135 6 4.6 0.9 Massachusetts 160 170 10 6.5 1.4 Michigan 440 394 46 10.4 2.7 Minnesota 100 99 0 0.5 0.1 Mississippi 150 159 9 6.2 1.4 Missouri 211 229 17 8.0 1.7 Montana 31 25 6 19.1 1.6 Nebraska 48 42 5 11.2 1.5 Nevada 62 60 2 3.0 0.5 New Hampshire 22 20 2 8.5 0.6 New Jersey 142 152 9 6.6 1.2 New Mexico 82 103 20 24.7 3.7 New York 617 637 20 3.3 0.7 North Carolina 328 364 36 11.0 2.4 North Dakota 10 13 3 33.0 1.8 Ohio 420 387 32 7.7 2.2 Oklahoma 136 142 6 4.4 0.8 Oregon 162 151 10 6.2 1.1 Pennsylvania 372 384 12 3.2 0.8 Rhode Island 31 29 2 6.0 0.8

OCR for page 341
APPENDIX G 347 TABLE G-2Continued Total Individuals ACS- SNAP QC Difference ACS SNAP QC (Difference) as % of State (thousands) (thousands) (thousands) ACS z South Carolina 180 201 21 11.5 2.5 South Dakota 25 22 3 11.0 0.9 Tennessee 276 317 40 14.6 3.2 Texas 1,071 1,129 58 5.4 1.8 Utah 66 63 3 3.9 0.5 Vermont 14 18 4 29.4 1.9 Virginia 179 188 8 4.6 0.9 Washington 219 215 5 2.1 0.4 West Virginia 70 81 12 16.7 2.4 Wisconsin 171 174 3 1.8 0.4 Wyoming 7 7 0 3.6 0.2 NOTES: The ACS-based estimates are of all individuals (aged 5-17) living in households report- ing receipt of SNAP benefits and include those living in noninstitutional group quarters. Estimates use person-level weights. The SNAP QC-based administrative estimates are of all individuals (aged 5-17) who are members of SNAP filing units. SNAP filing units refer to individuals who together are certified for and receive SNAP benefits. The estimates of individuals also include those who were living with SNAP participants but who were not receiving SNAP benefits if their income and assets were considered in determining the SNAP filing unit's eligibility and benefits. The ACS-based poverty levels are based on the povpip variable, which measures the pov- erty status of the family relative to the census poverty thresholds. The SNAP QC-based poverty levels are based on the tpov variable, which measures the poverty status of the SNAP unit relative to the SNAP poverty guidelines. The standard error (SE) for SNAP QC in Wyoming was noted only as less than 500. It was entered at .25 to support computation of the z-statistic. The z-statistic is the ACS estimate minus SNAP QC estimate divided by the SE of the difference. A test of the hypothesis that the difference between the ACS and SNAP QC estimates is zero is rejected if z is greater than 3 in absolute value. For any individual state, this is at the .1 percent significance level. For testing of all 52 states at the same time, it is at the 5 percent significance level. No persons in institutional group quarters are represented in the table. ACS = American Community Survey; SE = standard error; SNAP = Supplemental Nutri- tion Assistance Program (formerly Food Stamp Program); SNAP QC = SNAP Quality Control Data File. SOURCES: Prepared by the panel using 2009 ACS and FY 2009 SNAP QC estimates provided by Mathematica Policy Research, September 2011.

OCR for page 341
348 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS 4-year panel beginning in 2004, and a 4-year panel beginning in 2008. A new, redesigned panel of about the same size as the 2008 panel--45,000 households--is to be introduced in 2013 and followed for 3 or 4 years. The current SIPP content is built around a "core" of labor force, program participation, and income questions that are repeated at each wave of interviewing, with supplemental topical modules on particular topics being asked one or more times per panel. The survey collects data for each month of a 4-month recall period, with approximately the same number of interviews being conducted in each month of the 4-month period for each wave. Interviews are conducted by personal visit for the first two interview waves and by telephone thereafter, using a computer- assisted interview on a laptop computer. As discussed in Bates and Okon (2003), the 2004 SIPP panel instituted a variety of enhancements to better capture income reporting, including dependent interviewing techniques. The new methods allowed respondent-selected defined periods for report- ing job earnings: monthly, biweekly/bimonthly, annually, or hourly. The goal was to make retrieval and reporting more natural and consistent with how respondents typically think about their earnings. In cases where an amount other than monthly was selected, the computer program for the survey internally calculated a gross monthly amount based on pay dates, pay periods, hours worked, paycheck totals, and so on, and performed a variety of checks based on comparisons with answers to past questions, asking respondent to confirm estimated values when there appeared to be potential errors. There are many probes to make sure the respon- dent has reported all relevant income items for each month. Further the income questions are asked after dates of employment are established, and income then is reported for each spell of employment. As noted by Moore (2007), "panel surveys generally suffer to some extent from seam bias, the tendency for estimates of change measured across the `seam' between two successive survey administrations to far exceed change estimates measured within a single interview." The changes in survey methodology that were implemented in the 2004 SIPP were intended to reduce seam bias. Moore compared indications of seam bias in SIPP 2001 with those from the first waves of SIPP 2004 to evaluate the impact of changes to survey methodology in 2004. He reported sub- stantial reductions in seam bias from 2001 to 2004 that are attributable to the new survey methods. 9 "However, notwithstanding the clear improve- ments, seam bias still afflicts SIPP 2004 panel data. . . ." 9Seam bias generally refers to how a respondent reports a change in status. For example, a respondent who becomes unemployed during a 4-month period is more likely to report that event as occurring during the month of the interview than in the month he became unemployed. Moore (2007) evaluates variables related to change in status. A similar mecha-

OCR for page 341
APPENDIX G 349 Data are currently released in cross-sectional core and topical mod- ule files for each interview wave. As of mid-2011, core files were avail- able through wave 2 of the 2008 panel; topical module files were available through wave 8 of the 2004 panel. 10 The planned redesign of SIPP will change the interviewing cycle from every 4 months to once a year. Each annual interview will include the core question content on income, employment, program participation, and demographic characteristics using an event history calendar to facilitate recollection of monthly information for the previous year. Some content previously in topical modules will be included, and government agencies may pay for special supplements. Deriving Estimates from the 2004 SIPP Data The panel calculated percentage eligible for free and for reduced-price meals using the economic units and guidelines described in Appendix B that mirror the special tabulations the panel requested from the Census Bureau. However, in addition to computations based on annual income, we derived estimates based on monthly income under the assumption that eligibility status lasts for a school year. Both estimates were computed with and without accounting for categorical eligibility because of SNAP, TANF, and foster children. Preparing the Data The following steps were followed in preparing the SIPP database: 1. Merge people across all waves in the 2004 SIPP. SIPP includes monthly income data from October 2003 through December 2007, although not all data are based on four rotation groups and the full original sample size. (The sample size was reduced by 50 percent beginning with wave 9 in October 2006.11) Keep indi- nism may result in a respondent reporting an average value or the value for the most recent month for all 4 months of a wave rather than the exact values for each individual month. This type of misreporting should also have been reduced as a result of the methodology enhancements implemented in the 2004 SIPP. The panel, however, is not aware of empirical evidence of this. Pischke (1995) modeled the measurement error in monthly income data in the 1984 SIPP and found that changes in income tended to be reported in the month of the interview. 10See http://www.sipp.census.gov/sipp_ftp.html#sipp. 11The 2004 SIPP panel underwent a 50 percent sample reduction in wave 9. This occurred during the last wave of interviews in 2006, beginning in October. The first interviews with the smaller sample size collected income information for June through September 2006. Hence, only the data from January 2004 through May 2006 are based on a full set of rotation

OCR for page 341
350 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS viduals in the database if there are data for them for all months of 2004-2005 and most of 2006, so that past year's annual income would be available for each month of 2005 and most of 2006 to match the time pattern of income reported in the ACS for a given calendar year. 2. Keep only households with children aged 5 and up to 15, or aged 15 to 19 (inclusive) enrolled in school but not graduated from high school.12 3. Identify foster children and keep them separate from the house- hold in which they reside. They are categorically eligible for free school meals and will be added back into the tabulations later. For each household, create counts of the number of persons in the household (excluding any foster children) and household income (excluding foster child income). 4. Construct economic unit measures EU1-EU5 (see Appendix B) for each household, counting number of persons and summing total personal income for the relevant units. Note that foster children are excluded from these economic unit definitions (and are added in as being categorically eligible for free meals at a later step). 5. Identify the school year associated with the month of the data (for example, income data representing July 2004 through June 2005 would be associated with school year 2004-2005). Assume the guidelines change in July as is typical. 6. For the monthly income measure, use SIPP income reported for that month. Use the guideline associated with the relevant school year. Compute the ratio of monthly income to the guidelines. Also track eligibility throughout the school year, assuming the school year spans September through June and treating July and August as part of the previous school year. Thus children who are income- eligible for free meals during any month of the school year will be eligible for free meals for all subsequent months of the school year. More generally, eligibility established in any month in a school year ensures continued eligibility for the remainder of the school year at that level even if income increases. Note that since the ACS samples are independent across months of the year, one can obtain the right total number of children for a calendar year groups. For June 2006, the data are based on three full rotation groups and one reduced-size group. This covers almost two school years: 2004/2005 and 2005/2006, if a school year runs from July of one year to June of the following year. Note that the data set includes these partial panel participants, but the tabulations include information only for those in the data for the relevant calendar year and the preceding calendar year. 12In the SIPP, we do not know whether children are in private or public school and know enrollment only for those aged 15-19.

OCR for page 341
APPENDIX G 351 only by including the children from July and August. Keep cur- rent monthly eligibility, as well as cumulative monthly eligibility. 7. For the annual income measures, create the previous 12 months' income as the sum of income over those months, and also com- pute an "inflation"-adjusted income for the previous 12 months to mimic the fact that the ACS adjusts income for inflation to reflect real dollars as of July of the relevant calendar year. (Inflation adjustment factors come from the ACS subject definitions.)13,14 8. For each month, for each child in the sample, create indicators for the ratio of economic unit income to the guidelines to reflect eligibility for free, reduced-price, or full-price meals using (1) the ACS adjusted annual income and school year guidelines for second half of calendar year and (2) SIPP monthly income and school year guidelines. Recall that this is income eligibility only, excluding foster children and others categorically eligible but not income-eligible. 9. Construct indicators for uptake of free, reduced-price, or full- price school lunch based on the SIPP question about "usually" getting a lunch. The question is asked only of children aged 5-18. If the respondent says that some children usually get a lunch, then he or she is asked whether the children qualify for free or reduced-price meals under the NSLP. 10. Construct indicators for whether someone in the household has received SNAP benefits or public assistance (presumably mainly TANF) this month, cumulatively over the school year, and in the last calendar year. Create a separate version of the cumulative SIPP and adjusted and unadjusted ACS measures that also accounts for categorical eligibility, adding as eligible for free meals foster chil- dren or children in households with SNAP or TANF. 11. Tabulation results use longitudinal weights through 2006, and standard errors and confidence intervals use Taylor series approx- imations with the Primary Sampling Unit and strata information in the public use files. 13Inflation adjustment factors are the average for the previous calendar year (thus for July 2004, they are the average for July 2003-June 2004). For 2004, they range from 1.90615 to 1.95206. They then need to be translated to dollars for the relevant calendar year. Since the factors are used to inflate 1982 dollars to the current year, they must be multiplied by the average Consumer Price Index Research Series Using Current Methods (CPI-U-RS) for the relevant calendar year and divided by the Current Population Survey Research Series Using Current Methods (CPSU-RS) for 1982 to yield current dollars. For years 2004-2006, they range from close to 1 (in 2006) to around 1.04 (in 2004). 14For 2004, for example, they appear at the following link: www.census.gov/acs/www/ Downloads/data_documentation/SubjectDefinitions/2004_ACSSubjectDefinitions.pdf.

OCR for page 341
APPENDIX G 363 TABLE G-5 Share of Public School Enrollment by Choice Status Independent School Year Regular District Charter LEA Charter Magnet 2004-2005 0.933 0.017 0.024 0.026 2005-2006 0.927 0.018 0.025 0.030 2006-2007 0.928 0.020 0.026 0.026 2007-2008 0.923 0.021 0.027 0.030 2008-2009 0.915 0.022 0.030 0.033 NOTES: LEA = local education agency. Regular schools may include open enrollment schools. District charters are charters under the administration of the local LEA. Indepen- dent charters are separate LEAs, not part of the local LEA. SOURCE: Prepared by the panel. trict charters and the latter as independent LEA charters. As noted above, independent charters are potentially more problematic for use of the AEO as they represent a form of interdistrict choice and thus can affect both district-level and school-level decisions to opt for the AEO. The panel was unable to find national data documenting the prevalence of open enroll- ment schools and thus cannot comment on its potential impact. Although potentially problematic when it occurs, school choice cur- rently raises limited concerns, on average, regarding the use of the ACS for estimating eligibility for free and reduced-price school meals. In 2008- 2009, fewer than 15 percent of counties in the United States contained either a charter or magnet school (panel database). However, because charter and magnet schools are much more prevalent in urban areas, they accounted for about 9 percent of all enrollment. Thus, although charters and magnets are not common in most areas, they can enroll a large num- ber of students in some places. For example, charter or magnet school enrollment accounted for more than 10 percent of public school enroll- ment in 92 counties in 2008-2009 (panel database). Charter schools that are independent LEAs accounted for more than 20 percent of enrollment in just 9 counties in 2008-2009, including Washington, DC (35 percent), St. Louis (25 percent), and New Orleans (55 percent) (panel database). Thus for a very limited set of districts, the ACS may provide misleading estimates of eligibility for free or reduced-price meals. Summary The above analysis suggests the following: School choice is not sufficiently pervasive to cause concern for use of the ACS to estimate free or reduced-price eligibility for the AEO in most schools and school districts.

OCR for page 341
364 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS In an important subset of schools and districts, however, atten- dance at noncatchment area schools occurs frequently enough that these districts should carefully consider the likely difference between the ACS free or reduced-price-eligibility estimates and estimates based on actual enrollment. -- At the district level, this occurs when a substantial portion of students have exercised the ability to choose schools that are not part of the LEA, such as charter schools in independent LEAs. -- At the school level, this occurs when a relatively large per- centage of students have chosen to attend noncatchment area schools. IMPUTATION FOR ITEM NONRESPONSE Using the 2008 ACS Public Use Microdata Sample (PUMS) file, the panel developed the following tabulations of income eligibility; report- ing of SNAP benefits and public assistance income; and imputation flags for (1) any income item, (2) SNAP, and (3) public assistance income. Tabulations included income eligibility for the school meals programs for all related and unrelated students and excluded foster children. Income eligibility used household income and household size. There were seven tabulations: 1. income eligibility for all students; 2. income eligibility for all students in households where some income item was imputed; 3. income eligibility for all students in households that were receiv- ing SNAP benefits; 4. income eligibility for all students in households that were receiv- ing SNAP benefits and for which SNAP was imputed; 5. income eligibility for all students in households that were receiv- ing SNAP benefits and for which income was imputed; 6. income eligibility for all students in households where some resi- dent reported public assistance income; and 7. income eligibility for all students in households where some resi- dent reported public assistance income, and welfare income was imputed for some resident. Table G-6 shows results for the United States. In the United States, 28.8 percent of households with students had some income imputed, .2 percent had SNAP benefits imputed, and 1.0 percent had public assis- tance income imputed. Of the households receiving SNAP benefits,

OCR for page 341
APPENDIX G 365 TABLE G-6 Eligibility Distribution for Households with Students, Selected Characteristics Eligibility Percentage Percentage Percentage Reduced Percentage of All Household Characteristic Free Price Full Price Students With Students 22.51 11.70 65.79 With Students, Some Income 22.80 13.28 63.92 28.80 Imputed With Students and SNAP 68.81 13.72 17.48 17.39 With Students, SNAP, and 65.00 10.67 24.33 0.20 SNAP Imputed With Students, SNAP, and 54.77 17.34 27.89 5.87 Some Income Imputed With Students and Public 65.24 13.05 21.71 4.92 Assistance With Students, Public 54.44 15.55 30.01 0.99 Assistance, and Public Assistance Imputed SOURCE: Prepared by the panel using 2008 ACS Public Use Microdata Sample (PUMS) data. almost 6 percent had SNAP benefits imputed, and of the households that had someone receiving public assistance income, 20 percent had someone with public assistance income imputed. Note that in the households receiving SNAP benefits or public assis- tance income, most students (68.8 percent and 65.2 percent, respectively) were already income-eligible for school meals. Previous tabulations show that accounting for SNAP (but not public assistance) increases the per- centage eligible for free meals by 5.4 percent, accounting for public assis- tance (but not SNAP) increases the percentage eligible for free meals by 1.7 percent, and accounting for both increases the percentage eligible for free meals by 6.1 percent. Comparing rows in Table G-6 shows the impact of imputation on the eligibility percentages for the school meals programs. For example, the eligibility percentages in the third and fourth rows show the impact of imputation of SNAP benefits. Imputation of SNAP benefits (fourth row) tends to increase the percentage eligible for full-price meals while decreasing the percentages eligible for free and for reduced-price meals. However, this will have a minor impact on the eligibility distribution for all students because SNAP is imputed for only .2 percent of them. Comparing the eligibility percentages in the sixth and seventh rows shows the impact of imputation of public assistance income

OCR for page 341
366 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS among households that report receiving such income. The imputation of public assistance income tends to overstate eligibility for reduced- and full-price meals and understate eligibility for free meals. Since only 1 per- cent of all students live in households where public assistance income is imputed, however, this will have little impact on the overall eligibility distribution. Finally, a comparison of the third and fifth rows shows that for SNAP households with students, income imputation (for any source of income) tends to overstate the full-price- and reduced-price-eligibility percentages and understate the free eligibility percentage. Since roughly one-third of households that report receiving SNAP benefits have some income imputed, this could be a more significant issue. However, because the panel has chosen to use the ACS variables on SNAP benefits and public assistance income to determine categorical eligibility, the children mis classified by income imputation will be correctly assigned as eligible for free meals because of SNAP participation. CERTIFICATION ERRORS As described in Chapter 2, the Access, Participation, Eligibility, and Certification Study (APEC) (U.S. Department of Agriculture/Food and Nutrition Service, 2007b) provided national estimates for the percent- age of students who were misclassified by eligibility category in 2005- 2006. These certification errors are reproduced in Table G-7. The first three values, for example, indicate that among students certified for free meals, 86.0 percent were actually eligible for free meals, 8.1 percent were actu- TABLE G-7 Certification Category and Correct Eligibility Category in School Year 2005-2006 As a Percentage of Certification Category Correct Eligibility Category Certification Category Free Free 86.0 Free Reduced price 8.1 Free Full price 5.9 Reduced Price Free 34.0 Reduced Price Reduced price 40.9 Reduced Price Full price 25.1 Full Price Free 19.0 Full Price Reduced price 16.6 Full Price Full price 64.4 SOURCE: U.S. Department of Agriculture/Food and Nutrition Service (2007b) also called the APEC study.

OCR for page 341
APPENDIX G 367 ally eligible for reduced-price meals, and 5.9 percent were eligible only for full-price meals. The APEC certification errors apply to all certified students (including those directly certified) and denied applicants; they do not apply to students whose families did not apply for benefits. While it is likely that most of these students were not eligible for free or reduced- price meals, some may have been, and there is no current information about the percentage of eligible students who do not apply. Accordingly, the panel considered a range of assumptions to help illuminate the poten- tial impact of these errors on differences between ACS eligibility estimates and administrative data on certification. The panel used the APEC certification errors (reproduced in Table G-7) to evaluate the potential impact of certification errors on eligibility estimates for a variety of assumptions. These are illustrated in Tables G-8 through G-12. Each table shows the impact of certification errors for 13 free, reduced-price, and full-price certification distribu- tions. In forming these distributions, the percentage certified as eligible for free meals was varied from 45 percent to 90 percent in increments of 5 percent, and the percentage certified for reduced-price meals assumed values of 5 percent, 10 percent, or 15 percent. As a result, the percent- age full-price eligible ranged from 40 percent to 5 percent. In addition, three assumptions are displayed for the percentage of enrolled students who applied for benefits and were denied: 10 percent (Tables G-8 and G-11), 25 percent (Tables G-9 and G-12), and 40 percent (Table G-10). Finally, two different assumptions were made concerning the eligibility status of students who did not apply: either they were all assumed to be eligible only for full-price meals (Tables G-8 through G-10), or 9.5, 8.3, and 82.2 percent were assumed to be eligible for free, reduced-price, and full-price meals, respectively (Tables G-11 and G-12). Results were evaluated on both the eligibility percentages and the blended reimbursement rate (BRR) implied by the eligibility percentages. Table G-8 shows that if the percentage of enrolled students who applied for benefits and were denied is 10 percent, and all who did not apply were eligible only for full-price meals, then certification errors result in an overstatement of the BRR by 6-7 percent across all 13 certification distribu- tions. For the highest-percentage free- and reduced-price-eligible districts shown in the table, the overstatement of the BRR remains at 6-7 percent as the percentage of enrolled students who applied and were denied increases to 25 percent (see Table G-9) or 40 percent (see Table G-10). Under these assumptions, however, for districts with low percentages free and reduced-price eligible, the overstatement of the BRR is reduced to 3 percent under the 25 percent assumption (see Table G-9) and to 0 percent under the 40 percent assumption (see Table G-10). As shown in Tables G-11 and G-12, the assumption that some of the

OCR for page 341
TABLE G-8 Illustration of Impact on BRR of Two Assumptions: (1) 10 Percent of Students Who Must Pay Full Price Applied for but Were Denied Approval for Free or Reduced-Price Meals and (2) Students Who Did Not Apply Were Not Eligible for Free or Reduced-Price Meals (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) Alternative Distributions of Certified Students (%) Free 45 50 55 60 60 65 70 70 75 80 80 85 90 Reduced Price 15 10 15 10 15 10 5 15 10 5 15 10 5 Full Price 40 40 30 30 25 25 25 15 15 15 5 5 5 Distributions of Eligible Students, Corrected for Certification Error (%) Free 45 47 53 56 57 60 62 66 68 71 74 77 79 Reduced Price 10 9 11 9 11 10 8 12 10 9 13 11 9 Full Price 45 44 36 35 31 30 29 22 21 20 13 12 11 BRRs BRR, Certified Students ($) 1.60 1.62 1.83 1.85 1.95 1.97 1.99 2.18 2.20 2.22 2.41 2.43 2.45 BRR, Eligible Students ($) 1.50 1.53 1.71 1.74 1.81 1.84 1.87 2.02 2.05 2.08 2.23 2.26 2.29 Difference (E C) ($) 0.10 0.09 0.12 0.11 0.14 0.13 0.12 0.16 0.15 0.14 0.18 0.18 0.17 E - C Percentage Difference 6 6 7 6 7 6 6 7 7 6 8 7 7 C NOTE: BRR = blended reimbursement rate. SOURCE: Prepared by the panel. 368

OCR for page 341
TABLE G-9 Illustration of Impact on BRR of Two Assumptions: (1) 25 Percent of Students Who Must Pay Full Price Applied for but Were Denied Approval for Free or Reduced-Price Meals and (2) Students Who Did Not Apply Were Not Eligible for Free or Reduced-Price Meals (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) Alternative Distributions of Certified Students (%) Free 45 50 55 60 60 65 70 70 75 80 80 85 90 Reduced Price 15 10 15 10 15 10 5 15 10 5 15 10 5 Full Price 40 40 30 30 25 25 25 15 15 15 5 5 5 Distributions of Eligible Students, Corrected for Certification Error (%) Free 46 48 54 56 58 60 63 66 69 71 74 77 79 Reduced Price 11 10 12 10 12 10 9 12 11 9 13 11 10 Full Price 43 42 34 33 30 29 28 22 21 20 13 12 11 BRRs BRR, Certified Students ($) 1.60 1.62 1.83 1.85 1.95 1.97 1.99 2.18 2.20 2.22 2.41 2.43 2.45 BRR, Eligible Students ($) 1.55 1.57 1.74 1.77 1.84 1.87 1.90 2.04 2.07 2.10 2.23 2.26 2.29 Difference (E C) ($) 0.05 0.04 0.09 0.08 0.11 0.10 0.09 0.14 0.13 0.12 0.18 0.17 0.16 E - C Percentage Difference 3 3 5 4 5 5 4 7 6 6 7 7 7 C NOTE: BRR = blended reimbursement rate. SOURCE: Prepared by the panel. 369

OCR for page 341
TABLE G-10 Illustration of Impact on BRR of Two Assumptions: (1) 40 Percent of Students Who Must Pay Full Price Applied for but Were Denied Approval for Free or Reduced-Price Meals and (2) Students Who Did Not Apply Were Not Eligible for Free or Reduced-Price Meals (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) Alternative Distributions of Certified Students (%) Free 45 50 55 60 60 65 70 70 75 80 80 85 90 Reduced Price 15 10 15 10 15 10 5 15 10 5 15 10 5 Full Price 40 40 30 30 25 25 25 15 15 15 5 5 5 Distributions of Eligible Students, Corrected for Certification Error (%) Free 47 49 55 57 59 61 64 66 69 72 74 77 79 Reduced Price 12 11 13 11 13 11 9 13 11 10 13 11 10 Full Price 41 40 33 32 29 28 27 21 20 19 13 12 11 BRRs BRR, Certified Students ($) 1.60 1.62 1.83 1.85 1.95 1.97 1.99 2.18 2.20 2.22 2.41 2.43 2.45 BRR, Eligible Students ($) 1.59 1.62 1.78 1.81 1.87 1.90 1.93 2.06 2.08 2.11 2.24 2.27 2.30 Difference (E C) ($) 0.01 0.00 0.05 0.05 0.08 0.07 0.06 0.13 0.12 0.11 0.17 0.16 0.16 E - C Percentage Difference 0 0 3 2 4 4 3 6 5 5 7 7 6 C NOTE: BRR = blended reimbursement rate. SOURCE: Prepared by the panel. 370

OCR for page 341
TABLE G-11 Illustration of Impact on BRR of Two Assumptions: (1) 10 Percent of Students Who Must Pay Full Price Applied for but Were Denied Approval for Free or Reduced-Price Meals and (2) 9.5 Percent of Students Who Did Not Apply Were Eligible for Free Meals and 8.3 Percent for Reduced-Price Meals (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) Alternative Distributions of Certified Students (%) Free 45 50 55 60 60 65 70 70 75 80 80 85 90 Reduced Price 15 10 15 10 15 10 5 15 10 5 15 10 5 Full Price 40 40 30 30 25 25 25 15 15 15 5 5 5 Distributions of Eligible Students, Corrected for Certification Error (%) Free 48 51 56 58 59 62 65 67 69 72 74 77 80 Reduced Price 13 12 13 12 13 12 10 13 12 10 13 11 10 Full Price 39 38 31 30 27 26 25 20 19 18 13 12 11 BRRs BRR, Certified Students ($) 1.60 1.62 1.83 1.85 1.95 1.97 1.99 2.18 2.20 2.22 2.41 2.43 2.45 BRR, Eligible Students ($) 1.64 1.67 1.81 1.84 1.90 1.93 1.96 2.07 2.10 2.13 2.25 2.28 2.30 Difference (E C) ($) 0.04 0.05 0.02 0.01 0.05 0.04 0.03 0.11 0.10 0.09 0.17 0.16 0.15 E - C Percentage Difference 2 3 1 1 3 2 2 5 5 4 7 7 6 C NOTE: BRR = blended reimbursement rate. SOURCE: Prepared by the panel. 371

OCR for page 341
TABLE G-12 Illustration of Impact on BRR of Two Assumptions: (1) 25 Percent of Students Who Must Pay Full Price Applied for but Were Denied Approval for Free or Reduced-Price Meals and (2) 9.5 Percent of Students Who Did Not Apply Were Eligible for Free Meals and 8.3 Percent for Reduced-Price Meals (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) Alternative Distributions of Certified Students (%) Free 45 50 55 60 60 65 70 70 75 80 80 85 90 Reduced Price 15 10 15 10 15 10 5 15 10 5 15 10 5 Full Price 40 40 30 30 25 25 25 15 15 15 5 5 5 Distributions of Eligible Students, Corrected for Certification Error (%) Free 49 51 56 59 60 62 65 67 70 72 74 77 80 Reduced Price 14 12 14 12 14 12 10 13 12 10 13 11 10 Full Price 38 37 30 29 27 26 25 20 19 18 12 11 10 BRRs BRR, Certified Students ($) 1.60 1.62 1.83 1.85 1.95 1.97 1.99 2.18 2.20 2.22 2.41 2.43 2.45 BRR, Eligible Students ($) 1.66 1.69 1.83 1.86 1.91 1.94 1.97 2.08 2.11 2.14 2.25 2.28 2.31 Difference (E C) ($) 0.06 0.07 0.00 0.01 0.03 0.03 0.02 0.10 0.09 0.08 0.16 0.16 0.15 E - C Percentage Difference 4 4 0 0 2 1 1 5 4 4 7 6 6 C NOTE: BRR = blended reimbursement rate. SOURCE: Prepared by the panel. 372

OCR for page 341
APPENDIX G 373 students who did not apply were eligible for free or reduced-price meals does not change the overstatement of the BRR for districts with very high free and reduced-price eligibility percentages. For districts with lower levels of eligibility, however, the impact is more dramatic, even contribut- ing to an understatement of the BRR.