4

Data Analysis and Results

To determine the suitability of the American Community Survey (ACS) as a source of claiming percentages for reimbursement under an ACS Eligibility Option (AEO) for universal free school meals, the panel implemented the technical approach described in Chapter 3 and conducted extensive analyses of the ACS direct and model-based estimates produced by the U.S. Census Bureau. This chapter describes the principal results of these analyses and presents the panel’s main conclusions. Additional results from our analyses are reported in Appendix F.1

The chapter begins with an analysis of the differences between ACS and administrative estimates, including consideration of the many reasons why such differences might arise. The potential sources of differences include errors in each set of estimates. ACS estimates are subject not only to sampling error but also to nonsampling error from, for example, households not responding at all to the survey or responding incorrectly by misreporting their incomes or whether they received benefits from the Supplemental Nutrition Assistance Program (SNAP, formerly the Food Stamp Program). Although not subject to sampling error, administrative estimates reflect the effects of certification error, as discussed in Chapter 2, as well as data entry, tabulation, and transmission error. Some differences between the estimates are undoubtedly attributable to the use of survey

____________

1 To simplify an already complex set of analyses, the panel focused on school lunches. For a district considering actual implementation of the AEO, it will be important to consider breakfasts separately from lunches, given the different reimbursement rates for the two programs.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 93
4 Data Analysis and Results T o determine the suitability of the American Community Survey (ACS) as a source of claiming percentages for reimbursement under an ACS Eligibility Option (AEO) for universal free school meals, the panel implemented the technical approach described in Chapter 3 and conducted extensive analyses of the ACS direct and model-based estimates produced by the U.S. Census Bureau. This chapter describes the principal results of these analyses and presents the panel's main conclu- sions. Additional results from our analyses are reported in Appendix F.1 The chapter begins with an analysis of the differences between ACS and administrative estimates, including consideration of the many rea- sons why such differences might arise. The potential sources of differ- ences include errors in each set of estimates. ACS estimates are subject not only to sampling error but also to nonsampling error from, for example, households not responding at all to the survey or responding incorrectly by misreporting their incomes or whether they received benefits from the Supplemental Nutrition Assistance Program (SNAP, formerly the Food Stamp Program). Although not subject to sampling error, administrative estimates reflect the effects of certification error, as discussed in Chapter 2, as well as data entry, tabulation, and transmission error. Some differences between the estimates are undoubtedly attributable to the use of survey 1To simplify an already complex set of analyses, the panel focused on school lunches. For a district considering actual implementation of the AEO, it will be important to consider breakfasts separately from lunches, given the different reimbursement rates for the two programs. 93

OCR for page 93
94 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS versus administrative procedures, while others arise because the proce- dures are intended to obtain different data. For example, the ACS collects data on income received in the past 12 months on a rolling basis. Thus households interviewed in January report on income received during the period from the previous January through December, while households interviewed in December report on income received during the period from the previous December through November. In contrast, school meals program applications obtain data on current monthly income, which will typically be income for the month in which the application is being completed or the previous month--probably July, August, or September for most students. Even if the data obtained by the ACS and by program applications are fully accurate, eligibility based on annual income can be different from eligibility and certification based on monthly income. Yet another difference is that the ACS records where students live, while school meals program certification data are based on where students attend school. In areas with school choice options, such as charter and magnet schools or open enrollment policies, some students may not attend their neighborhood school or even any school in the district in which they reside. This phenomenon will be captured in the administra- tive data but not the ACS data. The second section of the chapter presents the panel's analysis of the precision, intertemporal stability, and timeliness, as well as the general relative performance, of the alternative estimates from the ACS, includ- ing the 1-year, 3-year, 5-year, and model-based estimates. As discussed in Chapter 3, stability in reimbursement is important to districts because it facilitates budgeting and other planning activities involved in operat- ing the school meals programs. Nonetheless, some instability in reim- bursement occurs naturally under traditional operating procedures as a result of changes in certification percentages and participation rates from year to year due to ups and downs in the economy, outreach efforts by school authorities, and other factors. However, basing reimbursements on ACS estimates will introduce additional instability due to sampling variability and other sources of error that cause estimates to fluctuate. Because they are based on larger samples and average the data collected in different years, 5-year ACS estimates will tend to be more precise and stable than 3-year estimates, which will be more precise and stable than 1-year estimates. However, the precision and stability carry a cost: the 5-year estimates and, to a lesser degree, the 3-year estimates will be less timely and less responsive to real changes in socioeconomic conditions. The panel's analyses explored these trade-offs. The panel also explored the role of participation--that is, the pur- chase or free receipt of meals by students. Participation is important because it is the basis for reimbursing districts for the meals they serve

OCR for page 93
DATA ANALYSIS AND RESULTS 95 under traditional operating procedures or Provisions 2 and 3. The ACS, however, does not collect data on participation. It provides estimates of eligibility, specifically the numbers and percentages of students eligible for free, reduced-price, and full-price meals. Unless students in the three eligibility categories participate at the same rate, which, generally, they do not, the distributions across the categories of students and of meals served will not be the same and may differ substantially. Thus claiming percentages based entirely on the percentages of eligible students in each category will differ from claiming percentages based on the percentages of meals served in each category. In fact, with students eligible for free and reduced-price meals participating at higher rates than students pay- ing full price, claiming percentages based solely on ACS estimates of eligible students--with no accounting for differences in participation-- could cause districts to be substantially under reimbursed should they adopt the AEO. This effect could be at least partially mitigated, how- ever, by the changes in participation that might occur under the AEO with free meals being offered to all students, substantially lowering the monetary cost of meals for those students formerly paying full price and increasing their participation rates relative to other students. In the third section of the chapter, the panel analyzes the role of participation and the potential effect of offering free meals to all students under the AEO. In Chapter 5, we propose an approach to implementing the AEO that incorporates into the AEO claiming percentages not only the ACS eligibility estimates but also the participation rates of students when all are offered free meals. DIFFERENCES BETWEEN ACS AND ADMINISTRATIVE ESTIMATES The panel compared ACS estimates of students eligible for school meals by category (free, reduced price, full price) with administrative data on students certified for each category. The administrative data are from the Common Core of Data (CCD) for most of our analyses at the district level. School districts report data for the CCD to state agencies, which submit the data to the National Center for Education Statistics. For our school-level analyses, the case study districts provided the adminis- trative data directly to us at our request. As described in Chapter 3, the administrative data are subject to error; thus, they are not a gold stan- dard. However, they were the best standard available to us. Although we generally characterize average differences between ACS and administra- tive estimates as measures of systematic error in the ACS estimates, the limitations of the administrative data should be kept in mind. Later in this chapter, we explore the potential effects of certification error in the

OCR for page 93
96 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS administrative estimates on the differences between ACS and administra- tive estimates. The analyses presented in this chapter focus on those districts for which the AEO is most relevant: the districts described as "very high FRPL [free or reduced-price lunch]" and "high FRPL" in Chapter 3. A very high FRPL district had at least 75 percent of its students eligible for free or reduced-price meals according to the CCD in one or more school years from 2004-2005 to 2009-2010. Although a high FRPL district never reached that threshold, it did have at least 50 percent of its students eligible for free or reduced-price meals during one or more of those years. For some of our analyses of these districts, we present separate results for large, medium, and small districts. The large districts have populations of at least 65,000, and thus have 1-year ACS estimates as well as 3- and 5-year estimates. The medium districts have populations of 20,000 to 65,000 and have 3- and 5-year but not 1-year ACS estimates. The remaining districts, with populations under 20,000, have only 5-year ACS estimates and are designated as small. Although all districts included in our analyses have model-based estimates, we focus in this section on the 1-, 3-, and 5-year direct estimates from the ACS. Systematic Differences Between ACS and Administrative Estimates The panel's analyses revealed that ACS estimates differ systematically from administrative estimates for districts that might be most interested in the AEO. Figure 4-1 plots ACS and CCD estimates of the percentage of students eligible for free meals in very high FRPL districts. The ACS estimates are 5-year estimates for 2005-2009, and the CCD estimates are for school year (SY) 2009-2010. Because the purpose of using ACS data is to provide current estimates, we compare the most recent ACS estimates with the most recent estimates from the CCD. Thus, the ACS 5-year esti- mates for 2005-2009 are compared with the CDD estimates for SY 2009- 2010.2 Some of the observed average difference between these two sets of estimates maybe attributable solely to their different reference periods and the fact that the economy was worsening, resulting in an upward trend in the percentage of students eligible for free or reduced-price meals. From 2005 to 2009 according to the CCD, the percentage of stu- dents eligible for free or reduced-price meals in very high FRPL districts rose from 76.3 percent to 79.7 percent. For high FRPL and all districts, 2We follow the same principle with 3-year estimates, comparing the estimates for 2005- 2007, 2006-2008, and 2007-2009 with CCD estimates for school years 2007-2008, 2008-2009, and 2009-2010, respectively.

OCR for page 93
DATA ANALYSIS AND RESULTS 97 100 Small Medium Large 80 ACS 5-Year Estimate 60 40 20 0 0 20 40 60 80 100 CCD Estimate FIGURE 4-1 Comparison of ACS 5-year (2005-2009) and CCD (2009-2010) esti- mates for very high FRPL districts: Percentage of students eligible for free meals. NOTE: ACS = American Community Survey; CCD = Common Core of Data; FRPL FIG4-1.eps = free or reduced-price lunch. SOURCE: Prepared by the panel. this percentage increased from 52.9 percent to 59.8 percent and from 43.2 percent to 47.5 percent, respectively.3,4 3For very high FRPL districts, the percentages of students eligible for free or reduced-price meals were 76.3, 75.4, 75.3, 77.6, and 79.7 for 2005, 2006, 2007, 2008, and 2009, respectively, according to the CCD. For the high FRPL districts, the corresponding percentages were 52.9, 53.1, 54.2, 56.3, and 59.8, and for all districts, they were 43.2, 43.3, 43.8, 45.3, and 47.5. 4Although the use of older data is a potentially serious limitation of the 5-year estimates relative to the 1-year and even the 3-year estimates, we also compared the 5-year ACS esti- mates with 5-year averages of CCD estimates to assess their differences when they include,

OCR for page 93
98 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS In Figure 4-1, the overwhelming majority of districts fall below the 45 line of equality between the estimates, indicating that the ACS iden- tifies a smaller percentage of students as eligible for free meals relative to the CCD. For many of these very high FRPL districts, the percentage of students eligible for free meals according to the ACS is substantially lower than the percentage based on the administrative data on certified students. In contrast, a different pattern pertains to the estimates of students eli- gible for reduced-price meals. According to Figure 4-2,5 the ACS estimate exceeds the CCD estimate for a majority of districts, but the difference often is just a few percentage points. Many districts are clustered around the line of equality between the ACS and administrative estimates for the reduced-price category. The scatter plots in Figures 4-1 and 4-2 suggest that for the typical very high FRPL district, the ACS substantially underestimates the per- centage eligible for free meals and slightly overestimates the percentage eligible for reduced-price meals. The net effect of these patterns is that on average, the ACS estimate is substantially less than the CCD estimate for the percentage of students eligible for free or reduced-price meals and for the blended reimbursement rate (BRR) based on eligible students, as shown in Figures 4-3 and 4-4, respectively. Tables 4-1, 4-2, 4-3, and 4-4 provide numerical estimates of the aver- age differences between ACS and CCD eligibility percentages and BRRs. 6 The first column in the top panel of Table 4-1 pertains to 5-year estimates for all very high FRPL districts and corresponds to the results presented in principle, the same trend within the reference period of the estimates. The results of that comparison are qualitatively the same as the results of our comparisons of 5-year ACS esti- mates with 1-year CCD estimates. Although statistically significant for all types of estimates and large for percentage free, percentage free or reduced price, and BRR, the differences, of course, are smaller than those from our main comparisons because the differences based on CCD 5-year averages ignore the loss of timeliness due to the use of older data by the ACS 5-year estimates. Further details can be found in Appen dix F, which also presents a comparison of 3-year ACS estimates with 3-year averages of CCD estimates. 5In this figure, the 5-year ACS estimates have a relatively large number of sampling zeros because the percentage eligible for reduced-price meals is relatively small, and some districts are small areas. One possible reason for zeros in the CCD data is that missing data are recorded as zeros. 6For reasons given above, Tables 4-1, 4-2, and 4-4 compare the ACS 5- and 3-year estimates with the CCD estimates for the most recent school year in the reference period of the ACS estimates. Accordingly, the ACS 5-year estimates for 2005-2009 and 3-year estimates for 2007-2009, for example, are compared with the CCD estimates for SY 2009-2010. Appendix F presents tables that compare the ACS 5- and 3-year estimates with 5- and 3-year averages of CCD estimates for the same time periods. Such comparisons reflect differences when data are aligned in time but do not reflect the loss of timeliness that would result from using the multiyear estimates in the AEO.

OCR for page 93
DATA ANALYSIS AND RESULTS 99 60 50 40 ACS 5-Year Estimate 30 20 10 Small Medium Large 0 0 10 20 30 40 50 60 CCD Estimate FIGURE 4-2 Comparison of ACS 5-year (2005-2009) and CCD (2009-2010) esti- mates for very high FRPL districts: Percentage of students eligible for reduced- price meals. FIG4-2.eps NOTES: This figure excludes two outliers. Both are small districts. One has a CCD estimate of 2 percent and an ACS estimate of 78 percent, and the other has a CCD estimate of 80 percent and an ACS estimate of 6 percent. ACS = American Community Survey; CCD = Common Core of Data; FRPL = free or reduced-price lunch. SOURCE: Prepared by the panel. in Figures 4-1 through 4-4. The last three columns in the top panel of Table 4-1 provide separate estimates for large, medium, and small districts, and the bottom panel provides estimates of average ACS-CCD differences for high FRPL districts. Tables 4-2 and 4-3 display average ACS-CCD differ- ences for 3-year and 1-year ACS estimates, respectively. Table 4-2 includes

OCR for page 93
100 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS 100 Small Medium Large 80 ACS 5-Year Estimate 60 40 20 0 0 20 40 60 80 100 CCD Estimate FIGURE 4-3 Comparison of ACS 5-year (2005-2009) and CCD (2009-2010) es- timates for very high FRPL districts: Percentage of students eligible for free or reduced-price meals. FIG4-3.eps NOTE: ACS = American Community Survey; CCD = Common Core of Data; FRPL = free or reduced-price lunch. SOURCE: Prepared by the panel. only large and medium districts because small districts do not have 3-year ACS estimates. Similarly, Table 4-3 includes only large districts because they are the only districts with 1-year ACS estimates. Table 4-2 provides results for each of the three available sets of 3-year estimates (2005-2007, 2006-2008, and 2007-2009), and Table 4-3 provides results for each of the five available sets of 1-year estimates. All differences in each of these

OCR for page 93
DATA ANALYSIS AND RESULTS 101 2.5 Small Medium Large 2.0 ACS 5-Year Estimate 1.5 1.0 0.5 0.5 1.0 1.5 2.0 2.5 CCD Estimate FIGURE 4-4 Comparison of ACS 5-year (2005-2009) and CCD (2009-2010) esti- FIG4-4.eps mates for very high FRPL districts: BRR. NOTES: This figure excludes three outliers, all of which are small districts with ACS BRRs of $0.26. Their CCD BRRs are $1.50, $2.10, and $2.10. ACS = American Community Survey; BRR = blended reimbursement rate; CCD = Common Core of Data; FRPL = free or reduced-price lunch. SOURCE: Prepared by the panel. tables are statistically significant, that is, significantly different from zero.7 Table 4-4 summarizes the results in the other tables by averaging across the three sets of 3-year estimates and the five sets of 1-year estimates. 7Statistical significance is determined by comparing the ratio of the average difference to its estimated standard error with critical values from a standard normal distribution.

OCR for page 93
102 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS TABLE 4-1 Average Differences Between ACS 5-Year Estimates for 2005-2009 and CCD Estimates for 2009-2010 All Large Medium Small Estimand Districts Districts Districts Districts Very High FRPL Districts (1,641) (122) (227) (1,292) Percentage free 21.7 19.5 20.4 22.2 Percentage reduced price 4.0 4.5 5.0 3.8 Percentage free or reduced price 17.8 15.0 15.4 18.4 BRR, $ 0.43 0.37 0.38 0.44 High FRPL Districts (4,214) (304) (710) (3,200) Percentage free 10.8 13.6 12.1 10.3 Percentage reduced price 2.3 2.7 2.8 2.1 Percentage free or reduced price 8.5 11.0 9.3 8.1 BRR, $ 0.21 0.27 0.23 0.20 NOTES: All average differences are statistically significant (different from zero) at the 0.01 level. ACS = American Community Survey; BRR = blended reimbursement rate; CCD = Common Core of Data; FRPL = free or reduced-price lunch. SOURCE: Prepared by the panel. For very high FRPL districts, several consistent patterns emerge from these tables of average ACS-CCD differences: The average ACS estimate of the percentage of students eligible for free meals is typically 15 to 22 percentage points lower than the average CCD estimate. The average ACS estimate of the percentage of students eligible for reduced-price meals is typically about 3 to 4 percentage points higher than the average CCD estimate. The ACS's overestimation of the percentage eligible for reduced- price meals is not sufficient to compensate for the underestima- tion of the percentage eligible for free meals. Thus, the average ACS estimate of the percentage eligible for either free or reduced- price meals is typically 12 to 18 percentage points lower than the average CCD estimate. For a BRR based on the distribution of students across categories, the average ACS estimate is usually about $0.30 to $0.40 lower than the average CCD estimate of roughly $2.10. Qualitatively similar patterns are observed for average high FRPL districts: overestimation of the percentage reduced-price-eligible, but underestimation of the percentage free-eligible, the percentage free or reduced-price-eligible, and the BRR. Also, all of the differences are statis-

OCR for page 93
TABLE 4-2 Average Differences Between ACS 3-Year Estimates and CCD Estimates for Last School Year in ACS Reference Period Large and Medium Districts Large Districts Medium Districts 2005- 2006- 2007- 2005- 2006- 2007- 2005- 2006- 2007- Estimand 2007 2008 2009 2007 2008 2009 2007 2008 2009 Very High FRPL Districts (337) (350) (349) (121) (123) (122) (216) (227) (227) Percentage free 17.1 18.6 20.1 15.5 18.2 19.5 17.9 18.8 20.4 Percentage reduced price 3.6 3.1 3.9 3.7 2.9 3.7 3.5 3.2 4.1 Percentage free or reduced price 13.5 15.5 16.2 11.8 15.3 15.7 14.5 15.6 16.4 BRR, $ 0.33 0.37 0.39 0.29 0.37 0.38 0.35 0.38 0.40 High FRPL Districts (972) (1,012) (1,014) (298) (303) (304) (674) (709) (710) Percentage free 8.4 10.2 12.7 9.8 11.2 13.6 7.8 9.8 12.3 Percentage reduced price 1.9 1.7 2.3 1.9 1.7 2.2 1.9 1.7 2.3 Percentage free or reduced price 6.5 8.5 10.5 7.9 9.5 11.4 5.9 8.1 10.1 BRR, $ 0.16 0.21 0.25 0.19 0.23 0.27 0.15 0.20 0.24 NOTES: All average differences are statistically significant (different from zero) at the 0.01 level. ACS = American Community Survey; BRR = blended reimbursement rate; CCD = Common Core of Data; FRPL = free or reduced-price lunch. SOURCE: Prepared by the panel. 103

OCR for page 93
140 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS TABLE 4-15 BRRs Based on Certified Students Versus BRRs Based on Meals Served: Illustration with Case Study District Schools Claiming Percentages Participation Rates (%) Certified Students Reduced Full Reduced Full School Free Price Price Free Price Price 1 98 94 23 65 7 27 2 96 9 85 63 11 26 3 85 64 35 65 9 26 4 96 91 71 59 16 25 5 96 79 44 68 8 24 6 91 82 59 57 20 23 7 63 55 18 67 10 23 8 74 71 71 68 9 23 9 93 94 83 64 14 22 10 45 31 6 72 7 21 11 57 44 16 74 6 20 12 96 86 66 76 6 18 13 89 95 27 75 6 18 14 89 87 74 68 15 17 15 77 67 33 77 9 15 16 99 93 33 80 7 14 17 97 98 55 83 4 13 18 90 89 82 83 5 12 19 82 67 35 78 11 11 20 96 90 90 84 5 10 21 62 41 28 77 13 10 22 70 47 22 82 8 10 23 95 93 60 88 3 9 24 92 92 68 84 8 8 25 86 80 40 87 7 6 26 95 95 95 89 6 5 27 94 67 60 88 7 5 28 90 84 78 86 10 4 29 84 77 78 89 7 4 30 90 83 60 87 9 4 NOTE: BRRs = blended reimbursement rates. SOURCE: Prepared by the panel.

OCR for page 93
DATA ANALYSIS AND RESULTS 141 Claiming Percentages Meals Served BRRs Certified Reduced Full Students Meals Difference Percentage Free Price Price ($) Served ($) ($) Difference 83 9 8 1.92 2.36 0.44 19 72 1 27 1.94 1.97 0.03 1 79 8 13 1.95 2.26 0.31 14 64 17 20 1.95 2.06 0.11 5 80 8 13 2.01 2.27 0.26 11 64 20 17 1.97 2.12 0.15 7 82 10 8 2.01 2.36 0.35 15 69 9 22 2.02 2.03 0.02 1 66 15 20 2.03 2.07 0.04 2 90 6 3 2.08 2.49 0.41 17 87 6 7 2.10 2.41 0.31 13 81 5 14 2.14 2.25 0.11 5 86 8 6 2.14 2.41 0.27 11 70 15 14 2.14 2.20 0.06 3 85 8 7 2.22 2.39 0.18 7 88 7 5 2.24 2.44 0.20 8 88 4 8 2.27 2.39 0.12 5 84 5 11 2.28 2.31 0.02 1 85 10 5 2.29 2.43 0.14 6 85 5 10 2.33 2.34 0.01 1 85 10 5 2.31 2.44 0.13 5 91 6 3 2.34 2.49 0.15 6 91 3 6 2.37 2.44 0.07 3 86 8 6 2.37 2.42 0.05 2 90 7 3 2.42 2.50 0.07 3 89 6 5 2.45 2.45 0.00 0 91 6 3 2.45 2.50 0.04 2 87 9 4 2.45 2.46 0.02 1 90 7 4 2.47 2.48 0.01 0 89 8 3 2.47 2.50 0.03 1

OCR for page 93
142 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS TABLE 4-16 Illustration of Potential Participation Effects of Universal Free Meals Under the AEO in Case Study Districts Participation Rates Actual, Pre-AEO (%) Illustrative, Post-AEO (%) Reduced Full Reduced Full District Free Price Price Free Price Price Austin, Texas 86 72 34 91 88 44 Chatham County, Georgia 75 72 48 80 77 58 Norfolk, Virginia 77 71 43 82 79 53 Omaha, Nebraska 92 84 61 97 94 71 Pajaro Valley, California 68 52 23 73 70 33 Claiming Percentages (based on meals served) Actual, Pre-AEO (%) Illustrative, Post-AEO (%) Reduced Full Reduced Full District Free Price Price Free Price Price Austin, Texas 73 8 19 69 9 22 Chatham County, Georgia 67 10 23 65 9 26 Norfolk, Virginia 59 12 29 56 12 32 Omaha, Nebraska 58 12 30 56 12 32 Pajaro Valley, California 77 9 14 72 10 18 BRRs Actual, Illustrative, Pre-AEO Post-AEO Difference Percentage District ($) ($) ($) Difference Austin, Texas 2.12 2.05 0.07 3 Chatham County, Georgia 2.01 1.96 0.05 3 Norfolk, Virginia 1.87 1.81 0.07 4 Omaha, Nebraska 1.84 1.80 0.04 2 Pajaro Valley, California 2.22 2.13 0.09 4 NOTE: AEO = American Community Survey (ACS) Eligibility Option; BRRs = blended reimbursement rates. SOURCE: Prepared by the panel. bursement rate per meal. However, another potentially important con- sideration is that any changes in participation rates under the AEO could impact the total reimbursement received by a district by affecting not only the BRR but also the total number of meals served. Thus, a district would have to assess the cost implications of a change in the scale of food service operations. A large increase in the total number of meals served might require, for example, that more staff be hired or that the kitchen facilities be expanded.

OCR for page 93
DATA ANALYSIS AND RESULTS 143 SUMMARY AND CONCLUSIONS The panel's evaluations of the ACS-based eligibility estimates encom- passed a wide range of issues. The main results of our analyses include the following: ACS estimates are systematically different from administrative estimates for high and very high FRPL districts. BRRs based on ACS estimates of eligible students are substan- tially less than BRRs based on CCD estimates of certified students for high and very high FRPL districts, on average. Average ACS-CCD differences are larger for very high FRPL dis- tricts than for high FRPL districts. There are several potentially important sources of systematic differences between ACS and administrative estimates, and the effects of these sources are likely to vary across districts. A statistical model can explain a substantial fraction--but far from all--of the variability across districts in the differences between ACS and administrative estimates. Relative to the inter temporal changes in BRRs normally expe- rienced by a district, as reflected in the administrative data on certified students, the typical large district would likely experience less variability if it used 3- or 5-year ACS estimates but greater variability if it used 1-year ACS estimates.39 The typical medium district would experience about the same variability as normal if it used 3-year ACS estimates and less variability than normal if it used 5-year ACS estimates. The typical small district would experi- ence somewhat less than normal variability if it used 5-year ACS estimates. For districts with enrollments of 400 or higher, ACS 5-year esti- mates would probably be as stable or more so than the districts' administrative estimates. The 5-year estimates might be less sta- ble than administrative estimates for smaller districts. For small very high FRPL districts, average differences between model-based ACS estimates and CCD BRR estimates are sub- stantially larger than average differences between ACS 5-year estimates and CCD estimates. Based on overall accuracy and consideration of error due to both variability and bias, the 5-year estimates would likely be more accurate than the 3-year estimates for medium districts. For large 39As noted previously, a typical school in a category has an enrollment at about the median enrollment for schools in the category.

OCR for page 93
144 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS TABLE 4-17 Illustration of Potential Participation Effects of Universal Free Meals Under the AEO in Case Study District Schools Claiming Percentages Participation Rates Based on Meals Served Illustrative, Post-AEO Actual, Pre-AEO (%) (%) Actual, Pre-AEO Reduced Full Reduced Full Reduced Full School Free Price Price Free Price Price Free Price Price 1 98 94 23 100 97 33 83 9 8 2 96 9 85 100 97 94 72 1 27 3 85 64 35 90 87 45 79 8 13 4 96 91 71 100 97 81 64 17 20 5 96 79 44 100 97 54 80 8 13 6 91 82 59 96 93 69 64 20 17 7 63 55 18 68 65 28 82 10 8 8 74 71 71 79 76 73 69 9 22 9 93 94 83 98 95 92 66 15 20 10 45 31 6 50 47 16 90 6 3 11 57 44 16 62 59 26 87 6 7 12 96 86 66 100 97 76 81 5 14 13 89 95 27 94 91 37 86 8 6 14 89 87 74 94 91 84 70 15 14 15 77 67 33 82 79 43 85 8 7 16 99 93 33 100 97 43 88 7 5 17 97 98 55 100 97 65 88 4 8 18 90 89 82 95 92 89 84 5 11 19 82 67 35 87 84 45 85 10 5 20 96 90 90 100 97 94 85 5 10 21 62 41 28 67 64 38 85 10 5 22 70 47 22 75 72 32 91 6 3 23 95 93 60 100 97 70 91 3 6 24 92 92 68 97 94 78 86 8 6 25 86 80 40 91 88 50 90 7 3 26 95 95 95 100 97 94 89 6 5 27 94 67 60 99 96 70 91 6 3 28 90 84 78 95 92 88 87 9 4 29 84 77 78 89 86 83 90 7 4 30 90 83 60 95 92 70 89 8 3 NOTE: AEO = American Community Survey (ACS) Eligibility Option; BRR = blended reimbursement rate. SOURCE: Prepared by the panel.

OCR for page 93
DATA ANALYSIS AND RESULTS 145 Illustrative, Post-AEO BRR Reduced Full Actual Pre- Illustrative Percentage Free Price Price AEO ($) Post-AEO ($) Difference ($) Difference 80 9 11 2.36 2.29 0.07 3 64 11 25 1.97 1.96 0.00 0 75 10 15 2.26 2.20 0.05 2 62 17 21 2.06 2.03 0.03 1 77 9 14 2.27 2.22 0.04 2 61 21 18 2.12 2.09 0.03 1 78 11 11 2.36 2.29 0.07 3 69 9 22 2.03 2.05 0.02 1 65 14 21 2.07 2.05 0.02 1 84 8 8 2.49 2.38 0.11 4 84 7 10 2.41 2.34 0.07 3 80 6 15 2.25 2.22 0.03 1 85 7 8 2.41 2.37 0.04 2 70 15 15 2.20 2.18 0.02 1 83 9 8 2.39 2.36 0.03 1 87 7 6 2.44 2.41 0.03 1 87 4 9 2.39 2.36 0.02 1 84 5 12 2.31 2.30 0.01 0 83 11 6 2.43 2.41 0.03 1 85 5 10 2.34 2.34 0.00 0 81 13 6 2.44 2.40 0.04 1 87 8 4 2.49 2.46 0.03 1 91 3 6 2.44 2.43 0.01 1 86 7 7 2.42 2.40 0.01 0 90 7 3 2.50 2.48 0.01 1 90 6 5 2.45 2.46 0.01 0 89 7 3 2.50 2.48 0.01 1 87 9 4 2.46 2.46 0.01 0 80 7 4 2.48 2.48 0.00 0 88 9 3 2.50 2.49 0.01 0

OCR for page 93
TABLE 4-18 BRRs for Case Study Districts Based on Certified Students Versus Meals Served Under Traditional Operating Procedures and the AEO Blended Reimbursement Rates Difference from Certified Students BRR Actual Meals Served, Illustrative Meals Served, Meals Served Pre-AEO Post-AEO Certified Actual, Pre- Illustrative, Difference Percentage Difference Percentage District Students ($) AEO ($) Post-AEO ($) ($) Difference ($) Difference Austin, Texas 1.71 2.12 2.05 0.41 24 0.34 20 Chatham County, Georgia 1.80 2.01 1.96 0.21 11 0.15 9 Norfolk, Virginia 1.59 1.87 1.81 0.29 18 0.22 14 Omaha, Nebraska 1.64 1.84 1.80 0.20 12 0.16 10 Pajaro Valley, California 1.81 2.22 2.13 0.41 23 0.33 18 NOTE: AEO = American Community Survey (ACS) Eligibility Option; BRR = blended reimbursement rate. SOURCE: Prepared by the panel. 146

OCR for page 93
DATA ANALYSIS AND RESULTS 147 districts, both 3- and 5-year estimates would likely be more accu- rate than 1-year estimates. However, whether the 3- or 5-year estimates would be more accurate is less clear. Although some results suggest that the 3-year estimates appear to strike the most effective compromise between precision and stability on the one hand and responsiveness to change on the other, the panel was unable to perform some analyses because the sets of estimates available to us were too limited. BRRs based on the distribution of certified students can be sub- stantially less than BRRs based on the distribution of meals served, although changes in participation after adoption of the AEO could reduce these differences. Based on the panel's empirical analyses, as well as consultations with experts and reviews of relevant documents, the panel reached the follow- ing conclusions: Conclusion 4-1: A one-size-fits-all approach for benchmarking ACS estimates of students eligible for school meals to administrative estimates to minimize the differences caused by such factors as underreporting of SNAP participation is not possible at present. Further research will be required to determine whether a techni- cally sound and operationally feasible set of procedures for estimat- ing the necessary adjustments to the ACS estimates can be devel- oped. Furthermore, even if such procedures were identified and used, additional adjustments based on a district's own data might improve the benchmarking of the ACS estimates to administrative estimates. Conclusion 4-2: Medium districts generally should prefer the 5-year ACS estimates to the 3-year estimates, and large districts generally should prefer either the 3- or 5-year estimates to the 1-year esti- mates. However, it is not clear whether large districts should prefer the 3- or 5-year estimates. Conclusion 4-3: Although all districts should thoroughly assess their estimates and the potential implications of adopting the AEO, as discussed in detail in Chapter 5, districts with enrollments below 400 should consider especially carefully whether reimburse- ments might fluctuate too much if they were based on ACS 5-year estimates.40 40Many districts fall in this category--about 30 percent of the very high FRPL districts.

OCR for page 93
148 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS TABLE 4-19 BRRs for Case Study District Schools Based on Certified Students Versus Meals Served Under Traditional Operating Procedures and the AEO BRR Meals Served Certified Students Actual, Pre-AEO Illustrative, Post-AEO School ($) ($) ($) 1 1.92 2.36 2.29 2 1.94 1.97 1.96 3 1.95 2.26 2.20 4 1.95 2.06 2.03 5 2.01 2.27 2.22 6 1.97 2.12 2.09 7 2.01 2.36 2.29 8 2.02 2.03 2.05 9 2.03 2.07 2.05 10 2.08 2.49 2.38 11 2.10 2.41 2.34 12 2.14 2.25 2.22 13 2.14 2.41 2.37 14 2.14 2.20 2.18 15 2.22 2.39 2.36 16 2.24 2.44 2.41 17 2.27 2.39 2.36 18 2.28 2.31 2.30 19 2.29 2.43 2.41 20 2.33 2.34 2.34 21 2.31 2.44 2.40 22 2.34 2.49 2.46 23 2.37 2.44 2.43 24 2.37 2.42 2.40 25 2.42 2.50 2.48 26 2.45 2.45 2.46 27 2.45 2.50 2.48 28 2.45 2.46 2.46 29 2.47 2.48 2.48 30 2.47 2.50 2.49 NOTE: AEO = American Community Survey (ACS) Eligibility Option; BRR = blended reimbursement rate. SOURCE: Prepared by the panel.

OCR for page 93
DATA ANALYSIS AND RESULTS 149 Difference from Certified Students BRR Actual Meals Served, Pre-AEO Illustrative Meals Served, Post-AEO Percentage Percentage Difference ($) Difference Difference ($) Difference 0.44 23 0.37 19 0.03 1 0.03 1 0.31 16 0.25 13 0.11 6 0.08 4 0.26 13 0.22 11 0.15 8 0.12 6 0.35 17 0.28 14 0.02 1 0.03 2 0.04 2 0.02 1 0.41 20 0.30 15 0.31 15 0.24 11 0.11 5 0.09 4 0.27 13 0.24 11 0.06 3 0.04 2 0.18 8 0.14 6 0.20 9 0.17 8 0.12 5 0.10 4 0.02 1 0.02 1 0.14 6 0.11 5 0.01 1 0.01 1 0.13 6 0.09 4 0.15 7 0.12 5 0.07 3 0.06 2 0.05 2 0.04 1 0.07 3 0.06 3 0.00 0 0.01 0 0.04 2 0.03 1 0.02 1 0.01 0 0.01 0 0.01 0 0.03 1 0.02 1

OCR for page 93
150 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS Conclusion 4-4: To develop accurate claiming percentages for use in implementing the AEO, it will be necessary to estimate not only the distribution of eligible students across the free, reduced-price, and full-price categories but also their expected participation rates with all meals being served free of charge. As documented in this chapter, the panel's analyses demonstrate that the ACS eligibility estimates, on average, are substantially and systemati- cally different from administrative estimates for high and very high FRPL districts. For all but the smallest districts, however, reimbursements based on ACS estimates might be equally stable over time and often more so than reimbursements based on administrative estimates, and this feature of the AEO might be attractive to districts along with its other benefits. Although a one-size-fits-all approach for benchmarking ACS estimates to administrative estimates is not feasible at present, a tailored approach to using ACS estimates could possibly allow more districts to offer free meals to all students under the AEO. In the next chapter, we propose an approach that FNS might consider for implementing the AEO and that some districts might find attractive if they wished to adopt the AEO in all or some of their schools.