Appendix F

Additional Information About the Panel’s Analyses

This appendix provides additional information about and results from the analyses conducted by the panel, as described in Chapter 4. Included are three parts. The first complements the comparisons discussed in Chapter 4 with some additional tables concerning the differences between American Community Survey (ACS) estimates and administrative estimates based on the National Center for Education Statistics’(NCES) Common Core of Data (CCD). The second part describes the model used to assess stability over time and provides detailed model results. The third part describes the panel’s exploration of the use of global regression models for predicting differences between ACS and CCD estimates for the blended reimbursement rate (BRR) using a variety of covariates from the CCD.

PART 1: COMPARISONS OF ACS ESTIMATES AND ESTIMATES BASED ON ADMINISTRATIVE DATA

Tables F-1 and F-2 display the differences between ACS multiyear averages and CCD multiyear averages computed over roughly the same time periods. Table F-1 displays comparisons for 5-year estimates and Table F-2 for 3-year estimates. These tables present differences by district size (small, medium, and large), and free or reduced-price lunch (FRPL) category (very high, high, and low to moderate) for percentage eligible for free meals, percentage eligible for reduced-price meals, percentage eligible for free or reduced-price meals, and the BRR.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 316
Appendix F Additional Information About the Panel's Analyses T his appendix provides additional information about and results from the analyses conducted by the panel, as described in Chapter 4. Included are three parts. The first complements the comparisons discussed in Chapter 4 with some additional tables concerning the differ- ences between American Community Survey (ACS) estimates and admin- istrative estimates based on the National Center for Education Statistics' (NCES) Common Core of Data (CCD). The second part describes the model used to assess stability over time and provides detailed model results. The third part describes the panel's exploration of the use of global regression models for predicting differences between ACS and CCD estimates for the blended reimbursement rate (BRR) using a variety of covariates from the CCD. PART 1: COMPARISONS OF ACS ESTIMATES AND ESTIMATES BASED ON ADMINISTRATIVE DATA Tables F-1 and F-2 display the differences between ACS multiyear averages and CCD multiyear averages computed over roughly the same time periods. Table F-1 displays comparisons for 5-year estimates and Table F-2 for 3-year estimates. These tables present differences by district size (small, medium, and large), and free or reduced-price lunch (FRPL) category (very high, high, and low to moderate) for percentage eligible for free meals, percentage eligible for reduced-price meals, percentage eligible for free or reduced-price meals, and the BRR. 316

OCR for page 316
APPENDIX F 317 TABLE F-1 Average Differences Between ACS 5-Year Estimates and 5-Year Averages of CCD Estimates All Large Medium Small Estimand Districts Districts Districts Districts Very High FRPL Districts (1,435) (113) (207) (1,115) Percentage free 17.7 15.2 17.3 18.0 Percentage reduced price 3.2 3.6 4.1 3.0 Percentage free or reduced price 14.5 11.7 13.2 15.0 BRR, $ 0.35 0.29 0.32 0.36 High FRPL Districts (3,782) (280) (628) (2,874) Percentage free 6.5 8.8 7.3 6.2 Percentage reduced price 1.9 2.2 2.3 1.8 Percentage free or reduced price 4.7 6.6 5.0 4.4 BRR, $ 0.12 0.16 0.13 0.11 Low to Moderate FRPL Districts (3,634) (263) (553) (2,818) Percentage free 1.4 3.7 2.9 9.4 Percentage reduced price 2.3 2.0 1.9 2.4 Percentage free or reduced price 0.8 1.7 1.0 1.4 BRR, $ 0.01 0.05 0.03 0.02 NOTES: The ACS 5-year estimates (for 2005-2009) are compared with the average of CCD estimates for 2005-2006, 2006-2007, 2007-2008, 2008-2009, and 2009-2010. ACS = American Community Survey; BRR = blended reimbursement rate; CCD = Common Core of Data; FPRL = free or reduced-price lunch. SOURCE: Prepared by the panel. The purpose of this comparison is to illustrate the differences that exist when the reference periods of the ACS and administrative estimates are as similar as possible. These tables display the same patterns as those observed in Chapter 4, where the administrative estimates pertain to the most recent year of the reference period for the ACS estimates. Namely, the ACS understates percentage free, percentage free or reduced price, and the BRR and overstates percentage reduced price. The differences are substantial in very high FRPL districts and are least pronounced in low to moderate FRPL districts; high FRPL districts fall in between. Over all districts, the BRR is understated by the 5-year ACS by 35 cents for very high FRPL districts and 12 cents for high FRPL districts, and is overstated by 1 cent in low to moderate FRPL districts. Chapter 4 highlights the systematic differences between ACS and CCD estimates for eligibility percentages and the BRR. The following tables compare enrollment estimates from the two sources. Tables F-3 and F-4 illustrate the differences between ACS multiyear estimates and CCD

OCR for page 316
318 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS TABLE F-2 Average Differences Between ACS 3-Year Estimates and 3-Year Averages of CCD Estimates Large and Medium Districts Estimand 2005-2007 2006-2008 2007-2009 Very High FRPL Districts (327) (333) (329) Percentage free 17.1 17.6 17.6 Percentage reduced price 3.5 2.9 3.2 Percentage free or reduced price 13.6 14.7 14.4 BRR, $ 0.33 0.35 0.35 High FRPL Districts (918) (964) (962) Percentage free 7.5 8.7 9.5 Percentage reduced price 1.9 1.7 1.9 Percentage free or reduced price 5.6 7.0 7.6 BRR, $ 0.14 0.17 0.19 Low to Moderate FRPL Districts (830) (916) (973) Percentage free 2.8 3.5 4.1 Percentage reduced price 1.6 1.3 1.3 Percentage free or reduced price 1.2 2.2 2.9 BRR, $ 0.03 0.06 0.07 NOTES: The ACS 3-year estimates are compared with 3-year averages of CCD estimates. For example, the ACS estimates for 2005-2007 are compared with the average of CCD estimates for 2005-2006, 2006-2007, and 2007-2008. ACS = American Community Survey; BRR = blend- ed reimbursement rate; CCD = Common Core of Data; FPRL = free or reduced-price lunch. SOURCE: Prepared by the panel. multiyear average estimates computed over the same time periods as the ACS estimates, as well as the differences between the ACS multiyear esti- mates and the CCD estimates for the most recent school year that overlaps the ACS reference period. (For the latter, the ACS estimate for 2005-2009 is compared with the CCD estimate for 2009-2010, and the ACS estimate for 2007-2009 is also compared with the CCD estimate for 2009-2010.) In addition to sampling error in the ACS estimates and various other errors in both the ACS and administrative estimates, enrollment estimates may differ because school district boundaries are different in different years. All of the ACS estimates are based on the school district boundar- ies recorded in the Census Bureau's Topologically Integrated Geographic Encoding and Referencing (TIGER) database for 2009-2010 and data reflecting the number of students that resided within those boundaries at some time during a calendar year. On the other hand, the CCD data reflect the district's enrollment as of October of a school year based on the boundaries for that year. School choice is another reason why enrollment estimates may differ. Children who live in the catchment area of a school

OCR for page 316
APPENDIX F 319 Large Districts Medium Districts 2005-2007 2006-2008 2007-2009 2005-2007 2006-2008 2007-2009 (118) (119) (116) (209) (214) (213) 15.1 16.4 16.8 18.2 18.3 18.1 3.6 2.8 3.0 3.4 3.0 3.4 11.5 13.6 13.8 14.7 15.3 14.7 0.28 0.33 0.33 0.36 0.37 0.36 (286) (293) (292) (632) (671) (670) 8.9 9.8 10.4 7.0 8.2 9.2 2.1 1.7 1.8 1.9 1.7 1.9 6.8 8.2 8.6 5.1 6.5 7.2 0.17 0.20 0.21 0.13 0.16 0.18 (270) (293) (303) (560) (623) (670) 3.3 4.3 4.7 2.6 3.1 3.9 1.8 1.3 1.4 1.6 1.3 1.2 1.4 3.0 3.2 1.0 1.8 2.7 0.04 0.08 0.08 0.03 0.08 0.07 district and attend public school may not attend a school associated with the local public school district; some may attend an independent charter school, for example. These differences are discussed more fully in Chap- ter 4. Differences in the inclusion of prekindergarten students might also contribute to differences in enrollment estimates. The differences shown in Table F-3 for the 5-year ACS estimates tend to be relatively small, but are largest (11 percent) for large very high FRPL districts (when compared with CCD estimates for 2009-2010). Other cat- egories of districts have differences of 4 percent or less. The 5-year ACS estimates tend to overstate enrollment in very high FRPL districts and to understate enrollment in low to moderate FRPL districts. Similar patterns are illustrated in Table F-4, where small districts are not included because there are no 3-year ACS estimates for them. Table F-5 shows the average differences between ACS 1-year esti- mates for enrollment and the CCD estimates for enrollment for each of 5 years. The ACS calendar-year estimates are compared with the CCD school year estimates for the most recent school year that overlaps the

OCR for page 316
320 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS TABLE F-3 Average Differences Between ACS 5-Year Estimates of Enrollment and Various CCD Estimates All Large Medium Small Estimand Districts Districts Districts Districts Very High FRPL Districts Difference from CCD for 09-10 358 4,038 233 33 As percentage of 09-10 CCD 9 11 4 4 Difference from CCD 5-year average 248 2,787 175 5 As percentage of CCD 5-year average 6 7 3 1 High FRPL Districts Difference from CCD for 09-10 25 19 27 26 As percentage of 09-10 CCD 1 0 0 3 Difference from CCD 5-year average 47 188 32 36 As percentage of CCD 5-year average 1 1 1 4 Low to Moderate FRPL Districts Difference from CCD for 09-10 124 1,040 192 30 As percentage of 09-10 CCD 4 4 4 3 Difference from CCD 5-year average 112 647 161 53 As percentage of CCD 5-year average 3 3 3 5 NOTES: The ACS 5-year estimates are compared with (1) CCD estimates for the most recent school year that overlaps the reference period of the ACS estimates (so the ACS estimates for 2005-2009 are compared with CCD estimates for 2009-2010) and (2) 5-year averages of CCD estimates (so the ACS estimates for 2005-2009 are compared with the average of CCD estimates for 2005-2006, 2006-2007, 2007-2008, 2008-2009, and 2009-2010). ACS = American Community Survey; CCD = Common Core of Data; FPRL = free or reduced-price lunch. SOURCE: Prepared by the panel. calendaryear. (Hence, the ACS estimate for 2009 is compared with the CCD estimate for 2009-2010.) These results are only for large districts that have ACS 1-year estimates. The percentage differences are again largest for the very high FRPL districts (averaging almost 10 percent) and low- est for the low to moderate FRPL districts (averaging about 5 percent); the high FRPL districts average .3 percent. Here the average differences appear to be increasing in magnitude over time for both the very high and low to moderate FRPL categories. Tables F-6 through F-8 display the average differences between vari- ous ACS estimates (5-year, 3-year, and 1-year) and the CCD estimate for the most recent school year that overlaps the reference period of the ACS estimate for low to moderate FRPL districts. These tables complement Tables 4-1, 4-2, and 4-3 in Chapter 4, which present results for the very high and high FRPL districts. Each table shows average differences for percentage free, percentage reduced price, percentage free or reduced price, and the BRR. Tables F-6, F-7, and F-8 show the same patterns of dif-

OCR for page 316
APPENDIX F 321 ferences as the tables in Chapter 4, but the magnitudes of the differences are much smaller. PART 2: MODELING OF VARIATION Let Adt denote the 1-year ACS estimate of the true BRR, Cdt, for school district d in year t, where Cdt is the BRR value as computed from the CCD.1 We write Cdt = t + Md + mdt where t is a common time trend across districts, Md is a district-specific deviation that is constant over time, and mdt is the district- and time- specific deviation from the common time trend and constant district deviation. We write Adt = Cdt + bt + Bd + bdt + edt where edt is sampling error with known variance s2dt, and bt + Bd + bdt rep- resents the difference between the CCD and ACS estimates after sampling error is removed. Because the CCD is treated as the gold standard in this discussion, we refer to bt + Bd + bdt as "bias," with bt representing a common time trend in the bias across districts, Bd representing a district-specific bias that is constant over time, and bdt representing the district- and time- specific deviation from the common time trend and constant district-specific bias. Biases here are due primarily to measurement error from the use of differ- ent concepts and measurements between the ACS and the CCD. We treat t and bt as fixed effects (nonrandom) and the remaining terms as random effects. Hence, Md, mdt, Bd, bdt, and edt are assumed to be zero mean random processes, with the following conditions on the theoretical variances and covariances: Md and Bd are correlated with each other but uncorrelated with mdt and gdt = bdt + edt. Both mdt and gdt are first-order autoregressive (AR(1)) processes, and their correlation with each other also has AR(1) form. All three AR(1) models have the same autoregressive coefficient.2 1As discussed in Chapters 2-4, administrative estimates are also subject to error. 2In SAS, this is called the UN@AR(1) covariance structure. Although preliminary investi- gations did indicate similar, weak correlations for mdt and gdt and weak cross-correlations, the assumption of common autoregressive parameters is primarily for simplicity. In particu- lar, it allows use of a built-in covariance structure, UN@AR(1), in SAS Proc Mixed.

OCR for page 316
322 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS TABLE F-4 Average Differences Between ACS 3-Year Estimates of Enrollment and Various CCD Estimates Large and Medium Districts Estimand 2005-2007 2006-2008 2007-2009 Very High FRPL Difference from CCD for 1 SY 1,438 1,529 1,276 As percentage of 1-year CCD 7 8 8 Difference from CCD 3-year average 1,183 1,290 1,013 As percentage of CCD 3-year average 6 7 6 High FRPL Districts Difference from CCD for 1 SY 133 13 13 As percentage of 1-year CCD 1 0 0 Difference from CCD 3-year average 118 85 80 As percentage of CCD 3-year average 1 1 1 Low to Moderate FRPL Districts Difference from CCD for 1 SY 484 383 439 As percentage of 1-year CCD 4 3 4 Difference from CCD 3-year average 347 371 428 As percentage of CCD 3-year average 3 3 4 NOTES: The ACS 3-year estimates are compared with (1) CCD estimates for the most recent school year that overlaps the reference period of the ACS estimates (so ACS estimates for 2005-2007 are compared with CCD estimates for 2007-2008) and (2) 3-year averages of CCD estimates (so ACS estimates for 2005-2007 are compared with the average of CCD estimates TABLE F-5 Average Differences Between ACS 1-Year Estimates of Enrollment and CCD Estimates Estimand 2005 2006 2007 2008 2009 Very High FRPL Districts Difference from CCD 3,149 3,941 4,628 5,057 4,418 As percentage of CCD 7 9 10 11 12 High FRPL Districts Difference from CCD 184 211 297 111 131 As percentage of CCD 1 1 1 0 0 Low to Moderate FRPL Districts Difference from CCD 767 1,295 1,554 1,650 1,839 As percentage of CCD 3 5 6 6 7 NOTES: Calendar year ACS estimates are compared with the CCD estimates for the most re- cent school year that overlaps the calendar year of the ACS. For example, the ACS estimates for 2009 are compared with the CCD estimates for 2009-2010. ACS = American Community Survey; CCD = Common Core of Data; FPRL = free or reduced-price lunch. SOURCE: Prepared by the panel.

OCR for page 316
APPENDIX F 323 Large Districts Medium Districts 2005-2007 2006-2008 2007-2009 2005-2007 2006-2008 2007-2009 3,816 4,078 3,376 106 147 148 8 9 9 2 3 3 3,122 3,428 2,667 88 101 113 7 8 7 2 2 2 208 60 46 100 44 38 1 0 0 2 1 1 239 115 103 64 71 69 1 0 0 1 1 1 1,054 811 1,005 225 204 206 4 3 4 4 4 4 673 736 901 190 199 214 3 3 3 4 4 4 for 2005-2006, 2006-2007, and 2007-2008). ACS = American Community Survey; CCD = Com- mon Core of Data; FPRL = free or reduced-price lunch; SY = school year. SOURCE: Prepared by the panel. TABLE F-6 Average Differences Between ACS 5-Year Estimates and CCD Estimates for Low to Moderate FRPL Districts All Large Medium Small Districts Districts Districts Districts Estimand (5,255) (354) (859) (4,042) Percentage free 4.7 7.1 6.1 4.1 Percentage reduced price 2.3 2.1 1.8 2.4 Percentage free or reduced price 2.4 5.0 4.3 1.7 BRR, $ 0.06 0.12 0.11 0.05 NOTES: The ACS estimates for 2005-2009 are compared with CCD estimates for the most recent school year that overlaps the reference period of the ACS estimates, namely school year 2009-2010. ACS = American Community Survey; BRR = blended reimbursement rate; CCD = Common Core of Data; FPRL = free or reduced-price lunch. SOURCE: Prepared by the panel.

OCR for page 316
324 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS TABLE F-7 Average Differences Between ACS 3-Year Estimates and CCD Estimates for Low to Moderate FRPL Districts Large and Medium Districts 2005-2007 2006-2008 2007-2009 Estimand (1,001) (1,117) (1,213) Percentage free 3.2 4.4 6.2 Percentage reduced price 1.5 1.2 1.4 Percentage free or reduced price 1.7 3.2 4.8 BRR, $ 0.05 0.08 0.12 NOTES: The ACS estimates for a 3-year period are compared with CCD estimates for the most recent school year that overlaps the reference period of the ACS estimates. For example, ACS estimates for 2005-2007 are compared with CCD estimates for school year 2007-2008. ACS = American Community Survey; BRR = blended reimbursement rate; CCD = Common Core of Data; FPRL = free or reduced-price lunch. SOURCE: Prepared by the panel. TABLE F-8 Average Differences Between ACS 1-Year Estimates and CCD Estimates for Low to Moderate FRPL Districts 2005 2006 2007 2008 2009 Estimand (295) (311) (313) (330) (354) Percentage free 3.3 3.4 4.8 5.3 5.3 Percentage reduced price 1.5 0.9 1.2 1.0 1.1 Percentage free or reduced price 1.8 2.5 3.6 4.3 4.2 BRR, $ 0.05 0.06 0.09 0.10 0.10 NOTES: The ACS estimates are compared with the CCD estimates for the most recent school year that overlaps the reference period of the ACS estimates. For example, ACS estimates for 2005 are compared with CCD estimates for 2005-2006. ACS = American Community Survey; BRR = blended reimbursement rate; CCD = Common Core of Data; FPRL = free or reduced-price lunch. SOURCE: Prepared by the panel. We constructed a data set with four variables: Y (either Cdt or Adt Cdt); Method (0 for Cdt and 1 for Adt i); District (1-393); and Time (1-5). The model is fitted in SAS using Proc Mixed.3 Box F-1 displays the SAS code, and Boxes F-2 through F-7 display the SAS output. 3Although fitting with Proc Mixed maximizes a Gaussian likelihood, this does not require that the error processes be jointly normally distributed. The residuals--CCD (estimated dis- trict effect) and ACSCCD (estimated district effect)--do tend to be symmetric and strongly unimodal, but with evidence of heavier tails than normal. Without normality of the error processes, Proc Mixed still produces sensible estimates of mean, variance, and covariance parameters, comparable to method-of-moments estimates. This is why the fitted model is able to reproduce empirical variances, such as variances of 1-year changes.

OCR for page 316
APPENDIX F 325 Large Districts Medium Districts 2005-2007 2006-2008 2007-2009 2005-2007 2006-2008 2007-2009 (313) (330) (354) (688) (787) (859) 3.9 5.2 6.8 2.9 4.0 5.9 1.7 1.4 1.7 1.4 1.2 1.2 2.2 3.9 5.1 1.5 2.9 4.7 0.06 0.10 0.13 0.04 0.07 0.12 BOX F-1 SAS Code for Analysis of Variability Proc mixed data = school; class District Method Time; model Y = Method Time Method * Time; random Method /subject = District type = un ggcorr; repeated Method Time /subject = District type = UN@AR(1) rrcorr; lsmeans Method * Time; SOURCE: Prepared by the panel. Box F-7 displays the least-squares means for Method*Time. These are the estimates of t for the 5 years, followed by estimates of bt for the 5 years. The 2 2 estimated G matrix in Box F-5 is the covariance matrix of (Md ,Bd). The estimated autocovariance function for mdt is given by 0.01032 * (0.1704)|h|. The estimated autocovariance function for gdt is given by 0.02878 * (0.1704)|h|, and the estimated cross-covariance function between mdt and gdt is given by 0.00944 * (0.1704)|h|. These are the values that fill out the 10 10 covariance matrix R shown in Box F-3. The vari- ance of gdt includes the design variance, but this is not used in building the model. Assumptions about the sampling error and its design variance are introduced below to extrapolate results from large districts to medium and small districts. Table F-9 shows variances of 1-year changes computed in the absence of a global (independent of district) time trend for large districts only. Model variances come from the SAS fit of the mixed model with UN@AR(1) covariance structure. Empirical variances are computed using

OCR for page 316
326 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS BOX F-2 SAS Proc Mixed Output: The Mixed Procedure Model Information Data Set WORK.SCHOOL Dependent Variable Y Covariance Structures Unstructured, Unstructured @ Autoregressive Subject Effects District, District Estimation Method REML Residual Variance Method None Fixed Effects SE Method Model-Based Degrees of Freedom Method Containment Class Level Information Class Levels Values District 393 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 . . . 383 384 385 386 387 388 389 390 391 392 393 Method 2 01 Time 5 12345 Dimensions Covariance Parameters 7 Columns in X 18 Columns in Z Per Subject 2 Subjects 393 Max Obs per Subject 10 Number of Observations Number of Observations Read 3930 Number of Observations Used 3930 Number of Observations Not Used 0 Iteration Evaluations 2 Res Log Like Criterion 0 1 825.49213626 1 2 3590.79275559 0.00012752 2 1 3591.50965749 0.00000058 3 1 3591.51280315 0.00000000 Convergence criteria met. SOURCE: Prepared by the panel.

OCR for page 316
330 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS BOX F-6 SAS Proc Mixed Output, Fit Statistics 2 Res Log Likelihood 3591.5 AIC (smaller is better) 3577.5 AICC (smaller is better) 3577.5 BIC (smaller is better) 3549.7 Null Model Likelihood Ratio Test DFChi-Square Pr>ChiSq 6 4417.00 <.0001 Type 3 Tests of Fixed Effects Effect DFDFF ValuePr> F Method 1 784 8932.18<.0001 Time 4 3136 43.37<.0001 Method*Time 4 3136 62.50<.0001 SOURCE: Prepared by the panel. BOX F-7 SAS Proc Mixed Output, Least Squares Means Least Squares Means Effect Method Time Estimate Error DF t Value Pr> |t| Method*Time 0 1 1.5894 0.01521 3136 104.49<.0001 Method*Time 0 2 1.6047 0.01521 3136 105.50<.0001 Method*Time 0 3 1.6334 0.01521 3136 107.38<.0001 Method*Time 0 4 1.6826 0.01521 3136 110.62<.0001 Method*Time 0 5 1.7617 0.01521 3136 115.82<.0001 Method*Time 1 1 0.2054 0.01144 3136 17.96<.0001 Method*Time 1 2 0.2216 0.01144 3136 19.37<.0001 Method*Time 1 3 0.2681 0.01144 3136 23.44<.0001 Method*Time 1 4 0.2940 0.01144 3136 25.70<.0001 Method*Time 1 5 0.2787 0.01144 3136 24.37<.0001

OCR for page 316
APPENDIX F 331 TABLE F-9 Model Versus Empirical Estimates for Variances of Year- to-Year Changes, Large Districts Only Standard Deviation Standard Relative to Average Large Districts Variance ($2) Deviation ($) CCD BRR (%) CCD Empirical 0.016 0.125 7.6 CCD Model 0.017 0.131 7.9 ACS1 Empirical 0.035 0.187 11.3 ACS1 Model 0.049 0.222 13.4 ACS3 Empirical 0.005 0.071 4.3 ACS3 Model 0.006 0.081 4.9 ACS5 Empirical NA NA NA ACS5 Model 0.002 0.049 2.9 Model Empirical 0.014 0.12 7.3 Model Model NA NA NA NOTES: The average value of the BRR computed from CCD data for large districts was $1.65. The ratio of the standard deviation to this value is a coefficient of variation. ACS = American Community Survey; ACS1 = ACS 1-year estimates; ACS3 = ACS 3-year estimates; ACS5 = ACS 5-year estimates; BRR = blended reimbursement rate; CCD = Common Core of Data; NA = not applicable. SOURCE: Prepared by the panel. the following sequence of steps. First, for each available pair of consecu- tive years, compute the year-to-year difference for each district. Second, for each available pair of consecutive years, compute the empirical vari- ance (across all 393 large districts) using the set of differences computed in the first step. Finally, average the empirical variances across all avail- able pairs of years. This analysis is not affected by any time trend in the data because any trend appears in the difference for each district as trend(t + 1) trend(t), which is constant across districts for a given con- secutive pair of years. That constant does not affect the empirical variance for each consecutive pair of years in the second step, so it does not affect the average empirical variance across all pairs of years in the final step. Comparison of empirical and model variances shows that the model does a fairly good job of capturing the variance of 1-year change in CCD and of 1-year change in ACS-CCD. There are, however, some discrep- ancies between the empirical and model variances for the 1-year ACS estimates. Nonetheless, the standard deviations (19 cents empirical vs.

OCR for page 316
332 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS 22 cents model) are not all that different from a practical point of view. Therefore, the panel believes the model can provide sensible quantita- tive guidance, particularly for comparing estimators, even if the specific model predictions should be treated with caution. Further research could develop and validate more refined models. Table F-10 shows the same results on variances of 1-year changes for medium districts only. Empirical variances are computed as described above. Model variances are computed from the model fitted to the large districts only, extrapolated to medium districts using the extrapo- lated design variance, as described below, at the median enrollment for medium districts. There are 835 medium districts used in this analysis, with median enrollment of 4,797 students. For medium districts, the CCD empirical variance is very similar, but not identical, to that for large districts. The CCD model variance is derived from the model fitted for large districts, which does not depend on enrollment. Therefore, the CCD model row is exactly the same for medium and large districts. TABLE F-10 Model Versus Empirical Estimates for Variances of Year-to-Year Changes, Medium Districts Only Standard Deviation Standard Relative to Average Medium Districts Variance ($2) Deviation ($) CCD BRR (%) CCD Empirical 0.017 0.130 7.9 CCD Model 0.017 0.131 7.9 ACS1 Empirical NA NA NA ACS1 Model 0.110 0.332 20.1 ACS3 Empirical 0.017 0.130 7.9 ACS3 Model 0.013 0.115 7.0 ACS5 Empirical NA NA NA ACS5 Model 0.005 0.069 4.2 Model Empirical 0.026 0.160 9.7 Model Model NA NA NA NOTES: The average value of the BRR computed from CCD data for medium districts was $1.65. The ratio of the standard deviation to this value is a coefficient of variation. ACS = American Community Survey; BRR = blended reimbursement rate; CCD = Common Core of Data; NA = not applicable. SOURCE: Prepared by the panel.

OCR for page 316
APPENDIX F 333 Table F-11 shows the same results on variances of 1-year changes for small districts only. Empirical variances are computed as for large and medium districts. Model variances are computed from the model fitted to the large districts only, extrapolated to small districts using the extrapo- lated design variance at the median enrollment for small districts. There are 3,989 small districts used in this analysis, with median enrollment of 627 students. As expected, the CCD empirical variance is much larger for small dis- tricts than for medium or large districts. The CCD model line again does not depend on enrollment, so it looks the same as for medium or large districts, except that the average CCD BRR has changed very slightly; thus the percentage changes slightly. The panel considered fitting a model for 3-year estimates for either large or medium districts (or both combined) but decided that it would be difficult to fit such a model given time constraints. This is because the 3-year estimates are correlated across years because of not only the tem- TABLE F-11 Model Versus Empirical Estimates for Variances of Year-to-Year Changes, Small Districts Only Standard Deviation Standard Relative to Average Small Districts Variance ($2) Deviation ($) CCD BRR (%) CCD Empirical 0.028 0.168 10.3 CCD Model 0.017 0.131 8.0 ACS1 Empirical NA NA NA ACS1 Model 0.569 0.755 46.1 ACS3 Empirical NA NA NA ACS3 Model 0.064 0.254 15.5 ACS5 Empirical NA NA NA ACS5 Model 0.023 0.152 9.3 Model Empirical 0.017 0.132 8.0 Model Model NA NA NA NOTES: The average value of the BRR computed from CCD data for small districts was $1.64. The ratio of the standard deviation to this value is a coefficient of variation. ACS = American Community Survey; BRR = blended reimbursement rate; CCD = Common Core of Data; NA = not applicable. SOURCE: Prepared by the panel.

OCR for page 316
334 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS poral correlation of the BRR values but also the moving average of the sampling error. Further research could be undertaken to fit such a model. The analysis above for medium and small districts relies on extrap- olating results from the model fitted to data for large districts only. Extrapolating the fitted model as a function of enrollment requires model ing the design variance for 1-year ACS estimates in medium and small districts (which could be derived at the Census Bureau but may not be able to be released under current disclosure rules). Suppose that ACS sample sizes are constant from year to year within a district, and the design variance s2dt s2d depends on the district but is constant from year to year. Given the design of the ACS, it is reasonable to assume that: Sampling error autocovariances are zero: d 2 if h = 0 Cov ( edt , ed ,t + h = Cov ( ed ,t + h , edt = ) ) 0 if h 0 where s2d is the sampling variance of the 1-year ACS estimator for district d. All cross-covariances with sampling error are zero. The design variance for 3-year ACS estimates is one-third of the design variance for 1-year ACS estimates, and the design variance for 5-year ACS estimates is one-fifth of the design variance for 1-year ACS estimates. The design variance within a district is determined largely by sample size, which is, in turn, highly correlated with enrollment. Figure F-1 displays a scatter plot of data and the regression model fit for log(design variance) as a function of log(enrollment) for the 1-year ACS estimates in large districts. The fitted linear relationship is given by log(design variance) = 4.5 0.9 log(enrollment). We choose log(enrollment) = 9.8 as a typical value for a large district because it is close to log(median(enrollment)) = 9.84. If we plug this value into the linear relationship above and transform to the design standard deviation, we get 0.1153, which is very close to the average design stan- dard deviation across districts and years, 0.1146. Next, we take the SAS fit, which models gdt = bdt + edt as AR(1), and approximate the fitted AR(1) by AR(1) + uncorrelated noise, where the noise has variance equal to the "typical value" .0133 = (0.1153)2. The resulting model for bdt ~ AR(1) has process variance 0.01548 and autoregressive parameter 0.3168. Finally, taking the model for bdt as fixed, let the variance for edt depend on enroll- ment through the above linear relationship. Tables F-10 and F-11, dis- cussed above, were constructed using this analysis, with enrollment taken

OCR for page 316
APPENDIX F 335 2 3 4 log(design variance) 5 6 7 8 9 10 11 12 13 log(enrollment) FIGURE F-1 Regression fit of log(design variance) versus log(enrollment). SOURCE: Prepared by the panel. FIGF-1.eps to be the observed median enrollment for medium districts and for small districts, respectively. The standard deviation (SD) and coefficient of variation (CV) (relative to mean (CCD) = $1.65) of 1-year change in 5-year estimates for various enrollments are shown in Table F-12. There are real differences in the amount of noise under which districts normally operate with traditional application and certification proce- dures. Small districts combined have a percentage standard deviation (CV) of 10.3 percent for CCD 1-year changes, but those with less than the median enrollment have a CV of 11.6 percent, while those with more than the median enrollment have a CV of 8.7 percent. These are compa- rable to the ACS5 (modeled) CVs at enrollments of 400-800, according to Table F-12, which is the same as Table 4-8 in Chapter 4. Figure F-2, which is the same as Figure 4-5 in Chapter 4, displays a transformation of the data in Table F-12. For a given district, the point (1/enrollment, CV2) can be plotted on the figure. If the plotted point is above the curve, the dis- trict currently experiences more variability in its administrative estimates

OCR for page 316
336 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS TABLE F-12 Intertemporal Variability of ACS 5-Year Estimates, by Enrollment Variability of 1-Year Change in ACS 5-Year Estimates of Blended Reimbursement Rates Coefficient of Variation (%) Enrollment Standard Deviation ($) (relative to BRR of $1.65) 100 0.34 20.5 200 0.25 15.1 400 0.18 11.2 800 0.14 8.3 1,600 0.10 6.3 3,200 0.08 4.8 6,400 0.06 3.8 12,800 0.05 3.2 SOURCE: Prepared by the panel. 400 300 CV2 200 100 0 0.000 0.002 0.004 0.006 0.008 0.010 1/enrollment FIGURE F-2 Squared coefficient of variation of year-to-year change in ACS 5-year estimate of BRR versus inverse of enrollment. NOTES: ACS = American Community Survey; BRR = blended reimbursement FIGF-2.eps rate; CV = coefficient of variation. SOURCE: Prepared by the panel.

OCR for page 316
APPENDIX F 337 than it would if it used ACS 5-year estimates (at least according to the model and ignoring timeliness bias). In this situation, the district might find use of the ACS 5-year estimates to be acceptable. On the other hand, if its plotted point is below the curve, the district currently experiences less variability in its administrative estimates than it would with the ACS 5-year estimates, and might find the latter unacceptably variable for use in determining reimbursements under the ACS Eligibility Option (AEO). Table F-13 shows standard deviations, biases, and root mean squared errors (RMSEs) for ACS 1-year, 3-year, and 5-year estimators, with and without a 2-year lag (reflecting the lag in the availability of ACS estimates for use in establishing claiming rates). For large districts, these values are computed in two ways: (1) using the AR(1) model originally fitted via SAS for gdt and (2) using the AR(1) + noise model for gdt. The latter model makes results consistent with the analysis for medium and small districts, all of which use the AR(1) + noise model. The other difference in the AR(1) analysis for the large districts is that bt is estimated from the data (see Box F-7) and incorporated in the bias computations, while in the AR(1) + noise analysis, it is assumed to be constant over time (or zero, without loss of generality). Again this is done to maintain consistency with the analysis for medium and small districts, for which estimation of bt from the data is not possible. TABLE F-13 Standard Deviation, Bias, and RMSE for ACS 1-Year, 3-Year, and 5-Year Estimates at Lags of 0 and 2 Years District ACS1, ACS1, ACS3, ACS3, ACS5, ACS5, Size no lag lag 2 no lag lag 2 no lag lag 2 Large SD (2) 0.170 0.221 0.135 0.137 0.124 0.126 SD (1) 0.169 0.221 0.134 0.137 0.123 0.125 Bias (2) 0.000 0.128 0.069 0.153 0.107 NA Bias (1) 0.025 0.143 0.096 0.131 0.107 RMSE (2) 0.170 0.256 0.152 0.205 0.164 NA RMSE (1) 0.172 0.263 0.165 0.189 0.163 Medium SD 0.243 0.282 0.168 0.170 0.147 0.148 Bias 0.000 0.115 0.062 0.130 0.092 NA RMSE 0.243 0.304 0.179 0.214 0.173 NA Small SD 0.537 0.556 0.324 0.325 0.260 0.261 Bias 0.000 0.104 0.059 0.107 0.079 NA RMSE 0.537 0.565 0.329 0.342 0.271 NA NOTES: The results for large districts were obtained using two methods: (1) using the AR(1) model for gdt and (2) using the AR(1) plus noise model for gdt. ACS = American Community Survey; NA = not applicable; RMSE = root mean squared error; SD = standard deviation. SOURCE: Prepared by the panel.

OCR for page 316
338 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS The bias and RMSE results reflect the specific mt estimated for the particular 5-year time window covered by the estimates available to the panel, separately for each district size class. For any size class, the esti- mate of mt is simply the year t average CCD BRR across all districts. For large districts, these estimated mt values are given in Box F-7. PART 3: MODEL-BASED PREDICTION OF SYSTEMATIC DIFFERENCES BETWEEN ACS ESTIMATES AND CCD ESTIMATES FOR BRR This part of the appendix describes the results of the panel's modeling of the differences between ACS estimates and CCD estimates for the BRR. The analysis was limited to very high FRPL districts with both 5-year ACS estimates and CCD estimates for 2009-2010 in the panel's evaluation data set prog09.merged.fns. To eliminate outliers that could adversely impact regression results, we excluded any districts that had either a percentage certified for free meals of less than 10 percent or a percentage certified for free or reduced-price meals of less than 20 percent. Districts with missing data for potential predictor variables were also excluded. The ACS estimate used in the analysis is the 5-year ACS estimate for the BRR (denoted ACS5 BRR below). The CCD estimate used is the BRR based on certification data in the 2009-2010 CCD (denoted CCD0910 BRR below). The dependent variable used in the analysis is the difference between ACS5 BRR and CCD0910 BRR divided by the standard error of ACS5 BRR. This variable is regressed on a variety of predictor variables from the 2009-2010 CCD as described below. Table F-14 provides regres- sion results for a variety of alternative models. In the table, p is the number of covariates in a model, and FOI stands for "first-order interactions." The "Additive" model is the most basic model, with no interactions or quadratic terms, and the "FOI, No Factor Interaction" model includes interactions among continuous covariates but not with or among the categorical covariates. Box F-8 lists the covariates used in the modeling. The results of our exploratory analyses of whether a global predictive model could be used for adjusting for differences between ACS and administrative estimates are discussed in Chapter 4.

OCR for page 316
TABLE F-14 Results for Various Models Predicting Differences Between ACS 5-Year Estimates for 2005-2009 and CCD Estimates for 2009-2010 Without FRPL Covariates With FRPL Covariates (1,433 districts) (1,366 districts) Model Covariates p R2 Adj. R2 RMSE AIC p R2 Adj. R2 RMSE AIC Additive 73 0.420 0.389 1.068 6009 81 0.628 0.604 0.806 6408 FOI, No Factor 159 0.572 0.519 0.917 6273 258 0.779 0.727 0.621 6765 Interactions FOI, No State/Locale 172 0.579 0.522 0.910 6271 334 0.804 0.740 0.585 6777 Interactions FOI, No State 317 0.650 0.550 0.830 6245 542 0.855 0.760 0.503 6774 Interaction FOI, All Variables 717 0.782 0.563 0.655 6124 981 0.923 0.728 0.366 6768 FOI and Quadratic 726 0.784 0.561 0.652 6116 996 0.926 0.726 0.359 6784 ACS5 BRR - CCD0910 BRR NOTES: The basic model is ~ covariates. AIC = Akaike Information Criterion; ACS = American Community Survey; SE ( ACS5 BRR ) FOI = first-order interactions; NA = not applicable; RMSE = root mean squared error. SOURCE: Prepared by the panel. 339

OCR for page 316
340 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS BOX F-8 Covariates Used in Regression Analysis The covariates in the "Without FRPL" models are as follows: 1.C0910_Num_Enroll (number of enrolled students) 2.C0910_Pct_inNonRegSch (percentage of students in nonregular--special education, vocational education, or alternative--schools) 3.C0910_Pct_inChartSch (percentage of students in charter schools) 4.C0910_Pct_inChartNonRegSch (percentage of students in charter or non- regular schools) 5.C0910_Pct_inChartMagSch (percentage of students in charter or magnet schools) 6. C0910_Pct_inChartMagNonRegSch (percentage of students in charter, magnet, or nonregular schools) 7. C0910_Pct_AIAN (percentage of students who are American Indian or Alaska Native) 8.C0910_Pct_AsianHNPI (percentage of students who are Asian, Hawaiian Native, or Pacific Islander) 9.C0910_Pct_Hispanic (percentage of students who are Hispanic) 10.C0910_Pct_Black (percentage of students who are black) 11.C0910_Pct_White (percentage of students who are white) 12.C0910_ChartDistance (index measuring distance to nearby charter-only districts) 13.C0910_ChartDistance_Enroll (index measuring distance to nearby charter- only districts, weighted by charter enrollment) 14.C0910_ChartDistance_Enroll_Rel (index measuring distance to nearby charter-only districts, weighted by charter enrollment relative to district's enrollment) 15.C_State (state) 16.C_Locale_Type (type of locale as defined in CCD) The "With FRPL" models add the following covariates: 17.C0910_Pct_Free (percentage of students certified for free meals) 18.C0910_Pct_Reduced (percentage of students certified for reduced-price meals) 19.C0910_Num_Free (number of students certified for free meals) 20.C0910_Num_Reduced (number of students certified for reduced-price meals) 21.C0910_ChartDistance_FRPL (index measuring distance to nearby charter- only districts, weighted by number of charter students certified for free or reduced-price meals) 22.C0910_ChartDistance_FRPL_Rel (index measuring distance to nearby charter-only districts, weighted by number of charter students certified for free or reduced-price meals relative to number in district) 23.C0910_Need (categorical variable for whether percentage of students certi- fied for free or reduced-price meals is < 50, 50-74, or 75) 24.C0910_CCDSchools_CharterCode (categorical variable for whether all, some, or none of the schools in district are charter schools) SOURCE: Prepared by the panel.