Using American Community Survey Data to Expand Access to the School Meals Programs (2012)

Appendix F

Additional Information About the Panel’s Analyses

This appendix provides additional information about and results from the analyses conducted by the panel, as described in Chapter 4. Included are three parts. The first complements the comparisons discussed in Chapter 4 with some additional tables concerning the differences between American Community Survey (ACS) estimates and administrative estimates based on the National Center for Education Statistics’(NCES) Common Core of Data (CCD). The second part describes the model used to assess stability over time and provides detailed model results. The third part describes the panel’s exploration of the use of global regression models for predicting differences between ACS and CCD estimates for the blended reimbursement rate (BRR) using a variety of covariates from the CCD.

PART 1: COMPARISONS OF ACS ESTIMATES AND ESTIMATES BASED ON ADMINISTRATIVE DATA

Tables F-1 and F-2 display the differences between ACS multiyear averages and CCD multiyear averages computed over roughly the same time periods. Table F-1 displays comparisons for 5-year estimates and Table F-2 for 3-year estimates. These tables present differences by district size (small, medium, and large), and free or reduced-price lunch (FRPL) category (very high, high, and low to moderate) for percentage eligible for free meals, percentage eligible for reduced-price meals, percentage eligible for free or reduced-price meals, and the BRR.

TABLE F-1 Average Differences Between ACS 5-Year Estimates and 5-Year Averages of CCD Estimates

NOTES: The ACS 5-year estimates (for 2005-2009) are compared with the average of CCD estimates for 2005-2006, 2006-2007, 2007-2008, 2008-2009, and 2009-2010. ACS = American Community Survey; BRR = blended reimbursement rate; CCD = Common Core of Data; FPRL = free or reduced-price lunch.
SOURCE: Prepared by the panel.

The purpose of this comparison is to illustrate the differences that exist when the reference periods of the ACS and administrative estimates are as similar as possible. These tables display the same patterns as those observed in Chapter 4, where the administrative estimates pertain to the most recent year of the reference period for the ACS estimates. Namely, the ACS understates percentage free, percentage free or reduced price, and the BRR and overstates percentage reduced price. The differences are substantial in very high FRPL districts and are least pronounced in low to moderate FRPL districts; high FRPL districts fall in between. Over all districts, the BRR is understated by the 5-year ACS by 35 cents for very high FRPL districts and 12 cents for high FRPL districts, and is overstated by 1 cent in low to moderate FRPL districts.

Chapter 4 highlights the systematic differences between ACS and CCD estimates for eligibility percentages and the BRR. The following tables compare enrollment estimates from the two sources. Tables F-3 and F-4 illustrate the differences between ACS multiyear estimates and CCD

TABLE F-2 Average Differences Between ACS 3-Year Estimates and 3-Year Averages of CCD Estimates

NOTES: The ACS 3-year estimates are compared with 3-year averages of CCD estimates. For example, the ACS estimates for 2005-2007 are compared with the average of CCD estimates for 2005-2006, 2006-2007, and 2007-2008. ACS = American Community Survey; BRR = blended reimbursement rate; CCD = Common Core of Data; FPRL = free or reduced-price lunch.
SOURCE: Prepared by the panel.

multiyear average estimates computed over the same time periods as the ACS estimates, as well as the differences between the ACS multiyear estimates and the CCD estimates for the most recent school year that overlaps the ACS reference period. (For the latter, the ACS estimate for 2005-2009 is compared with the CCD estimate for 2009-2010, and the ACS estimate for 2007-2009 is also compared with the CCD estimate for 2009-2010.)

In addition to sampling error in the ACS estimates and various other errors in both the ACS and administrative estimates, enrollment estimates may differ because school district boundaries are different in different years. All of the ACS estimates are based on the school district boundaries recorded in the Census Bureau’s Topologically Integrated Geographic Encoding and Referencing (TIGER) database for 2009-2010 and data reflecting the number of students that resided within those boundaries at some time during a calendar year. On the other hand, the CCD data reflect the district’s enrollment as of October of a school year based on the boundaries for that year. School choice is another reason why enrollment estimates may differ. Children who live in the catchment area of a school

district and attend public school may not attend a school associated with the local public school district; some may attend an independent charter school, for example. These differences are discussed more fully in Chapter 4. Differences in the inclusion of prekindergarten students might also contribute to differences in enrollment estimates.

The differences shown in Table F-3 for the 5-year ACS estimates tend to be relatively small, but are largest (11 percent) for large very high FRPL districts (when compared with CCD estimates for 2009-2010). Other categories of districts have differences of 4 percent or less. The 5-year ACS estimates tend to overstate enrollment in very high FRPL districts and to understate enrollment in low to moderate FRPL districts. Similar patterns are illustrated in Table F-4, where small districts are not included because there are no 3-year ACS estimates for them.

Table F-5 shows the average differences between ACS 1-year estimates for enrollment and the CCD estimates for enrollment for each of 5 years. The ACS calendar-year estimates are compared with the CCD school year estimates for the most recent school year that overlaps the

TABLE F-3 Average Differences Between ACS 5-Year Estimates of Enrollment and Various CCD Estimates

NOTES: The ACS 5-year estimates are compared with (1) CCD estimates for the most recent school year that overlaps the reference period of the ACS estimates (so the ACS estimates for 2005-2009 are compared with CCD estimates for 2009-2010) and (2) 5-year averages of CCD estimates (so the ACS estimates for 2005-2009 are compared with the average of CCD estimates for 2005-2006, 2006-2007, 2007-2008, 2008-2009, and 2009-2010). ACS = American Community Survey; CCD = Common Core of Data; FPRL = free or reduced-price lunch.

SOURCE: Prepared by the panel.

calendar year. (Hence, the ACS estimate for 2009 is compared with the CCD estimate for 2009-2010.) These results are only for large districts that have ACS 1-year estimates. The percentage differences are again largest for the very high FRPL districts (averaging almost 10 percent) and lowest for the low to moderate FRPL districts (averaging about 5 percent); the high FRPL districts average –.3 percent. Here the average differences appear to be increasing in magnitude over time for both the very high and low to moderate FRPL categories.

Tables F-6 through F-8 display the average differences between various ACS estimates (5-year, 3-year, and 1-year) and the CCD estimate for the most recent school year that overlaps the reference period of the ACS estimate for low to moderate FRPL districts. These tables complement Tables 4-1, 4-2, and 4-3 in Chapter 4, which present results for the very high and high FRPL districts. Each table shows average differences for percentage free, percentage reduced price, percentage free or reduced price, and the BRR. Tables F-6, F-7, and F-8 show the same patterns of differences

as the tables in Chapter 4, but the magnitudes of the differences are much smaller.

PART 2: MODELING OF VARIATION

Let A_dt denote the 1-year ACS estimate of the true BRR, C_dt, for school district d in year t, where C_dt is the BRR value as computed from the CCD.¹ We write

C_dt = µ_t + M_d + m_dt

where µ_t is a common time trend across districts, M_d is a district-specific deviation that is constant over time, and m_dt is the district- and time-specific deviation from the common time trend and constant district deviation. We write

A_dt = C_dt + β_t + B_d + b_dt + e_dt

where e_dt is sampling error with known variance σ²_dt, and β_t + B_d + b_dt represents the difference between the CCD and ACS estimates after sampling error is removed. Because the CCD is treated as the gold standard in this discussion, we refer to β_t + B_d + b_dt as “bias,” with β_t representing a common time trend in the bias across districts, B_d representing a district-specific bias that is constant over time, and b_dt representing the district- and time- specific deviation from the common time trend and constant district-specific bias. Biases here are due primarily to measurement error from the use of different concepts and measurements between the ACS and the CCD.

We treat μ_t and β_t as fixed effects (nonrandom) and the remaining terms as random effects. Hence, M_d, m_dt, B_d, b_dt, and e_dt are assumed to be zero mean random processes, with the following conditions on the theoretical variances and covariances:

• M_d and B_d are correlated with each other but uncorrelated with m_dt and g_dt = b_dt + e_dt.
• Both m_dt and g_dt are first-order autoregressive (AR(1)) processes, and their correlation with each other also has AR(1) form. All three AR(1) models have the same autoregressive coefficient.²

____________

¹ As discussed in Chapters 2-4, administrative estimates are also subject to error.

² In SAS, this is called the UN@AR(1) covariance structure. Although preliminary investigations did indicate similar, weak correlations for m_dt and g_dt and weak cross-correlations, the assumption of common autoregressive parameters is primarily for simplicity. In particular, it allows use of a built-in covariance structure, UN@AR(1), in SAS Proc Mixed.

TABLE F-4 Average Differences Between ACS 3-Year Estimates of Enrollment and Various CCD Estimates

NOTES: The ACS 3-year estimates are compared with (1) CCD estimates for the most recent school year that overlaps the reference period of the ACS estimates (so ACS estimates for 2005-2007 are compared with CCD estimates for 2007-2008) and (2) 3-year averages of CCD estimates (so ACS estimates for 2005-2007 are compared with the average of CCD estimates

TABLE F-5 Average Differences Between ACS 1-Year Estimates of Enrollment and CCD Estimates

NOTES: Calendar year ACS estimates are compared with the CCD estimates for the most recent school year that overlaps the calendar year of the ACS. For example, the ACS estimates for 2009 are compared with the CCD estimates for 2009-2010. ACS = American Community Survey; CCD = Common Core of Data; FPRL = free or reduced-price lunch.

SOURCE: Prepared by the panel.

for 2005-2006, 2006-2007, and 2007-2008). ACS = American Community Survey; CCD = Common Core of Data; FPRL = free or reduced-price lunch; SY = school year.

SOURCE: Prepared by the panel.

TABLE F-6 Average Differences Between ACS 5-Year Estimates and CCD Estimates for Low to Moderate FRPL Districts All Large Medium Small Districts Districts Districts Districts Estimand (5,255) (354) (859) (4,042)

NOTES: The ACS estimates for 2005-2009 are compared with CCD estimates for the most recent school year that overlaps the reference period of the ACS estimates, namely school year 2009-2010. ACS = American Community Survey; BRR = blended reimbursement rate; CCD = Common Core of Data; FPRL = free or reduced-price lunch.

SOURCE: Prepared by the panel.

TABLE F-7 Average Differences Between ACS 3-Year Estimates and CCD Estimates for Low to Moderate FRPL Districts

NOTES: The ACS estimates for a 3-year period are compared with CCD estimates for the most recent school year that overlaps the reference period of the ACS estimates. For example, ACS estimates for 2005-2007 are compared with CCD estimates for school year 2007-2008. ACS = American Community Survey; BRR = blended reimbursement rate; CCD = Common Core of Data; FPRL = free or reduced-price lunch.

SOURCE: Prepared by the panel.

TABLE F-8 Average Differences Between ACS 1-Year Estimates and CCD Estimates for Low to Moderate FRPL Districts

NOTES: The ACS estimates are compared with the CCD estimates for the most recent school year that overlaps the reference period of the ACS estimates. For example, ACS estimates for 2005 are compared with CCD estimates for 2005-2006. ACS = American Community Survey; BRR = blended reimbursement rate; CCD = Common Core of Data; FPRL = free or reduced-price lunch.

SOURCE: Prepared by the panel.

We constructed a data set with four variables: Y (either C_dt or A_dt – C_dt); Method (0 for C_dt and 1 for A_dt – i); District (1-393); and Time (1-5). The model is fitted in SAS using Proc Mixed.³Box F-1 displays the SAS code, and Boxes F-2 through F-7 display the SAS output.

____________

³ Although fitting with Proc Mixed maximizes a Gaussian likelihood, this does not require that the error processes be jointly normally distributed. The residuals—CCD (estimated district effect) and ACS–CCD (estimated district effect)—do tend to be symmetric and strongly unimodal, but with evidence of heavier tails than normal. Without normality of the error processes, Proc Mixed still produces sensible estimates of mean, variance, and covariance parameters, comparable to method-of-moments estimates. This is why the fitted model is able to reproduce empirical variances, such as variances of 1-year changes.

Box F-7 displays the least-squares means for Method*Time. These are the estimates of μ_t for the 5 years, followed by estimates of ß_t for the 5 years. The 2 × 2 estimated G matrix in Box F-5 is the covariance matrix of (M_d, B_d). The estimated autocovariance function for m_dt is given by 0.01032 * (0.1704)^|h|. The estimated autocovariance function for gdt is given by 0.02878 * (0.1704)^|h|, and the estimated cross-covariance function between m_dt and g_dt is given by -0.00944 * (0.1704)^|h|. These are the values that fill out the 10 × 10 covariance matrix R shown in Box F-3. The variance of g_dt includes the design variance, but this is not used in building the model. Assumptions about the sampling error and its design variance are introduced below to extrapolate results from large districts to medium and small districts.

Table F-9 shows variances of 1-year changes computed in the absence of a global (independent of district) time trend for large districts only. Model variances come from the SAS fit of the mixed model with UN@AR(1) covariance structure. Empirical variances are computed using

TABLE F-9 Model Versus Empirical Estimates for Variances of Year-to-Year Changes, Large Districts Only

the following sequence of steps. First, for each available pair of consecutive years, compute the year-to-year difference for each district. Second, for each available pair of consecutive years, compute the empirical variance (across all 393 large districts) using the set of differences computed in the first step. Finally, average the empirical variances across all available pairs of years. This analysis is not affected by any time trend in the data because any trend appears in the difference for each district as trend(t + 1) – trend(t), which is constant across districts for a given consecutive pair of years. That constant does not affect the empirical variance for each consecutive pair of years in the second step, so it does not affect the average empirical variance across all pairs of years in the final step.

Comparison of empirical and model variances shows that the model does a fairly good job of capturing the variance of 1-year change in CCD and of 1-year change in ACS-CCD. There are, however, some discrepancies between the empirical and model variances for the 1-year ACS estimates. Nonetheless, the standard deviations (19 cents empirical vs.

22 cents model) are not all that different from a practical point of view. Therefore, the panel believes the model can provide sensible quantitative guidance, particularly for comparing estimators, even if the specific model predictions should be treated with caution. Further research could develop and validate more refined models.

Table F-10 shows the same results on variances of 1-year changes for medium districts only. Empirical variances are computed as described above. Model variances are computed from the model fitted to the large districts only, extrapolated to medium districts using the extrapolated design variance, as described below, at the median enrollment for medium districts. There are 835 medium districts used in this analysis, with median enrollment of 4,797 students. For medium districts, the CCD empirical variance is very similar, but not identical, to that for large districts. The CCD model variance is derived from the model fitted for large districts, which does not depend on enrollment. Therefore, the CCD model row is exactly the same for medium and large districts.

TABLE F-10 Model Versus Empirical Estimates for Variances of Year-to-Year Changes, Medium Districts Only

NOTES: The average value of the BRR computed from CCD data for medium districts was $1.65. The ratio of the standard deviation to this value is a coefficient of variation. ACS = American Community Survey; BRR = blended reimbursement rate; CCD = Common Core of Data; NA = not applicable.
SOURCE: Prepared by the panel.

Table F-11 shows the same results on variances of 1-year changes for small districts only. Empirical variances are computed as for large and medium districts. Model variances are computed from the model fitted to the large districts only, extrapolated to small districts using the extrapolated design variance at the median enrollment for small districts. There are 3,989 small districts used in this analysis, with median enrollment of 627 students.

As expected, the CCD empirical variance is much larger for small districts than for medium or large districts. The CCD model line again does not depend on enrollment, so it looks the same as for medium or large districts, except that the average CCD BRR has changed very slightly; thus the percentage changes slightly.

The panel considered fitting a model for 3-year estimates for either large or medium districts (or both combined) but decided that it would be difficult to fit such a model given time constraints. This is because the 3-year estimates are correlated across years because of not only the tem-

TABLE F-11 Model Versus Empirical Estimates for Variances of Year-to-Year Changes, Small Districts Only

poral correlation of the BRR values but also the moving average of the sampling error. Further research could be undertaken to fit such a model.

The analysis above for medium and small districts relies on extrapolating results from the model fitted to data for large districts only. Extrapolating the fitted model as a function of enrollment requires model ing the design variance for 1-year ACS estimates in medium and small districts (which could be derived at the Census Bureau but may not be able to be released under current disclosure rules). Suppose that ACS sample sizes are constant from year to year within a district, and the design variance σ²_dt = s²_d depends on the district but is constant from year to year. Given the design of the ACS, it is reasonable to assume that:

• Sampling error autocovariances are zero:

where σ²_d is the sampling variance of the 1-year ACS estimator for district d.

• All cross-covariances with sampling error are zero.

• The design variance for 3-year ACS estimates is one-third of the design variance for 1-year ACS estimates, and the design variance for 5-year ACS estimates is one-fifth of the design variance for 1-year ACS estimates.

The design variance within a district is determined largely by sample size, which is, in turn, highly correlated with enrollment. Figure F-1 displays a scatter plot of data and the regression model fit for log(design variance) as a function of log(enrollment) for the 1-year ACS estimates in large districts. The fitted linear relationship is given by log(design variance) = 4.5 - 0.9 log(enrollment).

We choose log(enrollment) = 9.8 as a typical value for a large district because it is close to log(median(enrollment)) = 9.84. If we plug this value into the linear relationship above and transform to the design standard deviation, we get 0.1153, which is very close to the average design standard deviation across districts and years, 0.1146. Next, we take the SAS fit, which models g_dt = b_dt + e_dt as AR(1), and approximate the fitted AR(1) by AR(1) + uncorrelated noise, where the noise has variance equal to the “typical value” .0133 = (0.1153)². The resulting model for b_dt ~ AR(1) has process variance 0.01548 and autoregressive parameter 0.3168. Finally, taking the model for b_dt as fixed, let the variance for e_dt depend on enrollment through the above linear relationship. Tables F-10 and F-11, discussed above, were constructed using this analysis, with enrollment taken

FIGURE F-1 Regression fit of log(design variance) versus log(enrollment). SOURCE: Prepared by the panel.

to be the observed median enrollment for medium districts and for small districts, respectively.

The standard deviation (SD) and coefficient of variation (CV) (relative to mean (CCD) = $1.65) of 1-year change in 5-year estimates for various enrollments are shown in Table F-12.

There are real differences in the amount of noise under which districts normally operate with traditional application and certification procedures. Small districts combined have a percentage standard deviation (CV) of 10.3 percent for CCD 1-year changes, but those with less than the median enrollment have a CV of 11.6 percent, while those with more than the median enrollment have a CV of 8.7 percent. These are comparable to the ACS5 (modeled) CVs at enrollments of 400-800, according to Table F-12, which is the same as Table 4-8 in Chapter 4. Figure F-2, which is the same as Figure 4-5 in Chapter 4, displays a transformation of the data in Table F-12. For a given district, the point (1/enrollment, CV²) can be plotted on the figure. If the plotted point is above the curve, the district currently experiences more variability in its administrative estimates

TABLE F-12 Intertemporal Variability of ACS 5-Year Estimates, by Enrollment

SOURCE: Prepared by the panel.

FIGURE F-2 Squared coefficient of variation of year-to-year change in ACS 5-year estimate of BRR versus inverse of enrollment.
NOTES: ACS = American Community Survey; BRR = blended reimbursement rate; CV = coefficient of variation.
SOURCE: Prepared by the panel.

than it would if it used ACS 5-year estimates (at least according to the model and ignoring timeliness bias). In this situation, the district might find use of the ACS 5-year estimates to be acceptable. On the other hand, if its plotted point is below the curve, the district currently experiences less variability in its administrative estimates than it would with the ACS 5-year estimates, and might find the latter unacceptably variable for use in determining reimbursements under the ACS Eligibility Option (AEO).

Table F-13 shows standard deviations, biases, and root mean squared errors (RMSEs) for ACS 1-year, 3-year, and 5-year estimators, with and without a 2-year lag (reflecting the lag in the availability of ACS estimates for use in establishing claiming rates). For large districts, these values are computed in two ways: (1) using the AR(1) model originally fitted via SAS for g_dt and (2) using the AR(1) + noise model for g_dt. The latter model makes results consistent with the analysis for medium and small districts, all of which use the AR(1) + noise model. The other difference in the AR(1) analysis for the large districts is that b_t is estimated from the data (see Box F-7) and incorporated in the bias computations, while in the AR(1) + noise analysis, it is assumed to be constant over time (or zero, without loss of generality). Again this is done to maintain consistency with the analysis for medium and small districts, for which estimation of b_t from the data is not possible.

TABLE F-13 Standard Deviation, Bias, and RMSE for ACS 1-Year, 3-Year, and 5-Year Estimates at Lags of 0 and 2 Years

NOTES: The results for large districts were obtained using two methods: (1) using the AR(1) model for g_dt and (2) using the AR(1) plus noise model for g_dt. ACS = American Community Survey; NA = not applicable; RMSE = root mean squared error; SD = standard deviation.
SOURCE: Prepared by the panel.

The bias and RMSE results reflect the specific m_t estimated for the particular 5-year time window covered by the estimates available to the panel, separately for each district size class. For any size class, the estimate of m_t is simply the year t average CCD BRR across all districts. For large districts, these estimated m_t values are given in Box F-7.

PART 3: MODEL-BASED PREDICTION OF SYSTEMATIC DIFFERENCES BETWEEN ACS ESTIMATES AND CCD ESTIMATES FOR BRR

This part of the appendix describes the results of the panel’s modeling of the differences between ACS estimates and CCD estimates for the BRR. The analysis was limited to very high FRPL districts with both 5-year ACS estimates and CCD estimates for 2009-2010 in the panel’s evaluation data set prog09.merged.fns. To eliminate outliers that could adversely impact regression results, we excluded any districts that had either a percentage certified for free meals of less than 10 percent or a percentage certified for free or reduced-price meals of less than 20 percent. Districts with missing data for potential predictor variables were also excluded.

The ACS estimate used in the analysis is the 5-year ACS estimate for the BRR (denoted ACS5 BRR below). The CCD estimate used is the BRR based on certification data in the 2009-2010 CCD (denoted CCD0910 BRR below). The dependent variable used in the analysis is the difference between ACS5 BRR and CCD0910 BRR divided by the standard error of ACS5 BRR. This variable is regressed on a variety of predictor variables from the 2009-2010 CCD as described below. Table F-14 provides regression results for a variety of alternative models.

In the table, p is the number of covariates in a model, and FOI stands for “first-order interactions.” The “Additive” model is the most basic model, with no interactions or quadratic terms, and the “FOI, No Factor Interaction” model includes interactions among continuous covariates but not with or among the categorical covariates. Box F-8 lists the covariates used in the modeling. The results of our exploratory analyses of whether a global predictive model could be used for adjusting for differences between ACS and administrative estimates are discussed in Chapter 4.

TABLE F-14 Results for Various Models Predicting Differences Between ACS 5-Year Estimates for 2005-2009 and CCD Estimates for 2009-2010

NOTES: The basic model is ~ covariates. AIC = Akaike Information Criterion; ACS = American Community Survey; FOI = first-order interactions; NA = not applicable; RMSE = root mean squared error.
SOURCE: Prepared by the panel.