Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 65
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report 5 Technical Approach to Estimation The panel’s charge is to use data from the American Community Survey (ACS) and other sources to develop claiming percentages for reimbursement under a new Provision 4. As noted in Chapter 3, several issues contribute to potential biases and must be addressed: monthly income is the basis for eligibility, but annual income is measured by the ACS; there are schools (such as charter and magnet schools) that draw students from neighborhood schools, possibly changing the concentration of eligible students in the neighborhood schools; ACS estimates for small areas will be available only as 5-year averages and might not be timely in reflecting changing economic conditions. Addressing these issues will be a key challenge for the panel. Another challenge is that ACS direct estimates for school attendance areas and many districts are likely to have high sampling error. The panel feels that the most fruitful approach to addressing this challenge will be to build on the Census Bureau’s Small Area Income and Poverty Estimates (SAIPE) Program. The panel recognizes that this is a major undertaking, requiring close collaboration with the Census Bureau. It is also likely to require further research that probably cannot be accomplished within the time frame of our study. However, at a minimum, we expect to be able to implement and test straightforward adaptations to SAIPE methods and to assess the feasibility of a model-based approach that would support the school meals programs. In addition, through the case studies described in Chapter 4, the panel will obtain and evaluate the accuracy of digitized school attendance boundary data and develop
OCR for page 66
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report methods for deriving estimates of eligible students for schools and groups of schools. Finally, the panel will consider how to obtain participation-based claiming percentages from estimates of eligible students. This will rely on a review of existing research—including analyses conducted by states and school districts and evaluations of pilot studies—and the collection and analysis of administrative data from school districts that have adopted Provision 2. In the development and evaluation of methods, as described in this chapter and the next, the panel will consider the operational feasibility of potential approaches, as discussed in Chapter 7, and prioritize our work accordingly. REIMBURSEMENTS UNDER PROVISION 4 Chapters 2 and 3 review basic features of the school meals programs and the ACS and sketch how ACS data might be used to determine federal reimbursements for a school, a group of schools, or an entire district that provides universal free meals under a new Provision 4. In Chapter 2, the potential role of ACS data in implementing Provision 4 was characterized by the following two reimbursement formulas: and where Ge4t is the government outlay for reimbursable meals served in month t in Provision 4 schools, based on eligibility estimates, in dollars; Gp4t is the government outlay for reimbursable meals served in month t in Provision 4 schools, based on eligibility and participation estimates, in dollars; Rf, Rr, and Rp are reimbursement rates for free meals, reduced-price meals, and full-price meals, respectively; Mt is the total number of reimbursable meals served in month t; Ef is the estimated number of enrolled students who are eligible for free meals based on ACS and other sources; Er is the estimated number of enrolled students who are eligible for reduced-price meals based on ACS and other sources; E is the estimated number of enrolled students in Provision 4 schools based on ACS and other sources;
OCR for page 67
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report Ep = E – Ef – Er is the estimated number of enrolled students who are eligible for full-price meals; Cf is the claiming percentage for free meals—it is an estimate for the fraction of meals served to students eligible for free meals; Cr is the claiming percentage for reduced-price meals—it is an estimate for the fraction of meals served to students eligible for reduced-price meals; and Cp = 1 – Cf – Cr is the claiming percentage for full-price meals. With the first formula, reimbursements would be enrollment-based—that is, they would be based on the estimated distribution of enrolled students across eligibility categories (free, reduced price, and full price). With the second formula, reimbursements would be participation-based—that is, they would be based on the estimated distribution of meals served across the eligibility categories.1 As documented in Chapter 2, the enrollment and participation distributions are different, and a substantial concern is that the enrollment-based reimbursement formula would provide unfairly smaller reimbursements to districts than would the participation-based reimbursement formula. Despite this concern and the fact that the enrollment-based formula is a special case of the participation-based formula, presenting both formulas is helpful for highlighting some of the challenges that arise in using ACS data. As noted in Chapter 3, a challenge in deriving eligibility estimates is that the data collected by the ACS can be used only to approximate the eligibility criteria for free and reduced-price meals. For example, while the ACS collects annual income data, program eligibility is based on monthly income, and once a student is approved for free or reduced-price meals, that approval remains in effect for up to 30 days into the next school year. A challenge in deriving participation estimates is that the ACS collects no data to predict participation, beyond its information on eligibility. Hence, information about the relationship between the eligibility and participation distributions must come from other sources of data. Additional challenges that arise in using ACS data include identifying ways to enhance the precision and timeliness of estimates. The rest of this chapter discusses the panel’s plans to develop methods for estimating program eligibility from the ACS, adjust estimates to better reflect program eligibility criteria, improve timeliness and precision, and estimate claiming percentages that take participation into account. The chapter concludes with a discussion of the panel’s approach to analyzing 1 As noted in Chapter 2, a special case of the participation-based formula estimates claiming percentages using the enrollment percentages from the enrollment-based formula.
OCR for page 68
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report the potential costs and benefits to a district of adopting Provision 4 in some or all of the districts’ schools. ACS DIRECT ESTIMATES OF ELIGIBLE STUDENTS The ideal enrollment-based claiming percentages under Provision 4 would be the elements of ET in Figure 2-1, that is, the percentages of students who are eligible for free, reduced-, and full-price meals under the rules of the school meals programs. Those percentages, however, are unobserved and need to be estimated. The ideal estimates would have very low variances and be nearly unbiased. As discussed in Chapter 3, ACS direct estimates might need “adjustments” to remove potential biases, such as a bias resulting from the ACS measuring of annual, rather than monthly, income. In addition, small-domain estimation methods might be needed to improve precision. Before exploring the use of such methods or adjustments for potential biases, we will seek to identify the best possible measure of eligibility using ACS variables. In developing an approach to ACS direct estimation,2 the panel’s first empirical task will be to examine alternative combinations of ACS variables to determine which most closely reflects the eligibility criteria of the school meals programs. For a set of “test” districts—specifically, unified (K-12) school districts that can be identified in the ACS Public Use Microdata Sample files—we will derive estimates (and standard errors) of the following: total number of students (by public, private); number of students with family annual income no greater than 130 percent of poverty whose families do not receive Supplemental Nutrition Assistance Program (SNAP) or other welfare benefits (by public, private); number of students with family income greater than 130 percent of poverty and no greater than 185 percent of poverty whose families do not receive SNAP or other welfare benefits (by public, private); number of students whose families receive SNAP benefits (by public, private); 2 An ACS direct estimate for a domain—which is defined by geographic area, population group, and time period—is derived using ACS data for that domain only. Data for other domains are not used, as they would be by an indirect, that is, model-based estimator. Although an ACS 5-year period estimate is arguably indirect by this definition, we consider it to be direct for present purposes. Also, we consider estimates to be direct even if models are used to derive weights and obtain variances, as with the ACS.
OCR for page 69
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report number of students whose families receive other welfare benefits (by public, private); and number of foster children in school (by public, private). From these counts, we will estimate the percentage of students eligible for free meals and the percentage eligible for reduced-price meals. We expect that some combination of the last three variables will be used to identify students who are categorically eligible for free meals. It may be that the best estimates will be the most straightforward: students who are eligible for free meals are those who are categorically eligible (foster children or in families receiving SNAP or other welfare benefits) or have family income no more than 130 percent of poverty, whereas students who are eligible for reduced-price meals are those who are not categorically eligible and have family income greater than 130 percent of poverty and no greater than 185 percent of poverty. For the test districts, we will compare estimates of total and eligible students (by category) from the ACS with estimates from the Common Core of Data (CCD) and Form FNS-742, recognizing that the CCD and the FNS-742 provide estimates for CT (see Figure 2-1), the distribution of approved students, rather than for ET, the distribution of eligible students. In addition, the panel will consider the potential effects of errors in both the survey and the administrative data. We will evaluate differences for school districts, in aggregate and by district characteristics, such as level of need. After the panel has identified leading candidate methods for deriving direct estimates, we will work with the Census Bureau to obtain estimates for all school districts included in the bureau’s geographic inventory based on 1-year, 3-year, and (eventually) 5-year ACS data, taking into account the grade range for each district. One-year estimates will be publicly available only for districts with population greater than 65,000 (approximate school enrollment of 11,700).3 Three-year ACS estimates will be available for all school districts with total population greater than 20,000 (approximate school enrollment of 3,600). At the end of 2010, 5-year estimates will be available for all school districts. Some of these estimates will also be compared with CCD and FNS-742 data. For single-school districts, differences can be evaluated by level of need and for elementary, middle, and high schools. 3 The ACS product release schedule is based on total population in the geographic area. The 2006-2008 ACS shows that the nation has about 301,238,000 people, of whom 53,452,000 (about 18 percent) are school-age children in grades K-12. We estimate that in an area with 65,000 people, there will be a school-age population of about 11,700, and in an area with 20,000 people, there will be a school-age population of about 3,600.
OCR for page 70
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report POTENTIAL ADJUSTMENTS TO ACS ESTIMATES In addition to developing one or more basic approaches to deriving direct eligibility estimates from the ACS, the panel will also develop approaches for addressing the challenges discussed earlier. For simplicity, we will call these approaches “adjustments.” The challenges for which we will explore adjustments are (1) estimating eligibility from annual rather than monthly income data; (2) accounting for attendance at charter, magnet, and other schools that draw students from neighborhood schools; and (3) enhancing timeliness. Later in the chapter, we discuss methods for improving the precision of estimates and methods for estimating participation. Monthly Versus Annual Income The ACS collects data on annual income and annual program benefit receipt. However, eligibility for the school meals programs is based on monthly income and current participation. Several studies have examined the relationship between monthly income and annual income for determining poverty or program eligibility. For example, Naifeh (U.S. Census Bureau, 1998) used 1993 and 1994 Survey of Income and Program Participation (SIPP) data to consider seven different measures of poverty (income relative to the poverty threshold). She showed that, in 1994, the annual poverty rate was 12.6 percent, while the average monthly poverty rate was 15.4 percent. More recently, Newman (U.S. Department of Agriculture, Economic Research Service, 2006b) used SIPP data to show that “an estimated 27% of households that were income eligible for free or reduced price lunches at the beginning of the school year were no longer income eligible for the same level of subsidy by December due to monthly income changes.” A panel of the National Research Council (2003:6-7) considered the problem of measuring eligibility for the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) based on annual income data from the Current Population Survey (CPS). The panel made the following observations: The major limitation of the CPS for estimating WIC eligibility is that it measures only annual income and annual participation in WIC and other public assistance programs that confer eligibility for WIC. Use of a monthly measure of income instead of an annual measure, as is currently used, was chosen as the most appropriate time period to measure income to estimate eligibility because WIC regulations give great flexibility in the unit of time for which an applicant must report income and because variation in flows of income for families are better captured with a monthly
OCR for page 71
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report income measure. The panel proposes the following new CPS-based option to improve the current CPS estimation. … [T]o account for monthly income, apply a constant multiplier to the core estimates based on annual income. The panel used Transfer Income Microsimulation (TRIM) data, which simulates monthly income based on the March CPS, to estimate a multiplier for infants and children: 1.2 for infants and 1.05 for children. An alternative to using this TRIM-based multiplier is to use SIPP data to estimate a similar multiplier. The multiplier is used to approximate the incremental effect of using monthly income instead of annual income. The panel went on to note that SIPP has better measures of monthly income than does the TRIM model, and that if such a multiplier is used, it should be reestimated every few years. The panel recommended that the stability of the multiplier should be continually reassessed. Table 5-1 displays annual average poverty rates based on monthly income (first row), poverty rates based on annual income (second row), and the ratio of the former to the latter (third row), estimated from SIPP data. The last three rows pertain to children under age 18. Ratios of average monthly to annual poverty rates range from 1.22 to 1.32 for all people and are about 1.22 for children under age 18, indicating that a significantly higher proportion of students may be eligible for free or reduced-price school meals on the basis of monthly rather than annual family income. The panel will examine similar tables from SIPP for the 130 percent and 185 percent levels of poverty and will also consider estimates by age TABLE 5-1 Average Monthly and Annual Poverty Rates from SIPP Age Income Measure Percentage of People with Income Less Than the Poverty Threshold 1993 1994 2001 2002 2003 Total Average monthly 15.7 15.4 14.1 14.1 13.8 Annual 12.9 12.6 10.7 10.7 11.0 Ratio of average monthly/annual 1.22 1.22 1.32 1.32 1.25 Under Age 18 Average monthly N.A. N.A. 19.6 19.9 19.6 Annual N.A. N.A. 16.0 16.3 16.2 Ratio of average monthly/annual N.A. N.A. 1.23 1.22 1.21 NOTE: N.A. = not available. SOURCE: Data for 1993 and 1994 from U.S. Census Bureau (1998). Data from 2001-2003 from U.S. Census Bureau, Survey of Income and Program Participation, Dynamics of Poverty, 2001-2003. See http://www.census.gov/hhes/www/poverty/dynamics01/index.html [accessed May 2010].
OCR for page 72
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report or grade. One of the disadvantages of using SIPP to provide data for an adjustment is that the factors can be developed only at the national level or at a relatively high level of geographic aggregation. Based on our examination, the panel will consider whether an adjustment of some kind might improve the accuracy of ACS eligibility estimates. Charter and Magnet School Attendance Saporito and Sohoni (2007) note that the income distribution of students attending a neighborhood public school can be affected by students who are drawn away from that school to attend a private, charter, or magnet school. The ACS provides data to estimate eligibility for public and private schools separately. However, the panel is not aware of data on charter and magnet schools, home schooling, open enrollment, or other public school choice programs, except possibly at the school district level. The panel will explore how best to account for the effects of programs that draw children from their neighborhood schools in collaboration with the case study districts. Depending on the availability of local data, some adjustment may be possible. Timeliness The ACS 5-year period estimates will present substantial issues of timeliness. When released at the end of 2010, the 5-year estimates for 2005-2009 will represent an averaging of income received during a period starting in January 2004 and ending in December 2009. The midpoint of this interval, January 1, 2007, will be almost 4 years earlier than the release date. Therefore, the impact of an economic downturn with an accompanying increase in eligibility, for example, will be slow to appear in the estimates and then will be averaged over a period of years. Many statistical programs incorporate adjustments to estimates at one level of aggregation to estimates at a higher level, including the Census Bureau’s SAIPE program. Some form of simple ratio adjustment could be used to adjust the most recently available 5-year estimates for school districts to the most recently available 1-year estimates for a higher level, such as the state, county, or metropolitan area. This approach would partially address the timeliness issue faced by direct use of the 5-year estimates. For example, the full impact of broad trends in the economy would be felt an average of 2 years earlier with this hybrid approach than with the direct 5-year period estimates. The approach would not significantly improve the variance of the school district estimates, however, and the improvement in timeliness might come at some cost of accuracy at the school district level. The panel will be able to evaluate the effectiveness of
OCR for page 73
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report some approaches for adjustment using the 3-year ACS estimates because three releases of 3-year estimates will be available by the end of 2010. However, there will be only one release of 5-year estimates by 2010, so an evaluation of performance over time will not be possible until well after the panel’s work is completed. The panel will also consider adjusting 5-year estimates from the ACS with other data that reflect the current economic situation, such as administrative data on SNAP participation. Relatively current SNAP participation data are obtained by the Census Bureau at the state and county levels for use in its SAIPE program. Their availability at the local-area level has not been ascertained. For school districts and schools, it is possible that direct certification estimates might provide the most relevant source of data for adjusting direct ACS estimates.4 However, it is the panel’s understanding that direct certification is not currently done for Provision 2 or Provision 3 schools, although it is done in Philadelphia. Hence, the data for such an adjustment would not be readily available, unless districts that adopt Provision 4 are required to continue direct certification. PRECISION OF ESTIMATES Direct Estimates The direct ACS estimates for many school districts and schools are likely to exhibit large standard errors, as illustrated by Table 5-2.5 For each ACS product (1-year, 3-year, 5-year), the table gives approximate standard errors of the estimated proportion of free and reduced-price eligible students for three different-sized geographic areas. For example, for a school district with 16,000 students and a free and reduced-price eligible fraction of 0.7, the expected standard error for a 1-year ACS estimate is 0.072—so a 90 percent confidence interval would be 0.58 to 0.82.6 For a school, group of schools, or school district with an enrollment of 1,500 and a free and reduced-price eligibility fraction of 0.7, the standard error for a 5-year estimate is 0.101, and the 90 percent confidence interval would be 0.53 to 4 We will also consider whether direct certification information can be used more directly in the derivation of direct estimates by, for example, estimating the number of categorically eligible students from direct certification data and the number of noncategorically eligible students from ACS data. Another potential use of direct certification data is in model-based estimation, as discussed below. 5 For purposes of this report, we calculated standard errors using the formula for a simple random sample and a design effect of 3. For subsequent analyses, we will perform more refined calculations that reflect the design of the ACS. 6 The confidence interval is computed as the fraction of free and reduced-price eligible students plus or minus 1.645 times the value in the table.
OCR for page 74
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report TABLE 5-2 Approximate Standard Errors of ACS Direct Estimates by Type of ACS Release, School Enrollment, and Estimated Fraction of Free and Reduced-Price Eligible Students ACS Release School Enrollment Fraction of Students Eligible for Free and Reduced-Price Meals 0.5 0.6 0.7 0.8 0.9 1 year 12,000 0.091 0.090 0.084 0.073 0.055 1 year 16,000 0.079 0.077 0.072 0.063 0.047 1 year 20,000 0.071 0.069 0.065 0.057 0.042 3 year 4,000 0.089 0.088 0.082 0.071 0.054 3 year 7,000 0.068 0.066 0.062 0.054 0.041 3 year 10,000 0.057 0.055 0.052 0.045 0.034 5 year 500 0.191 0.187 0.175 0.153 0.115 5 year 1,500 0.110 0.108 0.101 0.088 0.066 5 year 3,000 0.078 0.076 0.071 0.062 0.047 SOURCE: Estimated by the panel. 0.87.7 Confidence intervals of 58-82 percent and 53-87 percent of students eligible for free or reduced-price meals are likely to be too wide for school districts to evaluate the costs and benefits of implementing a new Provision 4 for reimbursement of meal costs. The magnitude of the standard errors in the table and the width of the implied confidence intervals led the panel to consider model-based estimates to reduce the variation inherent in direct estimates for small areas. Improving Precision with Small-Domain Estimation Methods There is a substantial and growing body of statistical literature on small-domain estimation (e.g., Rao, 2003). Small-domain estimators typically use auxiliary data and borrow strength across domains to reduce the sampling variances of the direct estimates. By allowing small increases in the average bias, it is possible to achieve substantial reductions in variance, a trade-off making the small-domain estimates an attractive alternative to direct estimates. The most relevant example of small-domain estimation for the panel’s work is the SAIPE program, described in Chapter 3. One of the program’s 7 These are single-release standard errors. To compare estimates from one release to another, the panel will need to account for any correlation between releases.
OCR for page 75
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report primary objectives is the estimation of the numbers and proportions of related children ages 5-17 in poverty by state, county, and school district. The estimation process nests estimates at the three levels, controlling the estimates to achieve consistency with the level above. That is, county estimates are controlled to the state estimates, and school district estimates are controlled to the county estimates. For each state, the SAIPE estimate is a weighted average of a regression estimate and the direct 1-year ACS estimate. The regression estimate uses the ACS proportion of poor school-age children as the dependent variable. Because of their small sampling errors, the direct ACS estimates receive most of the weight in the SAIPE state average. At the county level, SAIPE also employs a regression model, but the log of the 1-year ACS estimated number of poor school-age children is the dependent variable.8 Although a composite is formed from the direct ACS estimate and the regression estimate, the regression estimate is the dominant component in the majority of counties. As noted in Chapter 3, SAIPE distributes the estimated numbers of poor children below the county level to the school district–county pieces within each county according to a shares model based on the number of children in poverty estimated from Internal Revenue Service (IRS) data. Direct ACS estimates at the school district level are not currently employed. The goals of the panel are sufficiently consistent with some of the goals of the SAIPE program to merit a careful consideration of a “SAIPE-like” approach to the derivation of eligibility estimates for the school meals programs. Therefore, we will establish a mechanism for collaboration with the Census Bureau and other agencies, such as the U.S. Department of Agriculture’s Food and Nutrition Service (FNS), for the development and testing of small-domain model-based estimates. Our initial work will focus on the most straightforward adaptations of the SAIPE estimation procedures to produce estimates of students eligible for free meals and students eligible for reduced-price meals at the state, county, and school district levels.9 Likely adaptations include the adjustment of the poverty thresholds from 100 percent to 130 percent and 185 percent for the dependent variables in the regression models as well as the independent variables to which such an adjustment would be applicable and operationally feasible. We would also probably investigate expressing the variables in a county 8 Although 1-year ACS county estimates are published only for larger counties, the Census Bureau uses the internal ACS data to include almost all counties in fitting the regression model. 9 The derivation of estimates for individual schools and school groups is discussed in this chapter’s next section.
OCR for page 76
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report model as proportions rather than log counts, similar to the specification for the state model. More substantial adaptations can be considered for the derivation of school district estimates, and the panel will consider as many of the adaptations described below as possible given the time frame of the study. As noted above, the current SAIPE method uses a shares model. However, in addition to this approach, we can consider the use of a regression model and the development of composite estimates that combine direct and regression estimates, as in the state and county models. For such a model and, perhaps, for the state and county models, we can investigate the availability and potential predictive contributions of new independent variables constructed from direct certification data or the CCD, for example. One challenge in using a variable that measures the extent of direct certification is that it might reflect not only how much families need assistance from the school meals programs but also how well direct certification is implemented by a school district or a state.10 After further consideration of these adaptations, the panel may conclude that some cannot be implemented and adequately evaluated within the schedule and resources available for the panel’s work. In that case, we plan to provide recommendations for future research. Additional methodological innovations that will be considered as subjects for future research—if they cannot be thoroughly assessed by this panel—include Using 3-year or 5-year estimates rather than 1-year estimates as dependent variables in the regression models to combine the variance-reducing properties of time averaging with the variance-reducing properties of regression modeling.11 Estimating multivariate—multigroup or multiperiod—models. Estimates for free and reduced-price eligible students could be estimated jointly, for example. Developing unit-level, rather than area-level, small-domain estimation models. With a unit-level model, the modeling occurs below the target level of estimation, such as modeling at the person or household level to derive estimates for states, counties, or school districts. In assessing the merits of a SAIPE-like approach, the panel will review the research support for the current SAIPE estimation methods. The SAIPE 10 U.S. Department of Agriculture, Food and Nutrition Service (2009) reports that direct certification rates vary substantially from state to state. 11 Each approach, however, contributes bias, and the interaction of these two sources of bias would pose research questions for which the SAIPE program does not currently provide answers.
OCR for page 77
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report program preceded full-scale implementation of the ACS by several years and originally used direct estimates from the CPS. At that time, Census Bureau staff conducted extensive research, and the findings from that research and the estimates of poor related children ages 5-17 that were developed were reviewed by a panel of the National Research Council (2000a). However, the evidence available for evaluating the school district estimates was limited, and there has been no subsequent independent evaluation of the changes to the SAIPE estimation methods, such as the use of ACS data. In addition to reviewing the findings from evaluations that have already been conducted, the panel will work with the Census Bureau to implement and evaluate the adaptations to current SAIPE methods that are feasible to explore within the time and resources available to the bureau and the panel. We will examine the precision of estimates and assess models for evidence of any systematic lack of fit. Chapter 6 describes in further detail our approach to evaluating estimates. While the panel will certainly evaluate new methods and estimates according to statistical criteria, such as precision and bias, we will also carefully assess the operational feasibility of any approach when developing our recommendations. Our strategy for determining whether an approach is operationally feasible is outlined in Chapter 7. At this point, however, we note that it will be important to determine that the release of any new estimates does not jeopardize the confidentiality of respondents to the ACS. Currently, SAIPE estimates are made publicly available for all school districts, some of which have only one school. The Census Bureau’s disclosure review board made the determination that the few variables released at the school district level for the SAIPE program do not constitute a disclosure risk. The Census Bureau’s disclosure review board will need to judge whether this is also the case for estimates of free and reduced-price eligible students prepared as an adaptation to SAIPE. ESTIMATES FOR SCHOOLS AND GROUPS OF SCHOOLS Defining Geographic Areas Our expectation is that estimates for individual schools or groups of schools would be produced only “on demand,” that is, at the request of a school district that is considering adoption of Provision 4 but, due to substantial heterogeneity of need across schools in the district, might not adopt Provision 4 district-wide. Because the Census Bureau does not maintain geographic data on school attendance-area boundaries, which can change frequently, a school district would have to provide suitably accurate information on boundaries. However, correct definition of the
OCR for page 78
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report school attendance boundaries with respect to census blocks (the building blocks of Census Bureau geography) will be a challenge.12 For schools, there are several ways to define their geographic domains. When school attendance areas are defined by school authorities, it may be in the form of a list of addresses, perhaps with ranges (for example, odd numbers in the 100 block of State Street to School A, even numbers to School B). In many respects, address lists have the least potential for error of all of the geographic representations, although there may be errors in geocoding (linking) the addresses to the Census Bureau’s Topologically Integrated Geographic Encoding and Referencing (TIGER) database, which is needed to identify the appropriate blocks to aggregate when deriving estimates for schools and groups of schools (see below). Address lists may be represented in map form, although several problems arise in defining geographic boundaries that are consistent with the address lists. Maps inherently have a variety of errors, including minor projection errors at the local level associated with representing the earth’s curved surface with a flat map and more substantive transformation errors associated with digitizing paper maps. Depending on the digitizing process, positional errors associated with digital boundaries can be relatively small or large. Because school attendance areas tend to follow well-defined features, such as streets, it should generally be relatively easy to estimate the positional errors in any set of digitized boundaries. For the ACS, the geographic support is the Census Bureau’s TIGER database, which is a digital map of streets and other features. As noted in Chapter 3, the accuracy of TIGER has been substantially improved through a recent major initiative in preparation for the 2010 decennial census, so positional errors are now in the 5-meter range for streets and other major features. When a positional coordinate (expressed via latitude and longitude) is used to compare school attendance-area boundaries to census data, the positional errors in the attendance-area boundaries are likely to be larger. It will be possible to estimate how large those errors are for the case study districts and to use simple models to determine the impacts of such errors on the estimates that are of primary interest in this study. Deriving Estimates Direct estimates of eligible students for schools or groups of schools will probably have to be derived by aggregating and weighting sample counts for blocks. For many schools, however, the attendance boundaries 12 In the future, if the School Attendance Boundary Information System project (National Science Foundation, 2009) is successful, it may be easier for school districts and researchers alike to have access to accurate, up-to-date digitized school attendance boundaries.
OCR for page 79
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report will run through blocks, and statistical algorithms for splitting blocks may need to be developed and evaluated. Goodchild, Anselin, and Deichmann (1993) describe a framework for areal interpolation—one solution to this problem. Saporito and Sohoni (2006, 2007) collected maps for the schools in the 21 largest school districts and computed estimates for race and ethnicity (available at the block level) and for income eligibility for free and reduced-price meals (available only at the block group level) from the 2000 census. They observed that while census blocks can frequently be located within a school boundary, block groups often cut across school attendance boundaries in unpredictable ways. They made use of an areal interpolation method that assumed that block group rates, such as eligibility rates for free or reduced-price meals, are uniform within block groups that span more than one school attendance boundary. In another application, FNS commissioned the U.S. Census Bureau to prepare school meals eligibility estimates for school attendance areas in Philadelphia using the 2000 census long-form income data. The work needed to digitize school attendance boundaries for the public schools in Philadelphia was described in their report.13 In more recent work pertaining to school attendance-area boundaries, researchers on the School Attendance Boundary Information System project (National Science Foundation, 2009) are assigning each ACS block entirely to one school attendance area and are not using any method of interpolation. For the work of this panel, it is unclear which of these approaches will be best in terms of not only accuracy but also feasibility of implementation. The case studies will be used to illuminate this issue. As an alternative to interpolation, the panel hopes that at least one case study school district will be able to geocode student address lists using TIGER line files and, thereby, provide information indicating how students in each school are distributed across census blocks. This information will shed light on the relative accuracy of different approaches to handling the geography of school attendance areas when developing ACS estimates of students eligible for the school meals programs. Given that geographic area definitions can be adequately addressed, the challenges in developing estimates for individual schools or groups of schools will be at least as great as, and probably greater than, the challenges in developing estimates for school districts. A direct estimator for a school or group of schools will be less precise than an estimate for the entire district. Furthermore, attendance at a particular school is probably less reliably predicted by a student’s geographic residence than is attendance in a district. This problem is particularly severe in districts 13 Doug Geverdt, 2005, Experimenting with School Attendance Area Free Lunch Estimates: A 2000 Census Special Tabulation Case Study, unpublished report.
OCR for page 80
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report with open enrollment or districts where there are many schools, such as charter and magnet schools, that can enroll students from throughout the district without regard to neighborhood school boundaries, and this may introduce errors in estimates for neighborhood schools whose enrollments are substantially affected.14 A SAIPE-like estimator could be considered for deriving ACS-based estimates of eligible students for schools and groups of schools. One such estimator is a shares model similar to what is currently used to obtain school district estimates from county estimates.15 Shares might be developed from enrollment, direct certification, or IRS data (if the IRS data could be tabulated for school attendance areas). An alternative to a shares model is a model with a regression equation (with direct ACS estimates as the dependent variable) and a composite estimator, similar to the current SAIPE state and county models. Independent variables in the regression models could be developed from the same auxiliary data used to estimate shares in the shares models. As described in Chapter 4, the panel will use the case studies to investigate the issues raised here as well as other issues that arise, including errors in digitized school attendance boundaries; the effects of open enrollment or charter, magnet, and other schools; algorithms for splitting blocks; and auxiliary data that can be used in shares or regression models. ESTIMATING PARTICIPATION This section describes the challenges associated with understanding the relationship between eligibility and participation and using what is learned to estimate claiming percentages for reimbursement under Provision 4. As shown in Chapter 2, the eligibility and participation distributions are different. According to national estimates for 2009 from the FNS National Data Bank, approximately 40 percent of enrolled students were approved for free meals and 9 percent were approved for reduced-price meals under the National School Lunch Program (NSLP). The remaining 51 percent had to pay full price. In contrast, of the NSLP lunches served on an average day in 2009, 52 percent were served to students approved for free meals, 10 percent were served to students approved for reduced-price meals, and 38 percent were served to students paying full price. These differences between the eligibility and participation distributions 14 Home schooling can also draw students from their neighborhood schools. 15 In SAIPE, direct ACS estimates are used in obtaining model-based estimates for counties, but federal income tax data are used to estimate shares for deriving school district estimates from the county estimates. Direct ACS estimates for school districts are not used by the current SAIPE estimation method.
OCR for page 81
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report are attributable to the differences across the eligibility categories in the likelihood of taking a meal—80 percent of students approved for free meals, 72 percent of students approved for reduced-price meals, and 46 percent of students paying full price took NSLP lunches on an average day in 2009 (see Table 2-4). Understanding Differences in Eligibility and Participation To better understand the differences between the eligibility and participation distributions, the panel proposes to explore the extent to which the ACS estimates of student eligibility for the schools in the case study school districts differ from the percentages of approved students from the certification process and the participation rates (based on meals served) in those schools. We will also compare data from the schools in the case study districts with the data for these districts in the CCD and the data reported on Form FNS-742. These comparisons will be done by level of need and grade range of school (elementary, middle, and high), attributes that are known to affect participation. Although this will not illuminate how participation might change if all meals were provided at no cost under Provision 4, it will help the panel to understand local differences in the relationship between the eligibility and participation distributions. The panel will also consider the changes in participation that might be expected under Provision 4 when all enrolled students are given access to free meals. Given typical economic assumptions about the role of prices in demand for school meals (that school meals are a normal good, for which demand increases when the price decreases), adoption of Provision 4 would be expected to increase demand among all students who were not already approved to receive free meals. The availability of free school meals for all students might also be expected to increase demand (increase the number of school meals consumed) among those eligible for free meals because it reduces the family’s burden of applying for benefits and removes any perceived stigma associated with participating in the program. There is an additional complication with using the ACS estimates, even if adjusted to reflect participation. The ACS eligibility estimates are estimates of “true” eligibility percentages for all students enrolled at a school. The current claiming percentages under traditional operating procedures, however, reflect participation rates based on numbers of meals served to students as they have been approved through the certification process, which, as noted in Chapter 2, does not assign all students to their true eligibility categories. Not all students, for example, who would be eligible for free or reduced-price meals apply to the program. Currie (2003) finds that for students with income less than 130 percent of poverty, only 87 percent participated in the school lunch program in 1998, while Currie
OCR for page 82
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report (2006) cites evidence that participation is higher for the poor than for the nonpoor. In school districts in which a large share of students eligible for free meals do not apply to the program (and are not directly certified), using the ACS eligibility estimates as claiming percentages might increase reimbursements compared with the existing claiming percentages (all other things equal). The formulas at the beginning of this chapter display the differences in eligibility-based reimbursement (using , , and ) and participation-based reimbursement (using Cf, Cr, and Cp). The panel will investigate how to make use of the eligibility estimates from the ACS (Ef, Er, Ep) along with the participation data described in the remainder of this section to estimate participation (Cf, Cr, Cp). Sources of Information on Participation The results from several pilot studies provide potentially useful information on school meals participation. Five states and many school districts have taken advantage of a pilot project to eliminate the reduced-price fee from lunch, breakfast, or both—information about these experiments is presented in a U.S. Government Accountability Office (GAO) report (2009a). The extra cost associated with eliminating reduced-price fees was paid by states or local agencies. The GAO report estimated that there was an average increase in lunch participation of 11 percent among reduced-price-eligible students across all school districts that eliminated reduced-price fees. The minimum percentage increase in reduced-price participation was reported to be 2 percent, and the maximum was reported to be 30 percent. School district officials in these school districts reported that this increase was higher than the increase in the participation rate of free-eligible students (5 percent) and full-price-eligible students (5 percent). (Note that there was no change in the benefits offered to free and full-price-eligible students.) Other potentially relevant pilot studies include the universal breakfast pilot (U.S. Department of Agriculture, Food and Nutrition Service, 2004) and the no-fee school meal pilot (U.S. Government Accountability Office, 1994). Although the findings from the pilot studies of eliminating reduced-price fees are informative, the panel proposes to develop alternative estimates of the change in participation associated with providing free meals to students approved for reduced-price meals. Specifically, we will use the state-level data from Form FNS-10 to estimate the change in participation for reduced-price-approved students in the five states that have eliminated fees for reduced-price meals compared with any changes in participation that have occurred in the other states. For this analysis, we
OCR for page 83
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report TABLE 5-3 Projected Percentage Increase in Student Participation from Traditional to Free Meals for Students in the National School Lunch Program Elementary School Pre-K Through 5th Grade Middle/Junior High 6th Through 8th Grade High School 9th Through 12th Grade Current Participation Projected % Increase Current Participation Projected % Increase Current Participation Projected % Increase 60-70 3 45-55 22 30-40 33 70-80 4 55-65 15 40-50 25 80-90 2 65-75 13 50-60 15 90-95 1 75-85 8 60-70 10 85-95 3 70-80 5 NOTE: Participation = average daily number of meals served divided by total enrollment. SOURCE: Texas Department of Agriculture (2009). will use the state-level FNS-10 data on number of meals as the dependent variable and a dummy variable for when (whether) the state has implemented a program eliminating the reduced-price category as the treatment indicator in a regression specification. In addition to this analysis, the panel will examine estimates developed by the state of Texas for districts considering whether to adopt Provision 2. The estimates, which are displayed in Table 5-3, give the percentage increases in participation expected for districts adopting Provision 2. Using the table and a companion worksheet (section 5.13 of the referenced document), a school can project the increase in participation (total number of meals served per day) that it might experience. The table was developed from average daily participation data for school districts that adopted Provision 2 in Texas. The percentage increases were computed by comparing meal counts in the 2 years before adoption of Provision 2 with meals counts in the 2 years after adoption of Provision 2 separately for elementary, middle, and high schools at various levels of initial participation (average daily number of lunches served divided by total enrollment). The analysis was first conducted in the mid-1990s and has been regularly updated. To shed additional light on the potential effects of Provision 2 on participation, the panel plans to collect data concerning changes in participation experienced by existing Provision 2 schools when they first began operations under Provision 2.16 In the year before the first base year, a 16 Provision 2 is the only one of the current provisions that yields the data to illuminate changes in participation by category. Under Provision 3, schools do not provide free meals to all students during the base year.
OCR for page 84
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report school would have taken applications and counted meals according to the traditional approach. In the base year, the school would have taken applications, provided all meals for free, and counted meals by approval category. In addition to identifying the base year for each school adopting Provision 2, data to be collected include the number of students approved by category, the average daily number of meals served by category, and the total number of meals served, all as of October for each year. Along with data from selected schools that did not operate under Provision 2 (as controls), we can estimate the overall increase in participation due to the adoption of Provision 2 and the increase in participation by category. Schools and school districts that are not on Provision 2 or Provision 3 are required to retain information on the number of students approved by category and average daily number of meals served by category for only 3 years,17 while those on Provision 2 need to retain their base year claiming percentage information as well as the current year number of total meals for which reimbursements were made. Thus, the panel will need to identify Provision 2 schools that began operating under Provision 2 within the most recent 2 or 4 years (depending on data availability), so that the participation and eligibility data are likely to be available for the base year and for at least one year immediately preceding the base year. Ideally, for each such Provision 2 school, a nearby similar school can be identified and the same data elements collected. This second school will be a “control” to help account for changes in participation that may be due to local conditions. With data identifying the Provision 2 schools, their first base year, and how many meals were served by year by category, a regression equation can be used to predict the effect of universal free meals on participation. An example of such a regression equation is where Lijt is the number of meals served in school i in category j (free, reduced price, full price) in time period t; Nijt is the number of students in school i in category j in time period t (note that Nijt and Lijt will not be defined for Provision 2 districts after the base year); Tt is an indicator for time period t; Aij is an indicator for school i category j; Gj is an indicator for being in category j; 17 Some states require that data be kept for 5 years.
OCR for page 85
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report Dit is an indicator variable for the base year of a Provision 2 school (first year of free meals); and Zi is other school characteristics, possibly including grade range (elementary, middle, or high), the ACS-estimated eligibility shares, and perhaps other school-level controls from the CCD. The number of students estimated in the population will be used as weights to yield estimates of participation that are representative of the population. Then, the estimated coefficient for B3 is the estimated effect of providing free meals per student in category t. We will also explore specifications with the overall number of meals divided by enrollment as a dependent variable, controlling for each time period’s claiming percentages. Depending on data availability, the model may be adapted to include information from Provision 2 schools after the first base year when only total number of meals served is available. The panel is exploring several approaches to identify and recruit school districts or states to provide data: The School Nutrition Association (SNA) has developed a profile of participants at their 2010 School Nutrition Association Legislative Action Conference. The profile includes information about the school district and whether it has Provision 2 schools. SNA has agreed to provide the panel access to this profile, so that we can conduct a voluntary survey of districts that appear especially relevant to this study. The states in which the case study school districts are located will be asked to help locate Provision 2 schools that have adopted Provision 2 recently. In these states, the case study school districts may serve as “control” school districts. Form FNS-742 obtains data by school district on the number of schools and students in Provision 2 and Provision 3 schools not in a base year. It also has information on the number of students approved for free and reduced-price meals. These data could be used to profile Provision 2 and Provision 3 participation by state to identify states to target should additional data be needed. The panel plans to conduct structured open-ended interviews with state and school district officials. Questions will be asked about Provision 2, about meal costs and revenue, and whether the school district officials would be willing to work with the panel in the future. The panel would also like to get a sense of factors that the districts think will be important when determining whether to adopt Provision 4. In addition
OCR for page 86
Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report to collecting information to analyze changes in participation, the panel expects to obtain information to analyze costs. ASSESSING COSTS AND BENEFITS Once we have some sense of the possible changes in reimbursements due to adoption of Provision 4, it remains to estimate the effects on costs and benefits. Our assessment will be primarily in terms of costs and revenues, although we recognize that a chief benefit, which is crucially important but hard to quantify, is that of expanding the school meals programs to reach more students and thereby improve their nutrition and readiness for learning. The financial impact for school districts will depend on the balance between the increased costs of providing more meals, on one hand, and the increase in reimbursements from higher participation, together with the savings from reduced administrative costs of certification, verification, and meal counting by category, on the other hand. We will draw on previous work on program costs, including U.S. Department of Agriculture, Food and Nutrition Service (2004) on the School Breakfast Pilot Project; U.S. Department of Agriculture, Food and Nutrition Service (1994b, 2008a) on the School Lunch and Breakfast Cost Studies, I and II; U.S. Government Accountability Office (2002) on the costs of three administrative processes; and U.S. Department of Agriculture, Food and Nutrition Service (2007b), which is the Access, Participation, Eligibility, and Certification (APEC) report. The GAO study, for example, estimated costs for (1) providing, accepting, and reviewing applications for free and reduced-price meals; (2) verifying eligibility for free and reduced-price meals; and (3) counting reimbursable meals and claiming federal reimbursement. Data were collected in five states for two districts in each state and two schools in each district. As discussed earlier in this chapter and in Chapter 4, we will also be collecting cost and revenue information via structured interviews and data requests from the case study school districts, volunteers identified via the SNA survey, and possibly other states or school districts. With all of the available information, we will estimate the savings from eliminating certification, verification, and meal counting by category. For the case study districts, we will use estimates of eligibility and participation as the basis for hypothetical claiming percentages for reimbursement under Provision 4. The implied hypothetical reimbursements will be assessed relative to reimbursements under the traditional approach (which the case study school districts are using now). Then, differences between costs and reimbursements will be compared for the traditional approach and Provision 4 to identify situations that render Provision 4 more or less attractive to school districts.