Developing and Evaluating Methods for Using American Community Survey Data to Support the School Meals Programs: Interim Report (2010)

The American Community Survey (ACS) is the only survey that might be large enough to estimate numbers of students eligible for free or reduced-price meals in school attendance areas to use in a new Provision 4 for establishing claiming percentages for reimbursement of school meal costs by the U.S. Department of Agriculture (USDA). The panel will consider estimates from the ACS in light of accuracy, timeliness, and geographical coverage. Assessment of these properties, and possible corrections of shortcomings, requires additional data sources or data products. This chapter begins with a description of the ACS and follows with descriptions of the other data sources that will be used in the study. Table 3-1 lists each data source or product and notes the primary issue that each will be used to address.

The chapter describes the administrative data collected by the Food and Nutrition Service (FNS) in support of the school meals programs, as well as information about schools provided by the National Center for Education Statistics (NCES). These data sources will be used for deriving and evaluating any estimates obtained using methods proposed by this panel that could potentially support a new Provision 4. The primary use of the FNS and NCES administrative data will be to assess bias in estimates based on the ACS.

One of the known reasons for potential bias in ACS-based estimates is that ACS measures annual income, whereas eligibility for school meals programs is based on monthly data. The Survey of Income and Program Participation (SIPP) is the source of information that could be used to adjust for any such bias. Hence this chapter describes SIPP.

It is expected that ACS-based estimates for small areas will be subject to relatively large sampling errors. One way to address sampling errors is to make use of so-called small-area estimates. The Census Bureau manages the Small Area Income and Poverty Estimates (SAIPE) Program, which produces estimates for the number of school-age children whose families have income no greater than the poverty threshold for all school districts in the country. As discussed in detail in Chapter 5, the panel will consider whether the methodology used to prepare SAIPE estimates can be modified to derive the eligibility estimates needed for the school meals programs. This chapter provides an introduction to the SAIPE program.

The panel is charged with developing methodology to produce estimates for school districts and for school attendance areas. The geographic data involved are the school district boundaries updated and maintained by the U.S. Census Bureau, as well as local school attendance boundary information that will be provided by the case study districts. This chapter describes the geographic support of the ACS and other surveys conducted by the Census Bureau. Data from the case study districts will also be used to assess the accuracy of estimates prepared by the panel and may be used to address timeliness issues. The data to be collected from the case study districts are described in more detail in Chapter 4.

The American Community Survey is a new continuous survey that collects data on income, family composition, and other content that was previously ascertained once every 10 years from the long-form sample of the decennial census of population. After a decade of testing and development, the ACS became fully operational in 2005 for households; people living in group quarters were added beginning in 2006. With the advent of the ACS, the 2010 and future censuses will include only the “shortform” items of age, sex, race, ethnicity, relationship to householder, and owner/renter status (see National Research Council, 2007).

The ACS samples 250,000 housing unit addresses every month from the Census Bureau’s Master Address File, for a total of 3 million housing unit addresses every year. Each month, about half of the households receiving a questionnaire in the mail fill it out and mail it back in; nonresponding households for which telephone numbers can be obtained are contacted using computer-assisted telephone interviewing (CATI). A one-third sample (approximately) of the remaining nonrespondents is designated for follow-up using computer-assisted personal interviewing (CAPI). High overall response rates have been achieved for the ACS. The response rate obtained by adding mailback and CATI respondents

together with a weighted estimate of respondents in the CAPI subsample was approximately 97.9 percent in 2008.¹

The goal of the ACS is to provide small-area estimates similar to those provided by the census long-form sample. Because the ACS sample is spread out over time, the data must be accumulated over months and years to provide reliable estimates. Every year beginning in late 2005, the Census Bureau releases ACS 1-year period estimates for states, counties, cities, school districts, and other geographic areas with at least 65,000 people. Beginning in late 2008, the Census Bureau also releases ACS 3-year period estimates for areas with at least 20,000 people. Finally, beginning in late 2010, the Census Bureau will release ACS 5-year period estimates for all geographic areas in Census Bureau databases, including block groups, census tracts, and small cities, towns, and school districts.

The ACS data provide the opportunity for constructing estimates of students who are eligible for free meals, reduced-price meals, and full-price meals for the attendance areas of schools, groups of schools, and school districts. Most school districts in the United States are small in population size. Thus, of the 14,125 school districts currently in the Census Bureau’s geographic inventory, only 892 had 65,000 or more residents in the 2000 census, and only 3,227 had more than 20,000 residents. Moreover, in medium- and large-sized school districts, attendance areas for individual schools or groups of schools are small. Because ACS estimates are not provided for school attendance areas, estimates for these would need to be constructed by aggregating blocks to approximate the school boundaries as closely as possible.

There are numerous technical and procedural issues to consider in using the ACS for deriving eligibility estimates and establishing claiming percentages for the school meals programs. Four of the most important issues are (1) constructing geographic areas and determining school attendance, (2) determining eligibility from ACS data, (3) minimizing sampling variability, and (4) enhancing timeliness. Subsequent chapters discuss in detail these issues and our approaches to investigating them.

1. Constructing Geographic Areas and Determining School Attendance. The ACS collects information about school attendance: whether attending within the last 3 months, public or private, and grade (or grade range). Hence, for a given public school attendance area, it is possible to obtain estimates for students who live in that area, attend public school, and are in approximately the appropriate grade range. However, as discussed at the end of this chapter, it can be challenging to align the

geographic information of the ACS (census blocks) with the geographic information used by school districts to identify school attendance areas. Moreover, as shown by Saporito and Sohoni (2007), charter, magnet, and other such schools may draw students from throughout a school district, altering the distribution of students attending the neighborhood public schools.

1. Determining School Meals Eligibility from ACS Data. The ACS collects data on gross money income for household members ages 15 and older, so it is possible to compare a family’s income with 130 percent and 185 percent of the applicable poverty guideline to determine its income eligibility status. However, the ACS income data pertain to the previous 12 months, whereas eligibility for the school meals programs is based on a current month’s income.

   The ACS also collects information about the receipt of Supplemental Nutrition Assistance Program (SNAP) benefits and the receipt of other welfare income. The receipt of SNAP benefits confers categorical eligibility for free school meals. However, other welfare income is “the amount of any public assistance or welfare payments from state or local welfare offices.” Although it might include payments from Temporary Assistance for Needy Families (TANF), which confers eligibility, it might also include payments from programs that do not confer eligibility.

   Another challenge in using the ACS data on benefit receipt and, more generally, ACS income data, is reporting error. The ACS is no exception to the well-known fact that survey respondents tend to underreport sources of income, including substantial underreporting of public assistance benefits (see Czajka and Denmead, 2008; Meyer and Sullivan, 2009).

2. Minimizing Sampling Variability. As illustrated in Chapter 5, ACS estimates can have large sampling errors for small geographic areas, including nearly all school attendance areas and many of the nation’s school districts. Large sampling errors would make it difficult for a district to assess the attractiveness of proposed Provision 4. Moreover, after adoption of Provision 4, variability in estimates could cause a district to be substantially under- or over-reimbursed from year to year.

   Later chapters discuss approaches to reducing sampling variability. A leading candidate is to use small-domain estimation methods, as in the Census Bureau’s SAIPE program, to improve the precision of estimates through statistical modeling and the incorporation of auxiliary data. Another approach is to aggregate data over time (as in the production of ACS 5-year period estimates) or over geographic areas (as in the aggregation of schools to form school groups within a district).

4. Enhancing Timeliness. At present, under Provision 2, percentages of meals served to free, reduced-, and full-price approved students in a base year are used to establish claiming percentages for a minimum of 3 future years. Under Provision 3, reimbursement for 4 future years is based on the number of meals served by category during the base year multiplied by a factor to adjust for changes in enrollment. If ACS 5-year period estimates replaced usage-based percentages for a base year, the claiming percentages would be more out of date than under current practice for the base and future years. Statistical modeling and adjustment methods to ameliorate this problem are discussed in later chapters.²

FNS collects state-level counts related to the school meals programs on the Report of School Program Operations, Form FNS-10, which is completed by the relevant state agency. The form has two parts. Part A, required to be submitted monthly, shows the number of meals served in the state under the school lunch and breakfast programs by category (free, reduced price, full price), the total number of meals, and the average daily number of meals. This information is used to compute state-level reimbursements for the school meals programs. Part B is to be completed once a year. In October, states report the number of meals served by category in private schools and residential child care institutions (RCCI). Also included are counts of pubic schools, private schools, and RCCIs that participate in the school meals programs (by program) and the enrollment of those schools. For the National School Lunch Program (NSLP), the form shows the number of students approved for free lunches and the number approved for reduced-price lunches.

To complete Form FNS-10, a state agency obtains the necessary information from school districts. Data are required to be kept for 3 years. FNS provides summary information on its website at http://www.fns.usda.gov/pd/cnpmain.htm [accessed May 2010].

FNS collects data on verification activities on the School Food Authority Verification Summary Report, Form FNS-742. The form is available at http://www.fns.usda.gov/cnd/Governance/Forms/SFA_Verification_Summary.pdf [accessed May 2010]. With few exceptions, each school district that oper-

ates the NSLP or School Breakfast Program (SBP) must report the information on this form annually. Section I of the form obtains information as of the last operating day in October. Included is the number of schools operating the NSLP or the SBP and the enrollment of those schools. The total number of free certified and reduced-price certified students are reported. In addition, the numbers of free certified students are separately identified as (1) not subject to verification (directly certified, homeless liaison list, income eligible Head Start, pre-K Even Start, residential students in RCCIs, nonapplicants approved by local officials); (2) certified based on a SNAP/TANF/FDPIR (Food Distribution Program on Indian Reservations) case number submitted on an application; (3) certified based on income reported on an application; and (4) certified in Provision 2 and Provision 3 schools not operating in a base year. The number of reduced-price certified students is also separately identified for Provision 2 and Provision 3 schools not operating in a base year.

Section II of Form FNS-742 provides information about verification. For each outcome, three counts are reported: number of free certified students based on the SNAP/TANF/FDPIR case number submitted on the application; the number of free certified students based on income provided on the application; and the number certified for reduced-price meals. The reported outcomes of verification include no change, responded and changed to free, responded and changed to reduced price, responded and changed to full price, did not respond, and reapplied and reapproved on or before February 15. For each outcome, the form also collects data on the number of applications and the number of students. Data from Form FNS-742 are maintained by FNS and used to prepare summary reports of verification activities.

The Common Core of Data (CCD), a program of the U.S. Department of Education’s NCES, annually conducts five surveys to collect fiscal and nonfiscal data about all public schools, public school districts, and state education agencies in the United States. The CCD provides an official listing of public elementary and secondary schools and school districts in the nation, which is used to select samples for other NCES surveys and provides basic information and descriptive statistics on public elementary and secondary schools and schooling in general. The data, supplied by state education agency officials, include information that describes schools and school districts, including name, address, and phone number; information about students and staff, including demographic characteristics; and fiscal data, including revenues and current expenditures. Most of these data are obtained from administrative records, presumably the same ones used by states as the basis for completing FNS forms.

For purposes of this study, the most relevant information from the CCD are the school and school district counts of enrolled students, students certified for free lunches, and students certified for reduced-price lunches. The school district fiscal data from the School District Finance Survey may also be useful. These data include revenues by source and expenditures by function and subfunction (including school meals).

Other potentially relevant NCES data include special tabulations of the ACS by school district geography prepared by the Census Bureau for NCES. These tabulations provide detailed demographic characteristics of the nation’s public school systems. However, the economic characteristics tables that present data related to poverty levels allow one to look at only those below the poverty level and at or above the poverty level and not the near-poverty levels (130 percent and 185 percent) relevant to school meals eligibility.³

SIPP is the only major household survey that collects information on both annual income and changes in monthly income. Hence, it may provide an important source of information for the panel concerning the relationship between school meals eligibility estimated from annual income (as measured by the ACS or other surveys) and eligibility estimated from monthly income, as is done in the NSLP and the SBP. Moving forward, the major concern with the redesigned SIPP is whether the event history calendar method with annual interviews will capture changes in income in the same way as the current design with 4-month interviews (see National Research Council, 2009).

SIPP is a continuing program of the U.S. Census Bureau, which began interviewing for the survey in late 1983 and is planning to introduce a major redesign in 2013. Under its current design, in which members of sampled households (panels) are interviewed every 4 months for 3 or 4 years, SIPP not only provides detailed annual and subannual information on income by source for a representative sample of U.S. households, but also tracks changes in program eligibility and participation for the household members as their incomes and other circumstances change. Programs covered in SIPP include SNAP, NSLP, SBP, TANF, and many others. In addition, SIPP collects data on taxes, assets, liabilities, labor

force participation, general demographic characteristics, and many special topics related to families’ economic circumstances.

The survey design is a continuous series of national panels, each representing the U.S. civilian noninstitutionalized population. Over the years, panels have varied in sample size, number of interview waves, and other features. For the 1984-1993 period, a new panel of households was introduced each February. Subsequent panels have not overlapped; they include a 4-year panel beginning in 1996, a 3-year panel beginning in 2001, a 4-year panel beginning in 2004, and a 4-year panel beginning in 2008. A new, redesigned panel of about the same size as the 2008 panel—45,000 households—is to be introduced in 2013 and followed for 3 or 4 years.

The current SIPP content is built around a “core” of labor force, program participation, and income questions that are repeated at each wave of interviewing, with supplemental modules on particular topics asked one or more times per panel. The survey uses a 4-month recall period, with approximately the same number of interviews being conducted in each month of the 4-month period for each wave. Interviews are conducted by personal visit for the first two interview waves and telephone thereafter using a computer-assisted interview on a laptop computer. Data are currently released in cross-sectional core and topical module files for each interview wave. Core files are available through Wave 2 of the 2008 panel; topical module files are available through Wave 8 of the 2004 panel (see http://www.sipp.census.gov/sipp_ftp.html#sipp [accessed May 2010]).

The planned redesign of SIPP will change the interviewing cycle from every 4 months to once a year. Each annual interview will include the core question content on income, employment, program participation, and demographic characteristics using an event history calendar to facilitate recollection of monthly information for the previous year. Some content previously in topical modules will be included, and government agencies may pay for special supplements.

The No Child Left Behind (NCLB) Act of 2001 directs the U.S. Department of Education to distribute Title I basic and concentration grants directly to school districts on the basis of the most recent estimates of school-age children in poverty available from the Census Bureau. These estimates are produced by the Census Bureau’s SAIPE program. SAIPE

estimates, which were first developed in the late 1990s (see National Research Council, 2000a, 2000b), are currently based on data from the ACS, the 2000 census, SNAP, aggregated federal income tax data, and a series of statistical models. The 2007 and 2008 estimates correspond with 2007-2008 school district boundaries, while the 2009 and 2010 estimates will correspond with 2009-2010 school district boundaries.

Annual SAIPE estimates of related children ages 5-17 living in families with income below the poverty line are used in allocating $14 billion to school districts for Title I of NCLB.⁵ As described in more detail below, the school district estimation process uses the estimated number of school-age children in poverty in a county from a statistical model and the estimated number of children in households below the poverty line based on federal income tax returns for each school district (or part of a district) in that county. The county-level model combines the results of a regression equation with direct (not model-based) 1-year ACS estimates, controlled to estimates from a state-level model. The county-level and state-level regression equations use administrative records data and 2000 census long-form sample estimates to predict school-age poverty for each county or state.

The SAIPE model estimates are produced for a given year with about a 1-year time lag—for example, 2008 estimates were released in December 2009; they incorporated administrative records information for 2007. This time schedule is only a few months later than the release of direct ACS estimates. The SAIPE model-based estimates have the advantage of reducing mean squared error compared with direct estimates for small geographic areas; however, their accuracy depends on the validity of the underlying model and may vary for different kinds of areas. SAIPE estimates are not available for census tracts or block groups, and they pertain to the official statistical poverty level and not the 130 percent and 185 percent ratios of income to the poverty guidelines that determine eligibility for free or reduced-price school meals. Therefore, as discussed in Chapter 5, we will investigate the development of SAIPE-like models for deriving estimates of students who are eligible for free or reduced-price meals in the school meals programs.

The SAIPE estimation process involves several steps. First, state-level poverty estimates are made for ages 0-4, 5-17, 18-64, and 65 and older.

There are two equations for ages 5-17, one for all children, and one for related children.⁶ These estimates are based on a weighted average of direct ACS estimates and a prediction from a regression model. The dependent variable in the model is the ACS 1-year direct estimate.⁷ Independent variables include the poverty rate from the 2000 census, the tax return poverty rate, the tax return nonfiler rate, a SNAP participation ratio, and the Supplemental Security Income (SSI) receipt rate. The regression-based and ACS-based estimates are combined, weighting each based on the uncertainty associated with it, with the more uncertain estimate having the smaller weight. The poverty ratios obtained are multiplied by population estimates to provide counts of the number of people in poverty, which are controlled to sum to the official national total from the ACS.

Second, county-level estimates are made. Like the state estimates, the county estimates are based on a weighted average of direct ACS estimates and regression predictions. The dependent variable in each regression model is the log of the number of people in a particular age category in that county as measured by the ACS. Predictor variables (appropriately transformed) include the number of child exemptions claimed on tax returns of people in poverty, the number of child exemptions on tax returns, the number of SNAP benefit recipients, the resident population, and the estimated number of people in the age category in poverty according to the 2000 census. Weighting of ACS and model estimates is based on the uncertainty associated with each estimate. For counties that have no ACS sample observations in the age category, the weight on the model’s prediction is 1. County estimates are adjusted so they sum to the state total from the previous step.

State- and county-level estimates are provided along with estimates of their uncertainty, measured as a margin of error. The margin of error is the half-width of a 90 percent confidence interval for an estimate and is equal to 1.645 times the standard error. The standard errors represent “uncertainty” arising from two major sources: ACS sampling variation and “lack of fit” of the regression model to what the ACS measures. In general, the former is larger than the latter.

Finally, school district-level estimates are made. For each school district, estimates are derived for the total population, children ages 5-17, and related children ages 5-17 in families in poverty. Margins of error are not

currently provided for school district-level estimates, although research on the estimation of standard errors for them is under way.

The 2008 school district estimates were based on the 2008 county estimates and tabulations of poverty from the 2000 census and income tax data for tax year 2007 from the Internal Revenue Service (IRS), using school district boundaries corresponding to school year 2007-2008. By construction, the SAIPE school district estimates are arithmetically consistent with the SAIPE county and state estimates.

For each school district, SAIPE estimates pertain to all resident school-age children (ages 5-17 inclusive), whether enrolled in public or private school or not enrolled. Where two districts divide the children of an area between them by grade, the estimates do so as well. In particular, some states have areas with separate “elementary” and “secondary” school districts, each exclusively responsible for providing education in some grades in their shared territory.⁸ In these areas, data for school-age children are allocated between districts on the basis of the grade range of the district and the grade assigned to the child.

In most areas, “unified” districts are responsible for providing education for all elementary and secondary grades—either by operating schools themselves or by purchasing instruction from neighboring school districts—for all residents of their territory. In these areas, data for all children ages 5-17, inclusive, are tabulated in the district in which they reside.

There are also some states that have school districts with different grade ranges in different parts of a district’s territory.⁹ In most cases, these are districts that are unified in part of their domain and secondary in the rest. The final tabulations and estimates reflect the combination of data honoring these distinctions.

Grade ranges for each district are collected during the boundary update and supplemented with phone calls to districts. SAIPE attempts to assign a single grade range to each district that, in the case of spatially overlapping districts, leaves no grade unclaimed and no grade claimed by more than one district. Occasionally, the pattern of grade ranges of over-

lapping districts does not permit each grade to be assigned to exactly one and only one district. In these few instances, three rules are applied:

1. If a “unified” district is present, data for children whose assigned grade is claimed by two districts or not claimed by either are allocated to the unified district.

2. If “elementary” and “secondary” districts are present, data for children whose assigned grade is claimed by both are allocated to the secondary district.

3. If “elementary” and “secondary” districts are present, data for children whose assigned grade is claimed by neither are allocated to the elementary district.

The SAIPE program procedure for deriving school district estimates works with geographical units called “school district–county pieces.” These pieces are defined as the intersections of school districts and counties—that is, all of a district if it does not cross county boundaries and each county part separately for districts that do. If a school district has territory in two counties, for example, estimates are made for the two parts separately and then combined.

For each school district piece, the tax-based child poverty rate is estimated, by using federal tax information obtained from the IRS, as the product of the county poverty rate for related children ages 5-17 and the ratio of the share of county “child tax–poor exemptions” to the share of “child tax exemptions” for the school district piece. For the 2008 school district estimates, the number of child tax exemptions and the number of child tax–poor exemptions were obtained from tax year 2007 IRS income tax data. For the 2007 school district estimates, tax year 2006 IRS income tax data were used. “Child tax–poor exemptions” are defined as the number of child tax exemptions on returns whose adjusted gross income falls below the official poverty threshold for a family of the size implied by the number of exemptions on the tax return. Because the age of each child is not reported on the income tax return, 2000 census estimates are used to adjust the IRS estimates to reflect the grade range of a school district (U.S. Census Bureau, 2007).

The school district piece poverty rate is multiplied by the official estimate of the relevant child population for the school district piece to obtain a poverty count for the school district piece. These counts are then ratio adjusted to agree with the SAIPE county estimates for the number of children ages 5-17 in poverty. Finally, the adjusted school district piece estimates are further adjusted using “controlled rounding” to obtain integer

values while still ensuring that pieces add up to SAIPE county totals. The final step is to reassemble the school district pieces into school districts by adding their controlled and rounded numbers of children in poverty.

For ACS and other surveys conducted by the Census Bureau, the corresponding geographic support is the Census Bureau’s Topologically Integrated Geographic Encoding and Referencing (TIGER) database, which is a digital map of streets and other features. The accuracy of TIGER has recently been substantially improved through a major initiative in preparation for the 2010 decennial census, so that positional errors are now in the 5-meter range for streets and other major features. Geographic areas that are available in TIGER include blocks, block groups, census tracts, school districts, small cities, towns, counties, and states.

SAIPE provides poverty estimates for all school districts that are in the TIGER database, updated by the School District Review Program, which was conducted most recently in 2008. The next update will be completed in 2010. SAIPE also tabulates and produces estimates for all occupied areas not assigned to any school district in a county. These areas are referred to as “balances” of the counties in which they occur, whether they compose a single compact area or not. Although estimates for “balance of county” areas are not published on the SAIPE website, they are provided to the U.S. Department of Education for implementing provisions of NCLB and are available upon request.

The panel is developing a methodology that could produce estimates for school attendance areas of students eligible for free or reduced-price school meals for use in a new Provision 4 for federal reimbursement of meal costs. Because the Census Bureau does not maintain geographic data on school attendance-area boundaries, to obtain and evaluate such estimates under its proposed methodology, the panel will need to provide digitized school attendance-area boundaries of sufficient accuracy to the Census Bureau. The panel expects to obtain digitized school attendance boundaries directly from case study districts (see Chapter 4).

Direct estimates for schools or groups of schools will probably have to be derived by aggregating ACS block sample data and weighting it. For some schools, however, the attendance boundaries will run through blocks and statistical algorithms for splitting block groups may need to be developed and evaluated. Goodchild, Anselin, and Deichmann (1993) describe methods for such approximations. Geverdt (2005) documents the work done to develop digitized boundaries in Philadelphia for developing estimates of school meal eligibility based on the 2000 census.

The School Attendance Boundary Information System (SABINS) (National Science Foundation, 2009) is a 2-year proposal by Salvatore Saporito that received funding from the National Science Foundation in 2009. The project is to establish a spatial database of school attendance boundaries for the 800 most populous school districts in the country. SABINS data are planned to be distributed via the National Historic geographic information system website (see http://www.nhgis.org/ [accessed May 2010]). The intention is that these boundaries would be compatible with the TIGER database to facilitate social science research. If this project is successful, it may make it easier for school districts to obtain accurate digitized school attendance boundaries.