3

Technical Approach

This chapter presents the framework established by the panel for evaluating the use of estimates based on American Community Survey (ACS) data, describes the data and information sources and how they were used, and provides an overview of the panel’s evaluation approach. It also presents intermediate results, such as those related to the use of ACS variables to define eligible students. Results of comparisons of estimates from alternative data sources are presented in Chapter 4.

The key variables of interest for this study are the percentages of students eligible or certified for free, reduced-price, and full-price meals and the percentages of meals served to students in each eligibility category. These are the eligibility, certification, and participation percentages shown in Figure 2-1 in Chapter 2. Estimates that can be computed from the ACS are eligibility rates (with eligibility determined using ACS variables), while estimates that can be computed from administrative data are certification rates that reflect students applying and being approved or directly certified through the application, certification, and verification processes. An ultimate goal is the determination of claiming percentages that reflect participation—meals served by category—under a universal feeding option, also shown in Figure 2-1.

While the panel analyzed all eligibility, certification, and participation percentages (free, reduced price, and full price), we focused on the blended reimbursement rate (BRR) described as part of the reimbursement equations presented in Chapter 2. Looking at changes in the free, reduced-price, and full-price percentages individually can be confusing because



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 49
3 Technical Approach T his chapter presents the framework established by the panel for evaluating the use of estimates based on American Community Survey (ACS) data, describes the data and information sources and how they were used, and provides an overview of the panel's evaluation approach. It also presents intermediate results, such as those related to the use of ACS variables to define eligible students. Results of comparisons of estimates from alternative data sources are presented in Chapter 4. The key variables of interest for this study are the percentages of students eligible or certified for free, reduced-price, and full-price meals and the percentages of meals served to students in each eligibility cat- egory. These are the eligibility, certification, and participation percentages shown in Figure 2-1 in Chapter 2. Estimates that can be computed from the ACS are eligibility rates (with eligibility determined using ACS vari- ables), while estimates that can be computed from administrative data are certification rates that reflect students applying and being approved or directly certified through the application, certification, and verification processes. An ultimate goal is the determination of claiming percentages that reflect participation--meals served by category--under a universal feeding option, also shown in Figure 2-1. While the panel analyzed all eligibility, certification, and participa- tion percentages (free, reduced price, and full price), we focused on the blended reimbursement rate (BRR) described as part of the reimbursement equations presented in Chapter 2. Looking at changes in the free, reduced- price, and full-price percentages individually can be confusing because 49

OCR for page 49
50 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS they are correlated, making the impact of changes difficult to assess. The advantage of the BRR is that it depends on all three percentages in a way that is of most direct interest to districts. Specifically, it gives the impact of changes in the percentages on the bottom line-- reimbursement. In fact, it is the average reimbursement per meal. The BRR is especially useful as a summary measure for ascertaining the differences in reimbursement that result from using different percentage distributions (eligible students, certified students, or meals served) as claiming percentages. Nonetheless, workshop participants told the panel that to consider participating in the ACS Eligibility Option (AEO), they would need to see all estimates (per- centages of students eligible for free, reduced-price, and full-price meals) in addition to the BRR and claiming percentages to help them assess whether to adopt the AEO.1 The panel's analytical results are focused throughout on school dis- tricts in which more than 75 percent of students were eligible for free or reduced-price meals in any school year from 2004-2005 through 2009-2010 because these districts are most likely to be interested in the AEO dis- trictwide. We call these districts "very high FRPL [free or reduced-price lunch]."2 Table 3-1 shows the distribution of these and other districts by size for all districts that have school meals program certification data for school year 2009-2010 from the Common Core of Data (CCD) and for which the Census Bureau derived ACS estimates. There are 1,291 such districts in the nation (about 10 percent of all districts), which enrolled nearly 13 percent of all students and 22 percent of students certified for 1Many of our analyses examine the individual free, reduced-price, and full-price percent- ages. As noted, however, the BRR is a useful way to summarize these percentages and focus attention on whether different sets of percentages substantially affect reimbursement, given that the difference of $.40 (currently) between the free and reduced-price meal reimburse- ment rates is very small relative to the difference of more than $2 between those rates and the rate for full-price meals. Based on the lunch reimbursement rates (with the $.02 increment) for 2010-2011 (see Table 2-6 in Chapter 2), the BRR with free, reduced-price, and full-price eligibility percentages of 80, 5, and 15 percent, respectively, is less than 2 percent higher than the BRR with percentages of 70, 15, and 15 percent, respectively ($2.3510 versus $2.3110). In contrast, the latter is nearly 10 percent greater than the BRR with percentages of 70, 5, and 25 percent ($2.1050). In other words, shifting 10 percent (of students or meals) from the reduced-price category to the full-price category has a much greater effect on reimbursement than shifting them to the free category. The Healthy, Hunger-Free Kids Act of 2010 explicitly acknowledges the BRR as a useful measure for analysis and decision making, requiring states to calculate and disseminate BRRs for districts for purposes of implementing and administering the Community Eligibility Option. 2The 75 percent figure was identified as a threshold for potential interest in a universal feed- ing provision in many phases of the panel's analysis. It is noted in publications by the Food Research and Action Center (see http://frac.org/newsite/wp-content/uploads/2009/05/ provision2.pdf). As discussed later, the 75 percent threshold also was mentioned by partici- pants in the panel's workshop and in its survey of Provision 2/3 districts.

OCR for page 49
TECHNICAL APPROACH 51 TABLE 3-1 Number and Percentage of U.S. School Districts* by Size and Percentage Approved for Free or Reduced-Price Meals Size Low FRPL High FRPL Very High FRPL Large Number of districts 468 305 110 Percentage of districts 3.6 2.4 0.9 Percentage of enrollment 24.1 19.2 8.7 Medium Number of districts 1,415 722 187 Percentage of districts 10.9 5.6 1.4 Percentage of enrollment 16.0 8.4 2.1 Small Number of districts 5,645 3,092 994 Percentage of districts 43.6 23.9 7.7 Percentage of enrollment 12.9 6.6 1.9 Total Number of districts 7,528 4,119 1,291 Percentage of districts 58.2 31.8 10.0 Percentage of enrollment 52.9 34.1 12.8 NOTE: FRPL = free or reduced-price lunch. *All school districts in the United States with Common Core of Data (CCD) free or reduced- price meals certification data for 2009-2010 and American Community Survey (ACS) esti- mates. Large districts have 1-year estimates. Medium-sized districts have 3-year estimates, but do not have 1-year estimates. Small districts have only 5-year estimates. SOURCE: Prepared by the panel. free or reduced-price meals. We also considered districts with more than 50 percent but never more than 75 percent of students eligible for free or reduced-price meals in the school years from 2004-2005 through 2009- 2010 because these districts might be interested in the AEO for a subset of schools. We call these districts "high FRPL." There are 4,119 such districts nationwide (32 percent of districts), enrolling 34 percent of all students and 44 percent of students certified for free or reduced-price meals. The data collected on form FNS-742 (described in more detail later) show that only 431 of these school districts were operating under Provi- sion 2 or 3, not in a base year, in 2009-2010. Of these, 296 were operating under Provision 2 or 3 districtwide, and 135 were operating under Provi- sion 2 or 3 for only some schools. Of those operating under Provision 2 or 3 districtwide, 79 percent had an FRPL percentage greater than or equal to 75, 10 percent had an FRPL percentage greater than or equal to 50 but less than 75, and 12 percent had an FRPL percentage less than 50. Of the districts where Provision 2 or 3 was implemented for only some schools, 32 percent had a district-level FRPL percentage greater than or equal to 75, 45 percent had an FRPL percentage greater than or equal to 50 percent but less than 75, and 23 percent had an FRPL percentage less than 50. We also refer to districts as large, medium, and small, depending on whether they had 1-year, 3-year, and 5-year ACS direct estimates available (population

OCR for page 49
52 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS of at least 65,000); 3-year and 5-year (but not 1-year) estimates available (population between 20,000 and 64,999); or only 5-year estimates avail- able (population under 20,000). SOURCES OF DATA ON ELIGIBILITY AND PARTICIPATION The ACS is the only national survey that may be large enough for use in estimating numbers of students eligible for free or reduced-price meals in school districts and school attendance areas under a potential new provision. The panel considered the use of estimates from the ACS in terms of their accuracy, timeliness, and geographic coverage. Assessment of these properties and the development of possible corrections for any shortcomings required comparison with additional data sources and data products. This section begins with a description of the ACS direct and model-based estimates and then describes the other data sources the panel compared with the ACS: the administrative data collected by the Food and Nutrition Service (FNS) in support of the school meals programs, admin- istrative information about schools and school districts collected and pro- vided by the National Center for Education Statistics (NCES) in the CCD, and school-level data provided to the panel by our case study districts. These data sources were used to assess not only any systematic differences between ACS and administrative estimates, but also the precision, tempo- ral stability, and timeliness of ACS estimates for all school districts in the country and for the schools in our case study districts. The American Community Survey The ACS is a continuous survey used to collect data on income, fam- ily composition, and other individual and household characteristics that previously were gathered once every 10 years from the long-form sample of the decennial census of population. After a decade of testing and devel- opment, the ACS became fully operational in 2005 for households; people living in group quarters were added beginning in 2006. With the advent of the ACS, the 2010 census (as will be true of future censuses) included only the short-form items of age, sex, race, ethnicity, relationship to house- holder, and owner/renter status (see National Research Council, 2007). The ACS samples about 240,000 housing unit addresses every month from the Census Bureau's Master Address File, for a total of nearly 2.9 mil- lion housing unit addresses every year (increased to 295,000 addresses per month in June 2011). Each month, about half of the households receiving a questionnaire in the mail fill it out and mail it back; nonresponding households for which telephone numbers can be obtained are contacted using computer-assisted telephone interviewing (CATI). A one-third sam-

OCR for page 49
TECHNICAL APPROACH 53 ple (approximately) of the remaining nonrespondents is designated for follow-up using computer-assisted personal interviewing (CAPI). High overall response rates have been achieved for the ACS. The response rate, obtained by adding mailback and CATI respondents together with a weighted estimate of respondents in the CAPI subsample, was approxi- mately 98 percent in 2009.3 The goal of the ACS is to provide small-area estimates similar in pre- cision to but more timely than those provided by the census long-form sample. Because the ACS sample is spread out over time, the data must be accumulated over months and years to provide reliable estimates for small areas. In late 2006 (for calendar year 2005), the Census Bureau began releas- ing ACS 1-year estimates for states, counties, cities, school districts, and other geographic areas with at least 65,000 people. In late 2008, the Census Bureau began releasing ACS 3-year estimates for areas with at least 20,000 people. Finally, in late 2010, the Census Bureau began releasing ACS 5-year estimates for all geographic areas in Census Bureau databases, including block groups, census tracts, small cities, towns, and school districts. The ACS data provide an opportunity to construct estimates of stu- dents who are eligible for free, reduced-price, and full-price meals for the attendance areas of schools, groups of schools, and school districts. Most school districts in the United States are small in population size. Thus of the 13,777 school districts for which ACS estimates were released in fall 2011, only 985 had 65,000 or more residents according to the July 2010 Census Bureau population estimates, and only 3,411 had more than 20,000 residents.4 Moreover, even in medium-sized and large school districts, attendance areas for individual schools or groups of schools are small. Because ACS estimates are not provided for school attendance areas, estimates for these areas would need to be based on boundary informa- tion or lists of census blocks provided to the Census Bureau by a state or local education agency. Numerous challenges must be addressed before the ACS can be used to derive eligibility estimates and establish claiming percentages for the school meals programs. Five of the most important issues are (1) con- structing geographic areas to represent school attendance areas; (2) deter- mining eligibility using ACS variables; (3) assessing systematic differences between ACS and administrative estimates; (4) assessing levels of vari- ability, temporal stability, and timeliness; and (5) accounting for participa- tion. Subsequent sections of this chapter address the first two issues and outline the empirical analyses needed to address the last three. Results of the data comparisons and analyses are provided in Chapter 4. 3See http://www.census.gov/acs/www/methodology/sample_size_and_data_quality/. 4See http://www.census.gov/acs/www/data_documentation/areas_published/.

OCR for page 49
54 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS Small Area Income and Poverty Estimates (SAIPE) and ACS Model-Based Estimates The No Child Left Behind Act of 2001 directed the U.S. Department of Education to distribute Title I basic and concentration grants directly to school districts on the basis of the most recent estimates of school-age chil- dren in poverty available from the Census Bureau. These estimates, from the SAIPE Program, were first developed in the late 1990s (see National Research Council, 2000a,b) and are currently based on data from the ACS, the 2000 census, the Supplemental Nutrition Assistance Program (SNAP, formerly the Food Stamp Program), aggregated federal income tax data, and a series of statistical models. The 2009 and 2010 SAIPE estimates cor- respond to 2009-2010 school district boundaries. Annual SAIPE estimates of related children aged 5-17 living in fami- lies with income below the poverty line are used in allocating $14 billion to school districts for Title I of the No Child Left Behind Act.5 The school district estimation process uses the number of school-age children in poverty in a county estimated from a statistical model and the estimated number of children in households below the poverty line based on fed- eral income tax returns for each school district (or part of a district) in that county. The county-level model combines the results of a regression equation with direct (not model-based) 1-year ACS estimates, controlled to estimates from a state-level model. The county- and state-level regres- sion equations use administrative records data and estimates from the 2000 census long-form sample to predict numbers of school-age children living in poverty for each county or state.6 The SAIPE model estimates are produced for a given year with about a 1-year time lag; for example, the 2009 estimates were released in Decem- ber 2010, incorporating administrative records information for 2008. This timing is only a few months later than the release of direct ACS estimates. As a result, SAIPE estimates are considerably more timely than the 5-year ACS estimates, the only other available option for small school districts. The SAIPE model-based estimates have the advantage of reducing mean- squared error relative to direct estimates for small geographic areas; how- ever, their accuracy depends on the validity of the underlying model and 5Related children are people under age 18 and related by birth, marriage, or adoption to the householder of the housing unit in which they reside; foster children, other unrelated individuals under age 18, and residents of group quarters under age 18 are not considered related children. 6It will not be possible to update the 2000 census variables in the state and county models because the 2010 census ascertained only basic demographic information on households, with the ACS obtaining the detailed socioeconomic data formerly included on the census long form.

OCR for page 49
TECHNICAL APPROACH 55 may vary for different kinds of areas. SAIPE estimates are not available for census tracts or block groups, and they pertain to the official statistical poverty level and not the 130 percent and 185 percent ratios of income to the poverty guidelines that determine eligibility for free or reduced-price school meals, respectively. The panel collaborated with the Census Bureau, which agreed to adapt the SAIPE approach and provide model-based ACS estimates of the percentages of students eligible for free and reduced-price meals in each school district in the United States and in the school attendance areas in the case study districts. The methodology developed to provide these model-based estimates is described in Appendix C, and the estimates are evaluated in Chapter 4. Administrative Data Both FNS and NCES in the Department of Education collect data from school districts nationwide that can be considered a benchmark for comparisons with the estimates from the ACS. FNS has two relevant data collection mechanisms--form FNS-742, School Food Authority Verifica- tion Summary Report (information for school districts), and form FNS-10, Report of School Program Operations (information at the state level only). NCES provides detailed information through the CCD, including data on enrollment, number of students certified for free meals, and number certified for reduced-price meals, for all public school districts and public schools in the country. The panel also collected detailed administrative data concerning enrollment, certification, and meals served from our five case study districts. Form FNS-7427 collects data on verification activities. With few exceptions, each school district that operates the National School Lunch Program (NSLP) or School Breakfast Program (SBP) must report the information on this form annually. Section I of the form obtains informa- tion as of the last operating day in October. Included are the number of schools operating the NSLP or SBP and the enrollment of those schools, the total number of free-certified and reduced-price-certified students, and the number of free-certified students who are separately identified as (1) not subject to verification (directly certified, homeless liaison list, income-eligible Head Start, pre-K Even Start, students in residential child care institutions [RCCIs], and nonapplicants approved by local officials); (2) certified based on a SNAP, Temporary Assistance to Needy Families (TANF), or Food Distribution Program on Indian Reservations (FDPIR) case number submitted on an application; (3) certified based on income 7The form is available at http://www.fns.usda.gov/fns/forms.htm.

OCR for page 49
56 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS reported on an application; and (4) certified in Provision 2 and 3 schools not operating in a base year. The number of reduced-price-certified stu- dents also is separately identified for Provision 2 and 3 schools not oper- ating in a base year. Section II of form FNS-742 provides information about verification. The reported outcomes of verification include no change, responded and changed to free, responded and changed to reduced price, responded and changed to full price, did not respond, and reapplied and was reapproved on or before February 15. For each outcome, three counts are reported: the number of free-certified students based on the SNAP/ TANF/FDPIR case number submitted on the application; the number of free-certified students based on income provided on the application; and the number of reduced-price-certified students based on income. The form also collects data on the number of applications and the number of students for each outcome. Data from form FNS-742 are maintained by FNS and are used to prepare summary reports of verification activities. Form FNS-10 collects state-level counts related to the school meals programs and is completed by state agencies. The form has two parts. Part A, which must be submitted monthly, obtains the number of meals served in the state under the NSLP and SBP by category (free, reduced price, full price), the total number of meals, and the average daily number of meals. This information is used to compute state-level reimbursements for the school meals programs. Part B is to be completed once a year. In October, states report the number of meals served by category in pri- vate schools and RCCIs. Also included are counts of public schools, private schools, and RCCIs that participate in the school meals programs (by program) and the enrollment of those schools. For the NSLP, the form shows the number of students approved for free lunches and the number approved for reduced-price lunches. To complete form FNS-10, a state agency obtains the necessary infor- mation from school districts. Data must be kept for 3 years. FNS provides summary information on its website at http://www.fns.usda.gov/pd/ cnpmain.htm. Form FNS-10 was the only comprehensive source of par- ticipation information available to the panel, but as noted, it is available only at the state level. The CCD, a program of NCES, conducts five census operations annually to collect fiscal and nonfiscal data on all public schools, public school districts, and state education agencies in the United States. It provides an official listing of public elementary and secondary schools and school districts in the nation, which is used to select samples for other NCES surveys, and it provides basic information and descrip- tive statistics on public elementary and secondary schools and school- ing in general. The data, supplied by state education agency officials,

OCR for page 49
TECHNICAL APPROACH 57 include information about schools and school districts: name, address, and phone number; information about students and staff, including demographic characteristics; and fiscal data, including revenues and current expenditures. Most of these data are obtained from administra- tive records, presumably the same ones used by states as the basis for completing FNS forms. For purposes of this study, the most relevant data from the CCD are the school and school district counts of enrolled students and numbers of students certified for free- and reduced-price meals. The CCD also con- tains demographic variables (race and ethnicity, English-language-learner status) that were used in the panel's analysis. Case Study Districts The panel invited six school districts to participate in this study as case studies, and five agreed. A district could be considered for partici- pation if it had taken applications for the school meals programs for all schools in the district for the past 5 years, had no outstanding counting/ claiming issues, was willing and able to provide digitized boundaries for the attendance areas for each school, and was willing to provide an extensive amount of school-level data for up to 6 school years. The panel decided that case studies should be selected from districts with "medium need," that is, free or reduced-price percentages of 50 to 75 percent. Another criterion was that the districts should be "heteroge- neous," that is, have at least 25 percent of schools with free or reduced- price percentages of more than 75 percent and at least 25 percent of schools with free or reduced-price percentages of less than 50 percent. The intent was to identify school districts that were likely to consider adopting the AEO for only a subset of schools. From among such dis- tricts, we wanted ones that varied in terms of enrollment but were not so small that estimates for schools or groups of schools would be too imprecise. As a rough guide, we chose to consider only the 65 medium-need, heterogeneous school districts with enrollment greater than 12,000 students based on CCD data for 2007-2008. Within this group, we planned to select 4 large school districts (enrollment of at least 25,000) and 2 medium-sized school districts (enrollment between 12,000 and 25,000). The resulting list of potential case study districts was further refined on the basis of diversity in the aggregate level of need for free and reduced-price meals, diversity in the pattern of heterogene- ity of need across schools, available information about state and district management and program operations, geographic diversity, and diver- sity in the race and ethnicity of students. The five school districts listed in Table 3-2 agreed to participate as case study districts: Austin, Texas;

OCR for page 49
58 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS TABLE 3-2 Case Study Districts Students in Schools Number of Number of Without Boundaries Participating Students (percentage of School District Schools (in thousands) enrolled)* Austin, Texas 114 83 3.0 Chatham County, Georgia 46 35 5.4 Norfolk, Virginia 56 36 10.0 Omaha, Nebraska 86 47 4.6 Pajaro Valley, California 32 19 7.4 *Omaha and Chatham County are also open enrollment districts. In open enrollment dis- tricts, many schools have geographic boundaries, but students are not required to attend neighborhood schools. SOURCE: Prepared by the panel. Chatham County,8 Georgia; Norfolk, Virginia; Omaha, Nebraska; and Pajaro Valley, California. The panel contacted state directors in the states of the potential case study districts to describe the study and ask for their assistance. With the approval of state directors, we contacted school district staff. To facilitate the development of the case studies, we obtained the support of the School Nutrition Association (SNA). The president of SNA, Dora Rivas, wrote a letter in support of the study that was included with our letters to state directors and to school district officials. From each case study district, the panel obtained digitized boundaries for school attendance areas for the most recent school year and detailed data for each school on enrollment, students approved for free and reduced- price meals, and reimbursable meals served under the SBP and NSLP by category for up to six school years. These data enabled us to conduct a limited analysis of the boundary information, to compare school-level data with CCD data for the same school, and to compare school-level data with ACS estimates to evaluate systematic differences and precision. We also used the case study data as part of our evaluation of the relationship between eligibility and participation as the basis for claiming percentages for reimbursement under the AEO and to illustrate how the AEO might work in practice. Appendix E, Part 1 describes the data collected from the case study districts and provides summary information. In addition to providing data and collaborating with the panel, the school food authority directors of the case study school districts were invited to participate in a workshop held in Washington, DC, in March 2011. The agenda for the workshop is provided in Appendix E, Part 2. 8The Chatham County School District is named Savannah-Chatham County Public School System on its public website.

OCR for page 49
TECHNICAL APPROACH 59 CONSTRUCTION OF EVALUATION DATABASES The school district-level evaluation database used by the panel con- sists of school district-level ACS direct estimates and ACS model-based estimates that the Census Bureau provided to us, together with district- level data we obtained from the CCD and form FNS-742.9 The database includes all 13,527 school districts with both ACS 5-year estimates and ACS model-based estimates. Merging ACS estimates with the CCD data was straightforward because the Census Bureau used the NCES ID to identify school districts. However, not all school districts are included in the Census Bureau's Topologically Integrated Geographic Encoding and Referencing (TIGER) files.10 Additionally, 41 districts had ACS direct esti- mates but were not in the CCD, and 227 districts had ACS model-based estimates but no 5-year ACS estimates.11 Merging with form FNS-742 data was more challenging because the ID numbers in that file vary by state and over time and are often different from NCES IDs. A recent study documenting the linkage between the FNS-742 and CCD districts in the country was helpful to the panel.12 The final school district-level evaluation database includes enroll- ment and eligibility percentages and their standard errors from ACS direct 5-year estimates (2005-2009), together with five 1-year model-based ACS estimates for calendar years 2005 through 2009 for each school dis- trict in the database. For districts with populations greater than 20,000, the database also includes three ACS direct 3-year estimates (2005-2007, 2006-2008, and 2007-2009), and for districts with populations greater than 65,000, it includes five ACS direct 1-year estimates (for 2005 through 2009). Included as well, when database records could be linked, are FNS-742 annual data for school years 2004-2005 through 2009-2010, including Pro- vision 2 or 3 participation (not in a base year) indicators, enrollment, and percentages certified by category, along with information on categorical 9The data set is named District Data School Meals.xlsx. 10TIGER is the database that associates codes for school districts and other political and statistical geographic areas with street segments and address ranges. 11The Census Bureau withheld ACS estimates for some districts--probably small districts-- because of disclosure concerns. Estimates were not withheld for any other reason (e.g., inade quate precision). No ACS model-based estimates were withheld. 12VSR-CCD Linkfile, a report delivered to FNS by Mathematica Policy Research on May 21, 2010, was provided to the panel by FNS. (VSR stands for Verification Summary Report.) The project director was Nancy Cole. The report notes reasons for differences between VSR (from the FNS-742) and CCD data. Although usually there was a one-to-one match, the primary exceptions occurred when school food authority (SFA) operations were centralized for mul- tiple school districts. Examples where this was common include Montana, New Hampshire, and New York City. In 2008-2009, there were 14,717 (unduplicated) SFAs in the VSR file, and 95.5 percent of these matched with the CCD data.

OCR for page 49
82 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS been gained on the basis of a period of low monthly income, while the 12-month income reported in the ACS was too high to meet the income eligibility criteria for the school meals programs. Additionally, broad- based categorical eligibility for SNAP (and hence free school meals) is conferred if a household qualifies for a noncash TANF or other benefit. A household (and hence students in a household) may qualify for noncash TANF benefits despite having income that exceeds the eligibility guide- lines for SNAP or the school meals programs. The panel compared ACS estimates of eligibility using our preferred definition of an economic unit and considering the household to be a single economic unit in order to evaluate the contribution of receipt of SNAP benefits and public assistance income to the percentages of children eligible for free, reduced-price, and full-price school meals. This analysis, using the 2008 PUMS file, is described in Appendix B. The addition of categorical eligibility due to receipt of SNAP benefits increases the per- centage eligible for free meals by a little more than 5 percentage points for both definitions of an economic unit. Accounting for categorical eligibility because of receipt of both SNAP benefits and public assistance income increases the percentage eligible for free meals by nearly 6 percentage points for both economic unit definitions.35 Based on these results, the panel believes that SNAP and public assis- tance should be included in ACS tabulations of eligibility to account for categorical eligibility. These variables appear to identify students who are not eligible for free meals based on ACS income alone. The only caveat is that because of underreporting of SNAP benefits and public assistance income on the ACS and other household surveys, this approach likely does not capture all such categorically eligible students. Group Quarters In addition to people living in households, the ACS includes indi- viduals who live in group quarters. These individuals are surveyed as part of the ACS, but using a separate methodology. According to the Census Bureau: Group quarters are places where people live or stay, in a group living arrangement that is owned or managed by an entity or organization 35Accounting for both SNAP and public assistance decreased the percentage eligible for reduced-price meals by about 2.5 percentage points for both economic unit definitions and the percentage eligible for full-price meals by about 3.5 percentage points. Accounting only for SNAP participation decreased the percentage eligible for reduced-price meals by 2.4 per- centage points and the percentage eligible for full-price meals by more than 2.6 percentage points under both economic unit definitions.

OCR for page 49
TECHNICAL APPROACH 83 providing housing and/or services for the residents. These services may include custodial or medical care as well as other types of assistance, and residency is commonly restricted to those receiving these services. People living in group quarters usually are not related to each other. Group quarters include such places as college residence halls, residen- tial treatment centers, skilled nursing facilities, group homes, military barracks, correctional facilities, workers' dormitories, and facilities for people experi encing homelessness. (U.S. Census Bureau, 2009:8-1) The ACS has separate categories for institutional group quarters, such as correctional facilities and nursing homes, and for noninstitutional group quarters, such as college dormitories, military barracks, migrant worker camps, and shelters. Only a subset of the noninstitutional group quarters population might include children attending public schools. The ACS survey of group quarters is based on independent state samples. For each state, a list of group quarters is constructed,36 and a sample of included facilities is selected. An ACS interviewer collects data from a sample of residents at each sampled facility. The questions asked of group quarters residents include all the person-level questions of the ACS except household relationship and only the food stamp question from the housing unit questionnaire. Group quarters facilities were not included in the 2005 sample but have been included since 2006. The Census Bureau provided the panel with useful information about the group quarters portion of the ACS, including the methods used for sample selection and estimation and the quality of the data at the state and substate levels. Because the group quarters survey is a state-based design, state-level estimates are of high quality. However, the quality of estimates at the substate level is highly variable, particularly by group quarters type. In part, this is because approximately half of all tracts listed with group quarters addresses in the Census Bureau's Master Address File sampling frame have had no sample units selected for 5 years. As a result, some areas and some types of group quarters are overrepresented in the sample, and some are underrepresented. For purposes of this study, ACS data must provide estimates of eligi- bility for the school meals programs for small geographic areas defined by individual schools or school districts. All children attending these schools are eligible to obtain school meals for free or at the reduced or full price whether they live in traditional housing units or group quarters. Students 36According to U.S. Census Bureau (2009:4-9), the ACS frame excludes domestic violence shelters, soup kitchens, regularly scheduled mobile food vans, targeted nonsheltered outdoor locations, crews of commercial maritime vessels, natural disaster shelters, and dangerous encampments for a variety of reasons, including concerns about privacy and confidentiality and the operational feasibility of repeat interviewing for a continuing survey.

OCR for page 49
84 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS attending public schools who live in group quarters (and are most likely migrant, runaway, or homeless youth) may be categorically eligible for free meals. Since the group quarters data are not reliable for small areas and since local school districts are likely to have good knowledge of stu- dents in their schools that come from group quarters, the panel concluded that group quarters students would be excluded from ACS tabulations. At our workshop, school district representatives indicated that they have information about the number of migrant, homeless, runaway, and other "group quarters" children in their jurisdictions. Our proposal, described in detail later in the report, allows districts to use local counts of categori- cally eligible children who do not live in traditional housing in computing final eligibility percentages and claiming percentages under the AEO. Summary of Conclusions on How to Estimate Eligibility for Free and Reduced- Price School Meals Based on the discussion presented above, the panel came to the fol- lowing conclusions, which are reflected in the specifications provided to the Census Bureau in our request for tabulations (see Appendix D). Conclusion 1: Providing a list of blocks associated with each grade in a school for the Census Bureau to use in constructing school atten- dance area estimates is an acceptable approach for tabulating ACS data for the school meals programs. School districts that plan to use this approach should evaluate blocks at the borders to ensure that large population groups are not assigned incorrectly. Conclusion 2: An appropriate definition of a public school student in the ACS is a person aged 20 or younger with no high school diploma or GED who attended public school within the past 3 months and was in a grade included in the school or school district.37 Conclusion 3: The appropriate income eligibility guidelines to use with ACS data are those for the school year that began in the last half of the past calendar year referenced by the ACS data. Conclusion 4: Because the ACS definition of income is sufficiently close to the definition of income for the school meals programs and the ACS measure of annual income is sufficiently close to other widely used measures of annual income, the ACS definition of income is 37This definition does not use a lower limit on the age of a student. The definition allows students in pre-kindergarten programs and kindergarten if the school includes such students.

OCR for page 49
TECHNICAL APPROACH 85 suitable for estimating income eligibility for the school meals pro- grams. It should be noted, however, that the ACS income estimates for a calendar year reflect an average of incomes received in the past 12 months spanning a 2-year period. This income measure will not be as responsive to changes in economic conditions as will income mea- sured in surveys for which the time frame covers a single calendar year, such as the CPS, and will also be less responsive than monthly income reported on applications for the school meals programs. Con- sequently, in areas where economic conditions are deteriorating (e.g., unemployment is rising), the ACS will likely understate the number of students eligible for free or reduced-price meals. Conversely, in periods of recovery, the ACS will likely overstate the number of stu- dents eligible for free or reduced-price meals. Conclusion 5: Based on the analysis performed by the panel and our interpretation of the school meals programs' definition of an economic unit, an appropriate definition of an economic unit for determining eligibility for free or reduced-price school meals should allow for multiple economic units in an ACS household. Conclusion 6: ACS data on the receipt of SNAP benefits and public assistance income should be used to account for categorical eligibility when deriving eligibility estimates for the school meals programs. Conclusion 7: ACS group quarters data should not be used in esti- mating students eligible for free or reduced-price meals. Instead, districts should be allowed to adjust ACS eligibility estimates using valid local data on students who do not live in traditional housing. Potential Limitations This section summarizes some of the limitations associated with using geographic boundaries and the ACS variables to define the public school student population in total and the percentage that is eligible for free and reduced-price school meals. Reasons for geography-related differences between actual enrollment in a school and residence in the school's catchment area include the fol- lowing: (1) there may be students who attend the school but live outside the school catchment area, (2) there may be students who live within the school catchment area but do not attend the school, (3) school boundaries change over time, and (4) the boundaries used for tabulating ACS data might not reflect the latest changes. The first two issues are related to the presence and effects of charter schools, magnet schools, open enroll-

OCR for page 49
86 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS ment policies, and other school choice programs. Choice programs could result in an understatement of the percentage of students eligible for free or reduced-price meals if such programs tend to draw the more affluent students away from their neighborhood schools. Enrollment estimates could be similarly affected. The collection of annual rather than monthly income in the ACS and the underreporting of SNAP benefits and public assistance income are likely to produce an underestimate of the percentage of students eligible for free meals when the ACS is used. As discussed below, this might necessitate some adjustment or benchmarking. The impact of the exclusion of students who live in nontraditional housing from ACS estimates will likely contribute to underestimation of both enrollment and the number of students eligible for free meals. The impact would probably be small in most districts, but it could be large in some. To address this issue, the panel believes that local districts either have or can obtain valid data that could be used for an adjustment. All of these potential limitations of ACS estimates are addressed further in Chapters 4 and 5. APPROACH TO EVALUATING ACS ESTIMATES Estimates from probability survey samples such as the ACS are eval- uated using a framework called "total survey error," which identifies the types of errors that occur at various points in the development of a survey estimate. Components of total survey error include sampling (reflecting the fact that data are collected on a portion, rather than all, of the population), coverage (the degree to which the frame used to draw the sample includes the entire target population), nonresponse (failure to obtain responses for the entire sample), specification (the degree to which a question asked matches the concept about which information is desired), measurement (unintentional or intentional errors in a respon- dent's answer), and processing (errors in applying coding, statistical pro- cessing, and estimation methods). In the context of estimating eligibil- ity for free and reduced-price school meals, the most problematic error components for the ACS are likely to be sampling, specification, and measurement error. The ACS has relatively high coverage and response rates, and processing errors in an ongoing survey tend to be small because of the repeated use of systems developed for the survey. Also important to consider, as indicated in the previous section on limitations, are errors in the panel's specifications for using the ACS data to estimate eligibility for the school meals programs. In March-April 2011, using the panel's specifications, the Census

OCR for page 49
TECHNICAL APPROACH 87 Bureau provided us with ACS 5-year estimates for 2005-2009 for enroll- ment, percentage of students eligible for free meals, percentage of stu- dents eligible for reduced-price meals, and percentage of students eligible for full-price meals (and standard errors for each estimate) for all school districts in the country. The Bureau also provided 3-year estimates for districts with populations greater than 20,000 for the 2005-2007, 2006-2008, and 2007-2009 periods and 1-year estimates for the largest districts--those with populations greater than 65,000--for each year from 2005 to 2009. In addition, the Census Bureau provided five 1-year model-based ACS estimates for the percentage of students eligible for free meals and the percentage eligible for reduced-price meals for each year from 2005 to 2009. It also provided one set of ACS direct 5-year estimates and five sets of 1-year model-based ACS estimates for all schools with boundaries in our five case study districts. This section describes the panel's approach to evaluating the quality of the ACS-based estimates of eligibility through a comparison with esti- mates from other data sources. Results of the comparison are presented in Chapter 4. In particular, ACS direct and ACS model-based estimates for school districts were compared with administrative estimates from the CCD, which, while not error-free, is the most complete and readily usable alternative source of data for school districts and schools available to the panel. ACS direct and model-based estimates were also compared with each other to help us determine which might be best to use in the AEO. Finally, estimates from the FNS-742 administrative data were compared with the CCD administrative estimates to help in assessing any differ- ences between these two benchmarks that might illuminate our com- parisons. At the school level, the ACS 5-year and model-based estimates were compared with estimates from administrative data provided by the case study districts. School-level data provided by the districts were also compared with CCD school-level data. A number of questions needed to be answered through this evaluation. Are ACS direct and model-based estimates for school districts consistent with administrative estimates from the CCD? Are ACS estimates for schools consistent with administrative estimates provided by the case study school districts and administrative estimates from the CCD? These comparisons would identify whether there are systematic differences between estimates from the survey and administrative data sources. How variable are the ACS estimates? We assessed precision, as measured by the variance, standard error (SE), or coefficient of

OCR for page 49
88 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS variation (CV),38 as well as variation over time. Variation over time will be important for school districts considering a new pro- vision because such variation causes changes in reimbursement from year to year, some of which are desirable and some of which are not from a district's perspective. Finally, what is the trade-off between temporal stability and responsiveness to real changes in socioeconomic conditions? Is the difference between ACS estimates and CCD estimates related to district characteristics? Among the characteristics we considered were size of district (measured by enrollment) and prevalence of students certified for free or reduced-price meals (measured by FRPL category). Our analyses needed to address another issue--the relationships among three distributions: (1) the distribution of students eligible or certified in a district by category (free, reduced price, full price); (2) the distribution of meals served in a district by category under traditional operating procedures when some students pay (based on their certifica- tion status) the reduced price or full price for a meal; and (3) the dis tribution of meals that a district would expect to serve by category under the AEO when meals are provided free to all students. Understanding these relationships is critical for developing claiming percentages that reflect not only the distribution of eligible students but also the rates at which they participate, that is, take meals when the meals are free for everyone. Under standard economic assumptions, we expect those partic- ipation rates (under the AEO) to be higher than the rates under traditional operating procedures, which will affect the distribution of meals served. It is appropriate that this participation effect of the AEO be captured in the percentages used to claim reimbursement under the AEO. Analyz- ing such claiming percentages, the projected reimbursements implied by 38Accuracy is assessed by comparing an estimator to a true value. The theoretical bias of an estimator is defined as its mean (its average or expected value over repeated sampling) minus the true value. An estimator is said to be "unbiased" if its bias is zero. It is approxi- mately unbiased if it is on average "close" to the true value; for example, "close" might mean that the (absolute value of the) bias is less than 5 percent of the truth. An accurate estimator is at least approximately unbiased. An estimator is said to be precise if its expected variation in repeated sampling is small. The theoretical variance measures expected variation as the average squared deviation of the estimator from its mean. The standard error of an estimator is the square root of its variance, and is expressed in the same units as the measurements and, thus, the mean. The CV expresses the variation in a way that does not depend on the unit of measurement. It is the ratio of the SE to the mean. The mean squared error (MSE) is measured as the average squared deviation of the estimator from the true value. It is equal to the sum of the variance and the squared bias. The root mean squared error (RMSE) is the square root of the MSE. It is in the same units as the measurements.

OCR for page 49
TECHNICAL APPROACH 89 those percentages, and the ACS eligibility estimates, a school district will be able to assess the financial viability of adopting the AEO. The key parameters of interest for our analysis include the percentage of enrolled students eligible for free meals, the percentage eligible for reduced- price meals, and the sum of the two: the percentage eligible for either free or reduced-price meals. In addition, the panel focuses on the BRR as a sum- mary measure of the distribution of students (or meals served) across the free, reduced-price, and full-price categories. The BRR is the average reim- bursement per meal under the assumption that reimbursement is based on eligibility, certification, or meals served percentages, and is calculated as a weighted sum of the percentages for the free, reduced-price, and full-price categories. The weights in the sum are the per meal reimbursement rates paid by the federal government. We used the rates that were in effect dur- ing 2008-2009 in a district eligible for the $.02 per meal increment: $2.59, $2.19, and $.26 for free, reduced-price, and full-price meals, respectively. (Constant reimbursement values were used so that comparisons over time would not be affected by inflation.) As described above, the panel classified districts based on two main characteristics: (1) percentage of students eligible for free or reduced-price meals and (2) size (small, medium, or large). The free or reduced-price per- centage is directly related to a district's potential interest in the AEO. The so-called "very high FRPL" districts had at least one free or reduced-price percentage equal to or greater than 75 percent over a span of several school years (2004-2005 through 2009-2010) and might consider district- wide adoption of the AEO. During those same school years, the so-called "high FRPL" districts had free or reduced-price percentages of 50 percent in at least one year but never as high as 75 percent and might consider the AEO, but perhaps only for a subset of schools. Districts with free or reduced-price percentages of less than 50 percent in every year are unlikely to benefit from the AEO. Two aspects of district size are important to the panel. The defini- tions of small, medium, and large presented above are related to the ACS direct estimates that would be available to a district. Population size is important as well because it is related to sample size and hence sampling error (larger samples are associated with smaller sampling error). In our analyses, we also used a related measure of size--enrollment. As noted previously, Table 3-1 shows the population of school districts categorized according to free and reduced-price percentage and district population size. As discussed above, we had available five 1-year ACS estimates for the large school districts, three 3-year estimates for the medium districts, and only one 5-year estimate for the small districts. Table 3-3 illustrates the theoretical sampling error associated with different enrollment catego- ries and different free or reduced-price percentages.

OCR for page 49
90 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS TABLE 3-3 Illustrative Approximate Standard Errors of ACS Direct Estimates by Type of ACS Release, School Enrollment, and Estimated Fraction of Free and Reduced-Price Eligible Students Fraction of Students Eligible for Free or Reduced-Price Meals School ACS Release Enrollment 0.5 0.6 0.7 0.8 0.9 1-year 12,000 0.091 0.090 0.084 0.073 0.055 1-year 16,000 0.079 0.077 0.072 0.063 0.047 1-year 20,000 0.071 0.069 0.065 0.057 0.042 3-year 4,000 0.089 0.088 0.082 0.071 0.054 3-year 7,000 0.068 0.066 0.062 0.054 0.041 3-year 10,000 0.057 0.055 0.052 0.045 0.034 5-year 500 0.191 0.187 0.175 0.153 0.115 5-year 1,500 0.110 0.108 0.101 0.088 0.066 5-year 3,000 0.078 0.076 0.071 0.062 0.047 NOTE: For purposes of this report, we calculated standard errors using the formula for a simple random sample and a design effect of 3. Data provided by the Census Bureau include the actual standard errors of all estimates. The standard error divided by the estimate and converted to a percentage gives the coefficient of variation (CV), which should be 10 percent or less by commonly used statistical standards; a higher CV indicates a less reliable estimate. In this table, the standard errors in boldface type are 10 percent or less of the estimated frac- tion of students eligible for free or reduced-price meals. SOURCE: Prepared by the panel. Systematic Differences To address the question of consistency between estimates from the ACS and alternative administrative data sources, the panel evaluated the difference between an ACS estimate (enrollment, percentage free, percentage reduced price, percentage free or reduced price, BRR) and the corresponding estimate from an alternative data source computed for each school district or school in our evaluation database.39 If the average of these differences over a large group of districts (or schools) were near zero, we would conclude that there is no systematic difference between the two estimates. We analyzed systematic differences by examining the average difference over districts or schools grouped by variables that we think may have a relationship to such differences: FRPL level and size. We also analyzed potential sources of differences using additional data, including SNAP administrative data and data from the Survey of Income and Program Participation (SIPP). In addition, we considered whether a 39The panel's evaluation data base is named prog9_merged_fns_wSE.xlsx.

OCR for page 49
TECHNICAL APPROACH 91 regression model could be used to adjust for differences between ACS and CCD estimates based on demographic and other variables that were available from the CCD. Precision, Intertemporal Stability, and Timeliness To evaluate precision, intertemporal stability, and timeliness, the panel concentrated on the BRR because it is stability of reimbursement that is of greatest importance to school food authority directors. For ACS estimates (direct and model-based), the primary measure of precision is the sampling error, as measured by the standard error.40 Because they are based on a larger sample, the 5-year ACS estimates for a district will have smaller sampling error than the 3-year or 1-year estimates. However, this greater precision comes at a price: a 5-year ACS estimate reflects the average observed over a 5-year period, and thus will be relatively slow in adjusting to real changes in the economy. Trade-offs between stability and timeliness are assessed by comparing the year-to-year variability in BRRs computed using CCD certification data versus the alternative ACS eligibility estimates (1-, 3-, and 5-year). The BRRs based on CCD certifi- cation percentages provide an indication of the year-to-year variation in reimbursement that is normally experienced by and, therefore, will likely be acceptable to districts. Data on school district reimbursements under the school meals programs were not available to the panel, so there is no way to compare ACS estimates with actual reimbursement data. Participation For the case study districts and schools within those districts, the panel compared BRRs based entirely on distributions of students with BRRs based on distributions of meals served. These distributions and the associated BRRs differ because students in the different categories par- ticipate at different rates, with, generally, students receiving free meals having the highest rate, students paying full price having the lowest rate, and students paying a reduced price having a rate between the other two. The BRRs based on the distribution of meals served reflect these differential participation rates, whereas the BRRs based entirely on the distribution of eligible or certified students take no account of participa- tion. Comparing the BRRs illustrates how a district would generally be underreimbursed if participation were not taken into account in develop- ing claiming percentages. 40Standard errors were provided by the Census Bureau for all ACS direct and model-based estimates.

OCR for page 49
92 USING ACS DATA TO EXPAND ACCESS TO THE SCHOOL MEALS PROGRAMS Taking participation into account, however, is complicated because participation rates will likely increase in each category--probably by dif- ferent amounts--if a district adopts the AEO and provides free meals to all students in some or all of its schools. As noted above, given standard economic assumptions about the role of prices in demand for school meals (that school meals are a normal good, for which demand increases when the price decreases), adoption of the AEO would be expected to increase demand among all students who were not already approved to receive free meals. The availability of free school meals for all students might also be expected to increase demand (increase the number of school meals consumed) among those eligible for free meals because it would reduce the family's burden of applying for benefits and remove any perceived stigma associated with participating in the program. Because the panel had limited data with which to assess the impact of increases in participation attributable to providing free meals, we simulated the potential effects of the AEO on participation and examined how the simu- lated participation effects would affect BRRs. In light of our results, our proposed procedure for implementing the AEO includes the operation of a base year during which all students receive free meals, applications are solicited from parents, and certification and verification are conducted. With this approach, as under Provision 2, the increases in participation can be estimated and reflected in claiming percentages. The claiming percentages will also incorporate eligibility estimates based on the most recently released ACS data. Assessment of the Need for Benchmarking The panel's central goal was to assess the suitability of ACS esti- mates to support the school meals programs from the perspective of the estimates' fitness for use. We found that the conceptual fit of the ACS estimates is acceptable, although it would benefit from additional research. Chapter 4 presents our analysis of any systematic differences between ACS and administrative estimates and considers the precision, temporal stability, and timeliness of ACS estimates. If there are districts in which ACS eligibility estimates fluctuate excessively in ways that are not consistent with real changes in socioeconomic conditions, there will be little a district can do other than decide not to adopt the AEO. If ACS estimates are fairly stable but differ systematically from administrative estimates, however, a procedure for benchmarking the ACS estimates to the administrative estimates could provide the best way to use ACS data in support of the school meals programs. Based on the results of our anal- yses (presented in Chapter 4 and in several appendixes), we developed procedures for implementing the AEO, presented in Chapter 5.