Using American Community Survey Data to Expand Access to the School Meals Progams
Panel on Estimating Children Eligible for School Nutrition Programs
Using the American Community Survey
Allen L. Schirm and Nancy J. Kirkendall, Editors
Committee on National Statistics
Division of Behavioral and Social Sciences and Education
NATIONAL RESEARCH COUNCIL
OF THE NATIONAL ACADEMIES
THE NATIONAL ACADEMIES PRESS
Washington, D.C.
http://www.nap.edu
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This study was supported by contract number AG-3198-C-09-0006 between the National Academy of Sciences and the U.S. Department of Agriculture. Support for the work of the Committee on National Statistics is provided by a consortium of federal agencies through a grant from the National Science Foundation (award number SES-1024012). Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the organizations or agencies that provided support for this project.
International Standard Book Number-13: 978-0-309-25720-6
International Standard Book Number-10: 0-309-25720-4
Additional copies of this report are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Cover credit: The photo on the left is of 2-year-old Joaquin Sanchez as part of a campaign by the California Milk Processor Board Positivity Ambassador blogger campaign with NibblesandFeasts.com. The photo was taken October 4, 2011, by Joaquin’s mother and blog author, Ericka Sanchez.
Copyright 2012 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Research Council. (2012). Using American Community Survey Data to Expand Access to the School Meals Programs. Panel on Estimating Children Eligible for School Nutrition Programs Using the American Community Survey, A.L. Schirm and N.J. Kirkendall, Editors. Committee on National Statistics, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council.
This page intentionally left blank.
PANEL ON ESTIMATING CHILDREN ELIGIBLE FOR SCHOOL NUTRITION PROGRAMS USING THE AMERICAN COMMUNITY SURVEY
ALLEN L. SCHIRM (Chair), Mathematica Policy Research, Washington, DC
DAVID M. BETSON, Department of Economics, University of Notre Dame
MARIANNE P. BITLER, Department of Economics, University of California, Irvine
F. JAY BREIDT, Department of Statistics, Colorado State University
ROBERT E. FAY, Westat, Rockville, Maryland
ALBERTA C. FROST, Alexandria, Virginia
MICHAEL F. GOODCHILD, Department of Geography, University of California, Santa Barbara
PARTHA LAHIRI, Department of Statistics, University of Maryland
PENNY E. McCONNELL, Fairfax County Public Schools, Springfield, Virginia
SARAH NUSSER, Department of Statistics, Iowa State University
JOHN PERKINS, Perkins Consulting Group, Austin, Texas
JAMES H. WYCKOFF, Currie School of Education, University of Virginia
NANCY J. KIRKENDALL, Study Director
ESHA SINHA, Associate Program Officer
AGNES E. GASKIN, Administrative Assistant
COMMITTEE ON NATIONAL STATISTICS 2011-2012
LAWRENCE D. BROWN (Chair), Department of Statistics, The Wharton School, University of Pennsylvania
JOHN M. ABOWD, School of Industrial and Labor Relations, Cornell University
ALICIA CARRIQUIRY, Department of Statistics, Iowa State University
WILLIAM DuMOUCHEL, Oracle Health Sciences, Tucson, Arizona
V. JOSEPH HOTZ, Department of Economics, Duke University
MICHAEL HOUT, Survey Research Center, University of California, Berkeley
KAREN KAFADAR, Department of Statistics, Indiana University
SALLIE KELLER, Provost, University of Waterloo, Ontario, Canada
LISA LYNCH, The Heller School for Social Policy and Management, Brandeis University
SALLY C. MORTON, Department of Biostatistics, University of Pittsburgh
JOSEPH NEWHOUSE, Division of Health Policy Research and Education, Harvard University
RUTH PETERSON, Criminal Justice Research Center, The Ohio State University
HAL S. STERN, Donald Bren School of Computer and Information Sciences, University of California, Irvine
JOHN THOMPSON, National Opinion Research Center at the University of Chicago
ROGER TOURANGEAU, Westat, Rockville, Maryland
ALAN ZASLAVSKY, Department of Health Care Policy, Harvard University Medical School
CONSTANCE F. CITRO, Director
Acknowledgments
The Panel on Estimating Children Eligible for School Nutrition Programs Using the American Community Survey (ACS) wishes to thank the many people and organizations that contributed to the preparation of this report. Without their help, the panel could not have completed the report. As chair, I want to thank my fellow panel members for their commitment to the work under a demanding time schedule. They consistently provided insightful and constructive input under tight deadlines as we developed this report. I have appreciated their always good humor, and it has been a pleasure working with them.
The panel thanks John Endahl, Jay Hirschman, and Cindy Long of the Food and Nutrition Service (FNS) for their patient explanation of the many rules, regulations, data sources, and evaluation studies pertaining to the school meals programs. We are also grateful for the expert advice of staff of the U.S. Census Bureau in helping us understand the data collected in the ACS and the estimates developed in the Small Area Income and Poverty Estimates (SAIPE) Program. Special thanks go to Wes Basel, Rick Denby, Doug Geverdt, and David Powers of the Census Bureau for participating regularly in our open meetings; offering advice; and, most important, performing the special tabulations of ACS data to derive the direct and model-based estimates we needed to conduct our work.
The panel also extends special thanks to the food service authority directors in our case study districts: Christine Carillo-Spano of the Austin Independent School District, Texas; Altheria Maynard of the Savannah-Chatham County Public School System, Georgia; Nicole Meschi of the
Pajaro Valley Unified School District, California; Helen Philips of the Norfolk Public Schools, Virginia; and Tammy Yarmon of the Omaha Public Schools, Nebraska. These individuals and their staff provided us with large amounts of data, answered many questions, and gave us valuable insights into the school meals programs. They are clearly dedicated to providing high-quality meals to as many children as possible.
Special thanks go as well to Salvatore Saporito, Stuart Hamilton, and Ashwini Wakchaure of the College of William and Mary and the School Attendance Boundary Information System (SABINS) project for participating regularly in the panel’s open meetings; making presentations concerning the SABINS project and its progress; offering general advice concerning geographic issues; and, most important, creating the files containing boundary information for all schools in our case study districts in the form needed by the Census Bureau.
In the process of preparing this report, the panel convened 10 meetings, 6 of them open meetings to benefit from presentations by many individuals. We would like to express our thanks for presentations concerning the school meals programs and administrative data by John Endahl, Ed Harper, Jay Hirschman, Cindy Long, Melissa Rothstein, Gary Vessels, and William Wagoner, FNS, and Christopher Logan of Abt Associates; presentations concerning the ACS, SAIPE, and geographic issues by Mark Asiala, Wes Basel, Douglas Geverdt, Todd Hughes, David Johnson, Donald Lurey, Alfredo Navarro, David Powers, Michael Ratcliffe, and Sharon Stern of the Census Bureau; a presentation concerning measurement of income and program participation by John Czajka, Mathematica Policy Research; a presentation concerning income variability and its impact on eligibility for school meals by Constance Newman, Economic Research Service, U.S. Department of Agriculture; a presentation concerning the Access, Participation, Eligibility, and Certification Study by Michael Ponza, Mathematica Policy Research; and a presentation concerning relevant U.S. Government Accountability Office (GAO) studies addressing the school meals programs by Kay Brown, GAO. The panel would also like to express its appreciation to the staff of the National Center for Education Statistics for attending panel meetings, describing their data and the quality of those data, and facilitating our collaboration with the Census Bureau.
The panel’s final open meeting, held on March 3 and 4, 2011, was a workshop with school food service authority directors from our case study districts and selected other individuals from the school food community with insights to offer about Provisions 2 or 3 and the school meals programs more generally. The purpose of the workshop was to help us better understand issues pertaining to a potential new provision of the school meals programs, as well as the information school dis-
tricts would need to determine whether to adopt this special provision. Participants included Onetha Bonaparte, school meals program coordinator, Savannah-Chatham County Public School System, Georgia; Tim Cipriano, executive director of food services, New Haven Public Schools, Connecticut; Lyman Graham, food service director, Roswell Independent School District, New Mexico; Lynn Harvey, section chief, Child Nutrition Services, Division of School Support, North Carolina Department of Public Instruction; Leo Lesh, executive director, Enterprise Management, Denver Public Schools, Colorado; Terry Mendez, administrator for food and nutrition services, Brownsville Independent School District, Texas; Nicole Meschi, director of food and nutrition services, Pajaro Valley Unified School District, California; Mary Jo Tuckwell, senior consultant, Food Services Group, inTEAM Associates, Wisconsin; and Tammy Yarmon, director, Nutrition Services, Omaha Public Schools, Nebraska. The information and insights provided by these individuals were tremendously helpful to the panel.
The panel was assisted by a highly able staff. Our work could not have been completed without the extraordinary dedication, seemingly boundless energy, and many contributions of Nancy Kirkendall, the study director. She provided technical and substantive insights, conducted and oversaw many analyses, drafted and revised key sections of our reports, and kept the panel and project on track, all while remaining unfailingly upbeat. I very much enjoyed working with Nancy. We would like to acknowledge Linda Meyers and Lynn Parker of the Food and Nutrition Board of the Institute of Medicine for their help in identifying individuals knowledgeable about the school meals programs. We are also grateful for the consistently wise counsel provided by Connie Citro, director of the Committee on National Statistics, at critical points in our work when the path ahead was not clear; for the tabulations and analysis provided by Esha Sinha, associate program officer; for the assistance of Agnes Gaskin, administrative assistant to the panel, in handling our logistical arrangements and meetings and preparing the final manuscript; for the help of Kirsten Sampson Snyder in managing the report review process; for the skillful editing of Rona Briere; and for the management of the production process by Yvonne Wise. We would also like to thank students who assisted in data processing and analysis under the guidance of panel members or staff: Addison James, Colorado State University; Jeffrey Moon, Indiana University of Pennsylvania and a junior fellow of the Joint Program in Survey Methodology; John Michael Salas, University of California, Irvine; and Stephanie Zimmer, Iowa State University. In addition, we are grateful to Umet Ozek, National Center for Analysis of Longitudinal Data in Education Research (CALDER), American Institutes for Research, who provided aggregate District of Columbia Public Schools
data for use in the panel’s analyses of the effects of school choice. Finally, we would like to thank several staff of Mathematica Policy Research: Kai Filion and Mary Grider, who devoted numerous hours to constructing the district-level evaluation database that supported many of our key analyses; Frank Yoon, who helped us analyze differences between ACS and administrative estimates; and Esa Eslami, Bruce Schechter, Joel Smith, and Carole Trippe, who prepared estimates of Supplemental Nutrition Assistance Program (SNAP) recipients from ACS and SNAP Quality Control data.
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the Report Review Committee of the National Research Council (NRC). The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: Marc P. Armstrong, professor and CLAS collegiate fellow, chair, Department of Geography, interim chair, Department of Communication Studies, dean’s administrative fellow, College of Liberal Arts and Sciences, The University of Iowa; Marilyn Briggs, co-director, Center for Nutrition in Schools, Department of Nutrition, University of California, Davis; Kathy F. Kuser, consultant, Lithia, Florida; Jean Opsomer, Department of Statistics, Colorado State University; Joseph Salvo, Population Division, New York City Department of City Planning; Eric Slud, area chief of mathematical statistics, Center for Statistical Research and Methodology, U.S. Census Bureau, and professor, Statistics Program, Mathematics Department, University of Maryland; and Grant I. Thrall, Department of Geography, University of Florida.
Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the report’s conclusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by V. Joseph Hotz, Department of Economics, Duke University and Charles Phelps, university professor and provost emeritus, University of Rochester. Appointed by the NRC’s Report Review Committee, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring panel and the institution.
Finally, we recognize the many federal agencies that support the Committee on National Statistics directly and through a grant from the National Science Foundation. Without their support and their commitment to improving the national statistical system, the work that served as the basis for this report would not have been possible.
Allen L. Schirm, Chair
Panel on Estimating Children Eligible for
School Nutrition Programs Using the
American Community Survey
This page intentionally left blank.
Contents
Overview of School Meals Programs
Administrative Process of the School Meals Programs
Special Provisions and Options for Operating the School Meals Programs
Alternative Reimbursement Formulas
Errors in Methods for Determining Reimbursements
Sources of Data on Eligibility and Participation
Construction of Evaluation Databases
Sources of Information for Designing and Implementing an ACS Eligibility Option
Framework for Evaluating the Use of Estimates Based on ACS Data
Differences Between ACS and Administrative Estimates
Precision, Intertemporal Stability, Timeliness, and Relative Performance of Estimates
5 A Plan for Implementing the AEO
A Procedure for Benchmarking ACS Estimates and Updating Claiming Percentages Districtwide
Implementation of the AEO Under Special Circumstances
6 Recommendations for Future Work
Improving Data Quality and Availability
Facilitating Implementation of the ACS Eligibility Option
A Glossary of Acronyms and Terms
B Estimates of Eligible Students Using the American Community Survey
Identifying Public School Students
Multiple Economic Units Among Related Individuals
State and District Analysis of Categorical Eligibility
C Model-Based Estimates for School Districts and School Attendance Areas
Panel’s Suggestions for Modeling Eligibility Percentages for the School Meals Programs
E Data Collected from School Districts
Part 1: Case Study Districts—Data Collection and Comparison of Enrollment and Certification
Part 2: Agenda for Workshop with School Food Authority Directors
Part 3: Survey of Districts Operating Under Provision 2 or 3
F Additional Information About the Panel’s Analyses
Part 1: Comparisons of ACS Estimates and Estimates Based on Administrative Data
Underreporting of SNAP Benefits
Determining Eligibility Using Annual Versus Monthly Income
This page intentionally left blank.
Tables, Figures, and Boxes
TABLES
2-6 Federal Reimbursement Rates for 2010-2011 School Meals Programs by Eligibility Category
4-1 Average Differences Between ACS 5-Year Estimates for 2005-2009 and CCD Estimates for 2009-2010
4-3 Average Differences Between ACS 1-Year Estimates and CCD Estimates, Large Districts Only
4-7 BRRs Based on Monthly and Annual Income Estimates: Bias and Ratio
4-8 Intertemporal Variability of ACS 5-Year Estimates by Enrollment
4-13 Alternative BRRs for Case Study Districts
5-1 Districts Operating Under Provision 2 or 3 in 2009-2010,Not in a Base Year
5-3 Step 1a: Calculate Averages of ACS Eligibility Percentages for Preliminary Benchmarking
5-5 Step 1c: Calculate Preliminary Benchmarking Adjustments
5-6 Step 1d: Calculate Preliminary Benchmarked Eligibility Percentages
5-7 Step 1e: Calculate Preliminary BRRs
5-8 Step 2: Preliminary Assessment of the AEO: Simulate Reimbursements
5-11 Step 4a: Calculate Averages of ACS Eligibility Percentages for Final Benchmarking
5-12 Step 4b: Calculate Averages of Administrative Certification Percentages for Final Benchmarking
5-13 Step 4c: Calculate Final Benchmarking Adjustments
5-14 Step 4d: Calculate Benchmarked Eligibility Percentages
5-16 Step 5: Final Assessment of the AEO: Simulate Reimbursements
5-19 Step 6: Calculate Initial Claiming Percentages and Make Final Decision About Adopting the AEO
5-20 Step 7a: Benchmark Newly Released ACS Eligibility Percentages
B-2 Percentage of Related Students Income-Eligible for School Meals by Economic Unit
B-3 Relationships Reported for Students Who Are Not Related tothe Householder
B-4 Household Composition of Unrelated Students Who Are Not Unmarried Partners
C-1 Regression Results for 2009
E-2 Data Received from Case Study Districts
E-3 Counts of Schools in Case Study Districts
F-1 Average Differences Between ACS 5-Year Estimates and 5-Year Averages of CCD Estimates
F-2 Average Differences Between ACS 3-Year Estimates and 3-Year Averages of CCD Estimates
F-3 Average Differences Between ACS 5-Year Estimates of Enrollment and Various CCD Estimates
F-4 Average Differences Between ACS 3-Year Estimates of Enrollment and Various CCD Estimates
F-5 Average Differences Between ACS 1-Year Estimates of Enrollment and CCD Estimates
F-9 Model Versus Empirical Estimates for Variances of Year-to-Year Changes, Large Districts Only
F-10 Model Versus Empirical Estimates for Variances of Year-to-Year Changes, Medium Districts Only
F-11 Model Versus Empirical Estimates for Variances of Year-to-Year Changes, Small Districts Only
F-12 Intertemporal Variability of ACS 5-Year Estimates, by Enrollment
G-5 Share of Public School Enrollment by Choice Status
G-6 Eligibility Distribution for Households with Students, Selected Characteristics
G-7 Certification Category and Correct Eligibility Category in School Year 2005-2006
B-1 Impact of alternative economic unit definitions by state
B-2 Impact of alternative economic unit definitions by school district
C-1 Median free and reduced-price eligibility rates estimated by the models over time
C-2 Average 5-year ACS eligibility rates for free and reduced-price meals by size of school district
F-1 Regression fit of log (design variance) versus log (enrollment)
BOXES
5-1 Calculating ACS and Administrative Averages and Benchmarking Adjustments
5-2 Preliminary Benchmarking of ACS Estimates
5-4 Calculating AEO Claiming Rates for Use in First Year after Base Year
5-5 Benchmarking Future ACS Eligibility Estimates
5-6 Updating AEO Claiming Rates
5-8 Benchmarking of ACS Estimates Provision 2 or 3 Districts
5-10 Calculation of AEO Claiming Rates for Use in 2014-2015 Provision 2 or 3 Districts
5-11 Benchmarking Future ACS Estimates and Updating of AEO Claiming Rates Provision 2 or 3 Districts
5-13 Other Uses of Data on Students Certified for Free and Reduce-Price Meals
B-1 ACS Questions on Schooling
B-2 ACS Questions on Achievement
B-4 Income as Defined by FNS “Eligibility Manual for School Meals,”
B-5 ACS Questions About Income
B-6 ACS Question About Relationship to Respondent
B-7 Definition of Economic Units for Sensitivity Analysis
B-8 ACS Questions Related to Categorical Eligibility
F-1 SAS Code for Analysis of Variability
F-2 SAS Proc Mixed Output: The Mixed Procedure
F-3 SAS Output Proc Mixed: Estimated R Matrix for Large Districts
F-4 SAS Output Proc Mixed: Estimated R Correlation Matrix for Large Districts
F-6 SAS Proc Mixed Output, Fit Statistics