Summary

The Food Economics Division of the U.S. Department of Agriculture’s (USDA’s) Economic Research Service (ERS) engages in research and data collection to inform policy making related to the leading federal nutrition assistance programs managed by USDA’s Food and Nutrition Service (FNS).¹ ERS also studies how food consumption and nutrition influence the U.S. population’s health and well-being, which in turn affect the cost of government health insurance programs. Food insecurity and inadequate nutrition are strongly associated with a range of health and social consequences, including acute birth outcomes, impaired academic performance, and behavioral control and acuity problems. Understanding why people choose foods, how food assistance programs affect these choices, and what the health impacts are must be informed by a multisource, interconnected, reliable data system. In conducting these data collection and research activities, ERS advances the public good.

The Consumer Food Data System (CFDS) is a “portfolio of data resources that measure, from the perspective of a consumer, food and nutrition conditions. and the factors that affect those conditions” (Larimore et al., 2018). It supports stand-alone surveys and specialized modules

___________________

¹ As stated by the agency, “FED [food economics division] conducts economic research and analysis on policy-relevant issues related to the food sector (food safety, food prices, and markets); consumer behavior related to food choices (food consumption, diet quality, and nutrition); and food and nutrition assistance programs (SNAP, WIC, National School Lunch Program). FED also provides data and statistics on food prices, food expenditures, and the food supply chain.” See https://www.ers.usda.gov/about-ers/agency-structure/food-economics-division-fed.

added to established federal surveys, and it links USDA-funded survey data to external sources, including other survey data; commercial data; and federal, state, and local government administrative data. The CFDS helps the agency fulfill its mission to “anticipate trends and emerging issues in agriculture, food, the environment, and rural America and to conduct high-quality, objective economic research to inform and enhance public and private decision making.”²

ERS asked the National Academies of Sciences, Engineering, and Medicine’s Committee on National Statistics to review and provide guidance for its CFDS program. The key component of the charge (reproduced in full in Chapter 1) was to provide a blueprint for increasing the value of the CFDS by “providing guidance for its advancement over the next 10 years” to enhance its capacity to support research that informs high-priority current and future policy questions. The charge also asked for guidance regarding future iterations of ERS’s National Household Food Acquisition and Purchase Survey (FoodAPS), which was the first comprehensive survey on food acquisitions from all sources.

THE SCOPE OF A CONSUMER FOOD AND NUTRITION DATA SYSTEM

High-quality, comprehensive data (i) produce descriptive information about population and program characteristics, (ii) serve a monitoring function to track nutrition, health, food security and safety, and other outcomes, and (iii) support research, including causal inference and program evaluation.

Descriptive information about food and nutrition safety net programs and the healthfulness of U.S. diets is important in its own right. Monitoring information provides a series of snapshots of outcomes nationally and, when available, at more granular state and local levels. Examples of questions answered through careful data monitoring include: How many people have limited access to sources of healthy and affordable foods? What is the healthfulness of the American diet? Who participates in USDA food assistance programs? How do food security and obesity change over time? And, in what ways are Supplemental Nutrition Assistance Program (SNAP) and low-income households similar to or different from the overall population?

The third functional role of the CFDS is to enable USDA staff and outside researchers to answer causal questions about important food-related outcomes. For example, how does access to grocery stores, restaurants, and the broader food environment affect food choices and diet-related health?

___________________

² See https://www.ers.usda.gov/about-ers.

And, how do SNAP, the Special Supplemental Nutrition Program for Women, Infant, and Children (WIC), and school meals programs and policies affect nutrition, food security, health, and use of health care systems?

Features of a (nonexperimental) data system that facilitate strong causal research designs include (i) the provision of sampling frames through administrative data that can be used for random assignment or survey purposes; (ii) the provision of comparison data that are nationally representative for use in understanding the study populations through nonexperimental evaluations; (iii) integration with policy information as explanatory variables (as is emphasized in parts of this report that address the SNAP rules); (iv) longitudinal or panel structures for use in fixed-effects models that control for unobserved time-constant confounding variables; and (v) inclusion of appropriate administrative data on program participation linked with nationally or regionally representative survey or administrative data on the population of potentially eligible persons.

DESIRABLE CHARACTERISTICS OF A CONSUMER FOOD AND NUTRITION DATA SYSTEM

Recognizing that tradeoffs must be made, the panel identified several characteristics of a data system that are desirable in terms of its usefulness for research and informing policy:

• Comprehensiveness. To monitor levels and trends in food behaviors and related outcomes and to identify the effects of public programs and policies on those behaviors, a comprehensive data system requires a variety of sources spanning multiple topics.
• Representativeness. Data on food behaviors and outcomes are most useful if it is representative of the U.S. population, both nationally and subnationally.
• Timeliness. To have maximum program and policy impact, the system collects data at regular intervals, repeats over time at an appropriate frequency, and releases data without undue delay.
• Openness. Because data programs are maintained with taxpayer funds, data should be accessible to the public and to the research community. Security and privacy concerns must be addressed before making de-identified data available.
• Flexibility. A flexible data system recognizes that food and consumer data will be used for some research applications that were planned in advance, as well as for applications generated by a broad, entrepreneurial, and inventive community of research users studying unanticipated changes in policy, food retail markets, or consumer preferences.

• Accuracy. Accurate measurement and reporting are the foundation of effective evidence-based policy making, so a desirable data system is one that seeks continuous quality improvement. Given increased reliance on data produced by state and local governments and commercial entities for purposes other than scientific study, continual assessment and improvement of the quality of these sources will be a central part of the CFDS.
• Suitability for causal analysis. While some policy questions can be answered with descriptive information, others require cause- and-effect inference. With this in mind, data design efforts should include (i) the collection and sharing of policy variables for use in implementing quasi-experimental designs, (ii) the use of administrative data for potential program evaluations with random-assignment research designs, and (iii) the creation of longitudinal survey and administrative data (either repeated cross-sections or panel data) for use in statistical analyses that offer causal insight.
• Fiscal responsibility. The CFDS should maximize the research value of federal dollars invested in the data system (including staff time) through its combined impact in descriptive information, monitoring functions, and estimation of causal effects.

Achieving these characteristics in a data system to support food and nutrition research requires a multipronged approach involving survey, administrative, and commercial data (Larimore et al., 2018).

EXPLOITING DIVERSE SOURCES OF DATA

The federal government’s statistical system—a survey-centric one reflecting best methodological practices of the 20th century—is now at a crossroads. Declining response rates have led to surveys becoming ever more costly and, at times, less accurate and generalizable. This well-documented development (Commission on Evidence-Based Policymaking, 2017; NASEM, 2017a), coupled with the emergence of lower-burden complementary and alternative data sources, has given rise to new data paradigms. The CFDS is well positioned in this changing data environment given its advances in blending surveys, administrative data (residing within USDA programs or elsewhere, such as the Census Bureau), and proprietary commercial data (including retail scanner data, household scanner data, and geospatial information on food stores and restaurants).

Although recent changes in the kinds of data available for research purposes have been profound, surveys continue to play an essential role. Some information, such as nutrition outcomes, cannot be acquired from administrative or other nonsurvey data sources. Traditionally, surveys have

also been the main source for data on eligibility, participation, and benefit amounts for safety net programs such as SNAP, WIC, and Temporary Assistance for Needy Families; but there are concerns about respondent burden and data accuracy for these purposes (Meyer et al., 2015). Administrative data residing within government agencies, sometimes linked to survey data, can provide accurate measures of program participation and benefit amounts. Commercial data—obtained directly from food vendors or from companies engaged as third-party private aggregators, such as Nielsen and IRI—have become increasingly desirable because of the high volume, detail, and frequency of information they can provide about food prices, food outlets, and the spectrum of food choices within those outlets. However, by their nature, commercial data are not designed for research purposes, and they are typically only made available under restrictive arrangements. Nonetheless,

The high value to USDA’s CFDS created by linkages to external datasets—whether commercial or administrative—is enhanced when data cover parallel concepts in the same geographic areas across time, allowing for an evaluation of the effects of policy changes and other interventions.

IMPROVING SURVEY COMPONENTS OF THE CFDS

As described in Chapter 2, USDA invests in multiple survey data sources, including (i) modules on major nationally representative surveys fielded by other government agencies, (ii) survey components in FNSsupported evaluation studies, and (iii) the partnership between ERS and FNS to create FoodAPS. FoodAPS provides descriptive data on where households acquire food in a typical week, which foods they acquire, and how much they pay (Todd and Scharadin, 2016). It is unique among data sources in tracking both food acquired to eat at home and food acquired away from home. It allows analyses that examine which factors are correlated with households’ decisions about where to shop for food (Ver Ploeg et al., 2017); which household characteristics are associated with increased childhood obesity risks (Jo, 2017); how SNAP benefits are used over the course of the benefit month (Smith et al., 2016); and how price variation across geographic areas is associated with food choices and whether this varies by SNAP participation (Basu et al., 2016). FoodAPS supports monitoring functions by allowing the choices of program recipients and eligible

nonrecipients of food assistance programs to be examined. Although the cross-sectional design imposes limitations, FoodAPS has also spawned some causal impact research. For example, Kuhn (2018) examines the impact of Electronic Benefit Transfers (EBT) on households’ intramonth consumption cycles.

A key innovation of FoodAPS is its use of linkages to nonsurvey data sources. One example of this design element is the use of official SNAP administrative records to create a frame for sampling SNAP recipients. Information on nutrient intake and the retail environment was added using commercially produced barcodes, product descriptions, and household location data.

The development and fielding of FoodAPS encountered the usual high level of technical burden associated with creating a new dataset with many linkages. Since FoodAPS cannot satisfy all analytic demands, resource allocation for it needs to be assigned in a way that leaves resources available for other data programs. Because the greatest strength of FoodAPS is in its capacity to generate descriptive and monitoring information for research and policy, and also because it is an expensive survey, it is not practical to envision it as an annual or even biannual program. That said, there is clear value to conducting the survey on a regular basis, because doing so would allow it to contribute to the construction of stylized facts for the monitoring function of the CFDS. Implementing a fixed and predictable schedule (e.g., as the Census Bureau does with the Economic Census) would generate efficiencies and predictability by creating a regular staffing cycle for the Food and Economics Division (FED). This is important if ERS is to manage the data system without having other valuable components of the CFDS suffer when FoodAPS’s resource demands are high.

The move to a regular schedule would also allow ERS to plan for the integration of new data sources, such as administrative data on multiple programs. The ordered planning cycle would facilitate continual process improvement and strengthen institutional memory of how a national survey is conducted. This approach would also avoid the need to pay the fixed costs of conducting new surveys at uneven time intervals. Finally, asking consistent questions over time would also improve the usefulness of the resulting data by, for example, allowing for comparability across assessments of time trends.

To the extent that FoodAPS is intended to support research beyond the monitoring of food acquisitions and related outcomes, such as longitudinal

and causal research, planners can learn from other surveys that match samples to longitudinal administrative data. While, for cost and other reasons, a true longitudinal structure is not feasible for FoodAPS, the survey could sample from the same geographical units—that is, the same primary sampling units (PSUs)—to create a repeated cross-sectional design. This would permit researchers to combine cross-PSU changes over time in socioeconomic conditions, policy choices, and the built environment to assess how economic, policy, and environmental factors affect food acquisition and related outcomes collected in FoodAPS.

More broadly within the survey domain, ERS has made effective use of modules attached to other surveys. Examples include the Food Security Supplement (in the Current Population Survey), the Flexible Consumer Behavior Survey (in the National Health and Nutrition Examination Survey³ [NHANES]), and the Eating and Health Module (in the American Time Use Survey). This approach, which ERS will no doubt continue to pursue, allows the strengths of established instruments, such as the set of explanatory covariates contained therein, to be exploited.

USE OF ADMINISTRATIVE DATA IN THE CFDS

Statistical agencies are investing more heavily in administrative data sources than they have in the past, for reasons to do with both the high cost of survey approaches and the accuracy of information. Administrative data can be used in a variety of ways, both on their own and in combination with other data. The case for expanded and better coordinated use of valuable administrative data—such as those that reside within federal, state, and local governments—is especially clear for purposes of program monitoring, evaluation, and improvement. This value is enhanced when the administrative records can be linked to data

___________________

³ NHANES is the only food intake survey in the United States. Because it is designed to collect information on consumers’ “knowledge, attitudes, and beliefs regarding nutrition and food choices,” it relates to many of the issues that fall within FED’s purview. See https://www.ers.usda.gov/topics/food-choices-health/food-consumption-demand/flexible-consumer-behavior-survey.

on the population of program eligibles, such as from sources such as the American Community Survey.

ERS has improved its capacity to collaborate across agencies, in part through a Census Bureau and USDA partnership—the Next Generation Data Platform—that allows the agency to access and analyze detailed SNAP participation data from many states and WIC data from several states. Subsequent linkages to survey data have improved USDA models of SNAP eligibility and participation rates (Scherpf et al., 2015). Using the Next Generation tools, the linked survey and program records have been found to more accurately reflect information about participants than the survey data alone. For this program, ERS relies on the Census Bureau’s infrastructure to negotiate sharing arrangements and to ingest, harmonize, and link records.

Another area with great potential for enhancing research is the expansion of access to policy databases maintained by several nutrition programs. For example, the SNAP Policy Database includes information on a host of SNAP policy choices, and the SNAP Distribution Database contains information on the timing of SNAP distributions by different states within the month. These are models of administrative data resources that allow research to be carried out on policy options, such as how different choices made by different governmental entities affect outcomes in their localities. These databases also enable research on the causal effects of program participation using the SNAP cycle.

Ideally, data would be included on cash purchases and SNAP or WIC redemptions for the same individuals and sales and redemptions at the same stores so complete acquisitions could be studied.

Recent legislative developments provide support for ERS as it moves to maximize the potential of administrative data. The Foundations for Evidence-Based Policymaking Act states, “the head of an agency shall, to the extent practicable, make any data asset maintained by the agency available, upon request, to any statistical agency or unit for purposes of developing evidence.” And the Farm Bill states that the Secretary shall provide guidance and direction on how states should form longitudinal databases supporting research on participation in and the operation of SNAP.

Coordinated data sharing would involve careful assessment of the quality and comparability, across locations, of the administrative data brought in. States and localities have different data systems and records, all of which need to be checked for consistency and harmonized.

ERS’s vision for the CFDS should include partnerships within the federal statistical system so that survey data may be blended with administrative or proprietary data with little error. If the Evidence Act makes information from other agencies available to ERS for statistical purposes, administrative data on workforce, housing, justice, and education could be incorporated into ERS studies of program participation. The Evidence Act requires that the Office of Management and Budget (OMB) establish a single application process for access to confidential federal data. Section 3564(f) notes that nothing in that Act preempts applicable state law regard-

ing the confidentiality of data collected by the states. It is expected that OMB and the statistical agencies will gather, interpret, and deconflict any laws and regulations related to data access.

USE OF COMMERCIAL DATA IN THE CFDS

The FED has a strong track record of using proprietary scanner and sales data to estimate detailed food prices and quantities of purchases, retail sales, consumption, and purchases of food for at-home and away-from-home eating. For example, data on consumer purchase transactions and retail point-of-sales and information from food labels have been used to help answer questions about the cost of a healthy diet and about how the nutrient content of food products changes over time.

To fully analyze program participation through changing social, economic, and policy conditions, the use of administrative data alone from those programs is insufficient. Data from surveys and commercial sources can provide more comprehensive information, whether on the full population of households or retailers, to model take-up rates; or to model the population effects of participation on health outcomes; or to model population subgroups, such as veterans. Data available through commercial research organizations or partnerships with commercial food providers are especially useful for improving information about the food environment. Such data can help address critical questions in areas such as (i) dietary patterns and nutrition; (ii) the food environment, including the availability of stores and restaurants, food prices in an area, and community characteristics; and (iii) industry response and agricultural sector adaptations to these many changes (Larimore et al., 2018).

In summary, because of their potential value to research, ERS should continue to invest in acquiring and understanding commercial data.

As these new sources of data become available for use by food researchers and evaluators, there is also a need for a deep understanding of

their strengths and weaknesses. As a general class of data, “organic data,” which arise out of the broader information ecosystem, are not designed for research purposes, but can still have great value in part because they tend be massive, with millions or more of observations. They are also often generated in close to “real time” (retail scanner data capture the exact time and date of each scanned transaction), and in a way that is unobtrusive for measuring phenomena since there is no direct engagement with subjects. For example, retail scanner data are captured as part of the natural store checkout process.

While commercial data will certainly play a growing role in food research, measurement, and assessment, hurdles need to be overcome before their full potential can be realized. Chief among these are access, coverage, and transparency issues. Often, one of the most difficult aspects of using commercial data is negotiating access (NASEM, 2017a). For example, while non-ERS-affiliated users can obtain access to retail data from Nielsen through the Kilts Center for Marketing at the University of Chicago’s Booth School of Business, they face limitations—for example, to information on precise geographic locations—that impede some kinds of analyses.

Regarding coverage and representation, to obtain valid and reliable conclusions it is critical that the data be representative of the populations or subpopulations of interest and that the degree of representativeness be known. For example, in some commercial databases, lower-income consumers are underrepresented; at the retail level, smaller, independent stores or private-label products may be excluded. In some cases, design data can be used to correct for coverage issues and selection biases in organic data. Additionally, in terms of transparency, organic data often lack the traditional types of documentation researchers are accustomed to having. This applies not only to the data content but also to the ability to trace the origins of the data or changes made to the data at various points before reaching the researcher. For example, the consumer panel widely used by researchers does not collect individual prices paid by consumers when they shop at stores where firm-side data are available; instead, what is reported in the data is the average weekly price from these other sources, sometimes averaged across various geographies.

Overcoming the above hurdles will guide ERS’s quest for accurate and applicable data sources.

For example, ERS has an admirable tradition of using Nielsen and IRI data while also comparing findings, totals, and coverage with other sources and while documenting the strengths and weaknesses of these scanner and sales datasets. Often, ERS-funded work is the sole source of information about the accuracy of these proprietary data.

DATA QUALITY AND DATA ACCESS

ERS must continue to envision a future when there is much more blending of mixed data types. Whenever a major survey such as FoodAPS is designed, the role of administrative data or other data types should be considered in the overall design and estimation strategy, and the considerations should include the coverage, quality, timeliness, accessibility, and cost of those data. Even with the inevitable trend toward mixed-data models, surveys will continue to be important to statistical agencies for the foreseeable future. Surveys provide household- and individual-level data that cannot always be acquired through other means.

In the new data paradigm, administrative and commercial data must be evaluated for quality as would-be survey data. As ERS continues to enhance data products through more expansive use of proprietary data and links to state, local, and other federal administrative data, quality assessment will be critical. Other questions that are important for evaluating these sources include: (i) Are the data longitudinal? (ii) Can the changing

___________________

⁴ Some sources, such as the IRI Consumer Panel, include weights that are provided to ERS as part of the data purchase. Other sources, such as InfoScan data, do not come with weights.

platforms among proprietary providers and state/local program administrators be harmonized? and (iii) Are the internal algorithms used to compile the data transparent?

While standards are emerging for gauging the quality of stand-alone data and of linkages in sources such as those contained in the CFDS, the quality of data can only be thoroughly assessed through their regular use by researchers.

Taking advantage of multiple data sources requires that the ERS FED partner with other agencies to leverage strengths. For example, ERS may decide it is cost-effective to leverage Census survey methodology expertise for some data projects. In other cases, the agency should take advantage of interagency work on developing standards to assess survey and administrative and proprietary data.

This report presents a series of recommendations that span the current and past CFDS and also makes suggestions for the future. The most important recommendations, not listed in priority order, relate to (i) checking data and linkage quality, (ii) enhancing access to existing data and future data sources by outside researchers as well as through existing relationships, with greater geographic detail, (iii) finding ways to incorporate more administrative data into the CFDS, (iv) systematically focusing on the CFDS role in serving monitoring needs (e.g., measuring food security consistently) and causal research needs through longitudinal designs, and (v) creating policy databases to enhance causal research.