Evaluating Food Assistance Programs in an Era of Welfare Reform: Summary of a Workshop (1999)

Workshop participants noted that a major consideration in planning future evaluations is assessing the data currently available and, if necessary, determining what other data are still needed. Research to date has produced a vast amount of information on many issues involved in the evaluation of food assistance programs, including nutrition, health status, and dietary behaviors and trends. Furthermore, a voluminous amount of research has been done on the food assistance programs themselves, from which much can be learned and applied in future program evaluations (for example, see Food and Nutrition Service, 1997). This information is particularly useful in monitoring trends and shifts in food assistance program participation and effectiveness. Nevertheless, as the provisions of the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) alter food assistance programs, new questions must be posed to capture the effects of program changes. Researchers will have to determine if the existing data are meeting these new demands and, if not, what data needs exist. In order to successfully use the full capabilities of the existing data to answer the questions in future program evaluation efforts, it would also be helpful if existing surveys, studies, and datasets that collect appropriate baseline information and are good sources of pre-reform data are identified as such, and these baseline data are made readily available to researchers.

Workshop participants discussed the importance of using efficient data collection methods. Whether at the national, state, local, or individual-case level, an

improved data collection program should not only address questions that are apparent now, they agreed, but it should also be focused on creating a foundation for future policy research. Participants identified four principles for a data collection program to accomplish that task:

The data that are currently available on the food assistance programs and the approaches that can be used to evaluate their effectiveness have primarily come from national surveys, state and local data resources, and administrative records. Past evaluation efforts, such as those conducted by the Food and Nutrition Service, can also be a valuable resource for future program evaluations. No single one of these resources is better than the others; all of the methods, however, have their strengths. The applicability of the data provided by the particular methods depends on the specific questions being asked. For example, surveys are very good at collecting qualitative information on each case, such as quality-of-life measures, whereas administrative data are an efficient source for quantitative information, such as the duration of receipt of benefits. State and local data often provide information that is useful in evaluating program changes on a smaller scale. Thus, when assessing the data needs for future program evaluations, all current data resources should be considered and the most appropriate method or combination of methods used.

Workshop participants emphasized the importance of using the data provided by ongoing national surveys, which have many advantages. These surveys

often collect a great deal of detailed demographic information on each ''case" in the survey that is infrequently represented in the other sources of data, such as education, levels of job training, adequacy of housing and health care, nutritional information, and the situations surrounding the need for public assistance. Whereas administrative data can provide information on participants only while they are active in the programs, survey data are an optimal tool to obtain information on participants long after they leave the programs. Longitudinal data also allow for the measurement of effects over time. Data obtained from surveys provide much of the information currently used in determining program effectiveness, such as the estimated number of people eligible for program participation, from which participation rates are derived. Most of the information on food intake and nutrition of the population has come from long-term, large-scale surveys. For example, the National Health and Nutrition Examination Surveys, the Continuing Survey of Food Intakes by Individuals, and the Diet and Health Knowledge Survey provide a wide array of information about the diets and health of the entire population and can be extremely helpful answering many of questions associated with food assistance program evaluation. Survey representatives provided overviews of these datasets at the workshop.

The National Health and Nutrition Examination Survey (NHANES) is the primary vehicle for the collection of objective health information and nutritional status measures for the U.S. population. Since the 1960s, NHANES health and nutrition data have been used to set policy, evaluate and plan programs, and track progress in meeting national health promotion and disease prevention objectives. Originally called the National Health Examination Surveys, these surveys began in the 1960s. In 1970, an expanded nutrition component was added to provide data with which to assess nutritional status and dietary practices, and the name was changed to the National Health and Nutrition Examination Survey. Data for NHANES are collected in household interviews and direct standardized physical examinations in mobile examination centers. The surveys use complex, multistage, stratified, clustered samples of the civilian, noninstitutionalized population. Physical examinations and objective measures are used because the information collected cannot be furnished or is not available in a standardized manner through interviews with the people themselves or through records maintained by the health professionals who provide their medical care. NHANES is conducted by the National Center for Health Statistics (NCHS). Data files from NHANES are available on CD-ROM, and the results are published in peer-reviewed publications, government reports, and on the Internet.

The National Health and Nutrition Examination Survey has had three phases to date, with a fourth phase currently being planned. The first National Health and Nutrition Examination Survey (NHANES I), 1971-1974 and 1974-1975, is a

cross-sectional study of the U.S. population. The data collected permitted the estimation of the prevalence of specific diseases and other health-related measures for analyses and measured and monitored indicators of the nutritional status of the population. The second National Health and Nutrition Examination Survey (NHANES II), 1976-1980, continued NHANES I, but also gathered data on participants in the Food Stamp Program and programs for the home delivery of meals, allowing comparisons between participants and nonparticipants of similar socioeconomic status. NHANES II covered illness or "target conditions" included in NHANES I as well as several new conditions, including diabetes, kidney pathology, liver function, allergy, and blood tests for environmental pollution. NHANES III (1988-1994) had a general structure similar to that of the previous phases, but added 30 topics of study, including high blood pressure, high blood cholesterol, obesity, passive smoking, lung disease, osteoporosis, HIV, hepatitis, Helicobacter pylori infection, immunization status, diabetes, allergies, growth and development, blood lead levels, anemia, food sufficiency, dietary intake, and biochemical and urinary assessments.

The next phase (NHANES IV) is currently being planned as a continuous survey. Data collection will begin in 1999. Many risk factors and health conditions will be measured, including physical fitness, weight, heart disease, osteoporosis, dental disease, nutritional status (based on nutritional biochemistries, hematological determinations, and anthropometry), serum lipids, and hypertension. Similar to NHANES III, interview questions will be asked regarding dietary behavior for the past 24 hours, including specific foods eaten, alcohol and water intake, dietary supplement usage, and nonprescription drug use. NHANES IV will be linked to the National Health Interview Survey¹ (NHIS) in design at the primary sampling unit level and in content. Additional linkages between NHIS and NHANES may be made, but that decision will be evaluated as the survey progresses.

The Continuing Survey of Food Intakes by Individuals (CSFII) is conducted by the Food Surveys Research Group of the Agricultural Research Service of USDA. CSFII uses a sample of 16,000 individuals (5,000 people annually) representing the population in 50 states and the District of Columbia over three years. Unlike NHANES, which collects a wide array of health information, the CSFII is devoted solely to data on individuals' dietary intake. In-person inter-

views are conducted for two nonconsecutive days, and people are asked to provide 24-hour dietary recall to ascertain information such as the amount of food eaten, the number of servings that were from the 30 specified food groups, the number of nutrients or dietary components consumed, and the percentage of the recommended dietary allowances (RDAs) provided. Questions about participation in food assistance programs, household expenditures on food, and food sufficiency in the household are also asked. In 2000, CSFII will be merged with NHANES. Data files from CSFII are available on CD-ROM.

The Diet and Health Knowledge Survey (DHKS), conducted as a telephone follow-up to the CSFII, is designed to provide more information on the factors that affect food choices and people's knowledge and attitudes about the Dietary Guidelines for Americans. In DHKS, information is collected from the same individuals as in the CSFII. Using adults 20 years of age and older from CSFII (6,000 individuals over three years), DHKS interviewers conduct telephone surveys two to three weeks after CSFII interviews. Questions establish participants' awareness of the relationship of health problems to diet, knowledge of foods that provide necessary nutrients and the established dietary recommendations for those nutrients, perceptions of the importance of following the Dietary Guidelines for Americans and their own diet quality, food safety and handling behaviors, and attitudes about using food labels. The survey popularly known as the "What We Eat in America Survey" refers to the CSFII/DHKS 1994-1996. Data files from DHKS are available on CD-ROM.

Several other national surveys can also provide data useful to food assistance program evaluations. For example, the National Longitudinal Study of Adolescent Health (called Add Health) has a nutrition component, and several other surveys obtain information about the health of their participants (see National Research Council, 1998b). Surveys, such as the Survey of Income and Program Participation (SIPP) and the Current Population Survey (CPS), obtain information about and estimates of people who are eligible to participate in programs and changes over time. The Panel Study of Income Dynamics (PSID) is also an important data resource. PSID, which began in 1968, is a longitudinal survey of a representative sample of U.S. men, women, and children and their families. The original sample size was 5,000 families, and each year the same families are contacted. PSID obtains very useful information on family economics, such as income, occupation, expenditures on food and housing, and participation in public assistance programs. Data collected through the National Vital Statistics System by NCHS is another useful resource for food assistance program evaluation.

These data, which are based on vital records filed at the state level, provide information on births (including selected maternal and infant risk factors), deaths (including infant deaths and deaths from specific diseases), fetal deaths, marriages, and divorces. The data from these surveys are available on CD-ROM, in technical journals, and on the Internet. Workshop participants stressed the importance of using the data provided by these national surveys for program evaluations; however, participants also acknowledged that surveys that rely on self-reporting have inherent limitations, including loss of data due to recall error and accuracy of respondents' answers to survey questions.

The National Nutrition Monitoring and Related Research Program was established by Congress in 1990 under the National Nutrition Monitoring and Related Research Act (P.L. 101-445). The act called for continuous monitoring of the dietary, nutritional, and health-related status of the U.S. population and of the nutritional quality of the food consumed in the United States. The program obtains this information from surveys, surveillance systems, and other monitoring activities, such as NHANES and CSFII/DHKS. The design of this national nutrition monitoring system is driven by a number of nutrition and health outcomes; workshop participants mentioned that it might be useful to look at how they align with the types of issues important to evaluating food assistance programs.

Although the national surveys provide much needed information to answer an array of questions, are they able to capture the effects of variations in food assistance programs at the state and local levels? Workshop participants noted that, although national studies are needed to answer some targeted questions, state-level data will be needed to effectively evaluate food assistance programs. The national surveys are costly, and, with the anticipation of future budget cuts, smaller studies may become more cost-effective. Furthermore, as individualized state programs develop, small, focused studies that use natural variation across states to identify best practices may be the most appropriate. Some effort must be put into tracking on a state-by-state level the programs that are devised and their variations across states.

Workshop participants suggested that integrated quality control (QC) systems might be a useful source of state-level data on the food assistance programs. For many years, the major welfare programs, including the Food Stamp Program, have had federal-state QC systems for the purpose of correcting faults in program administration that contribute to erroneous payments. QC systems attempt to measure the extent and dollar value of "errors" in administration, identify the types and causes of error, and specify and monitor corrective actions taken to eliminate or reduce errors. States can be held liable for the cost of benefit

payments made in excess of federally established error-tolerance levels, often referred to as target error rates. PRWORA leaves present QC laws intact.

The core of the QC systems is the QC case survey. The case survey for each program compiles the results of a statistically valid sample of cases drawn in each state. QC personnel, who verify the eligibility information for each case and conduct a full field investigation, including home visits, subject each selected case to a thorough review. The QC data offer a rich source of information for learning what is happening in the programs.

The Food and Nutrition Service (FNS) is using the data collected from QC systems to characterize food stamp participants and the benefits they receive (Carlson, 1996). This ongoing review of food stamp households, conducted by state reviewers, is designed to measure the accuracy with which eligibility and benefit determinations are made. The sample for the most recent release provides a broad array of detailed demographic and economic information on more than 51,000 food stamp households participating during FY1995. Characterization of food stamp participants and benefits and how they vary from state to state must be very complete to be meaningful.

There are other state-level data collection efforts as well. For example, FNS is conducting the WIC Participant Characteristics (PC) studies. In addition to the annual national estimates of WIC participation, primarily based on data from the CPS, FNS produces annual state-level estimates using both the CPS data and state-level economic and administrative data. The PC studies were negotiated between FNS and the state WIC agencies in 1988, with consultation from the Centers for Disease Control. Currently, the WIC PC studies collect data on such factors as income, nutritional risk factors, and participation in other assistance programs for the 7 million WIC participants.

PRWORA requires that states prepare quarterly reports that include all of the information necessary to calculate TANF participation rates, imposing penalties to ensure prompt reporting. Because the existing administrative databases are not able to provide this information, surveys will have to be designed. These surveys could provide part of the foundation of a statistical system for monitoring the impact of welfare reform, including the effects of food assistance programs. Because the states are concerned about meeting the reporting requirements of the Department of Health and Human Services (DHHS), however, there is a risk that they will use their own data definitions, which will vary. Some workshop participants suggested that DHHS play a role in assisting states to ensure that data are reported in standardized ways so that an accurate national picture of the food assistance programs and those participating in them is achieved.

Although state-level data currently do exist, workshop participants agreed that more data are needed to accurately capture the effects of welfare reform on the food assistance programs and variations across states and communities. The heterogeneity of the data and the very large differences in the quality of the data

at the state and local levels need to be studied. Furthermore, stateand local-level data on many issues are not available.

Workshop participants agreed that administrative records are an extremely informative resource. The use of administrative records is expected to become easier in the future because of technological advances. Used either as an alternative to the national surveys or in conjunction with survey data, administrative records provide information on large sample sizes and are an excellent record of certain kinds of events. For example, information that people often have trouble remembering in interviews, such as the exact amount of benefits they received or the date on which they receive assistance, are carefully recorded in administrative databases (Brady and Snow, 1996). Administrative records, however, often do not provide information about such characteristics as job history, disabilities, adequacy of health care, and nutritional status. Due to state variations in program structures, the national surveys may not adequately provide the state-level data needed for effective analyses of these smaller-scale variations. Thus, it is expected that future program evaluations will make greater use of administrative data.

Administrative data for evaluating food assistance programs can be obtained from a number of existing sources because of the overlap of food assistance programs with other public assistance programs. For example, the unemployment insurance system and payroll tax records provide information about earnings, whereas Medicaid and Supplemental Security Income records provide information about the use of all forms of public assistance. Workshop participants suggested that it is important to pursue efforts to use administrative records for descriptive purposes. Families who leave cash assistance programs may still receive food stamps and WIC; those administrative records can be used as surveillance mechanisms.

Administrative records, however, are not sufficient in themselves to provide all of the needed information. Workshop participants suggested linking administrative data to national surveys to obtain information on such factors as other program participation, what happens to participants when they leave a program, and what their circumstances were before coming to the program. For example, the Social Security Administration has matched earnings data with information from Supplemental Security Income, SIPP, and the CPS. Similar methods could be implemented to match state data on food stamps and TANF with SIPP. Linking to administrative data offers more promise, providing information on current and past program usage, including pre-reform use.

The linking of data systems poses some serious methodological challenges. For example, units of analysis may differ (e.g., TANF data are based on cases; unemployment insurance data are based on individuals; and tax data are collected

on filing units), the timing of collection may vary (e.g., unemployment insurance data are reported monthly, tax data are collected annually), and sampling frames may not match (e.g., a sample of the people receiving assistance, the flow of people in and out of the programs).

A serious concern about the increasing use of administrative data is protecting the privacy of program participants, especially as more advanced databases are designed that include individual information, such as name, gender, date of birth, and other data elements that will make linkage to other systems easier. Also, legal issues limiting access to confidential records could present a problem to using administrative records. There are also ethical issues to consider when confidentiality has been promised to survey participants. Workshop participants agreed that these privacy problems may ultimately require legislation for their resolution.

The FNS research program has focused recently on measurement of hunger, cost-effectiveness of the electronic benefits transfer system, assessments of nutrients in the child care food program, and cost saving in the WIC program (see Food and Nutrition Service, 1997). Workshop participants agreed that the Food and Nutrition Service is doing important and useful work that should be continued.