Evaluating Food Assistance Programs in an Era of Welfare Reform: Summary of a Workshop (1999)

Workshop participants identified numerous research issues, methodologies, and data needs that the ideal food assistance program evaluation effort should consider. Designing such an effort, however, may not be realistic, given such factors as cost and methodological limits. The workshop devoted one session to a discussion of the research needs that participants considered the most important and practical and could be done efficiently to ensure that the true impacts of welfare reform on food assistance programs are measured.

When examining what research issues are involved in evaluations of food assistance programs, the main issues fall into two categories: those that deal with economic measures of program effectiveness, and those that deal with nutrition and health issues. It is practically impossible to suggest one category is inherently more important or relevant than the other, particularly when visualizing the "ideal" evaluation program, which would logically include aspects of both categories. It is important to note that there was no consensus among workshop participants as far as prioritizing the areas of research in terms of importance or need. As the main goal of the workshop, however, was to help identify the key issues for a broad program that examines food assistance as a whole, remaining consistent with the mission of the Economic Research Service, the nutritional issues were emphasized. What follows are the key issues that should be considered when designing an evaluation plan for the food assistance programs, as identified by the workshop discussions.

Any program evaluation, by its very design, has the capability to answer two key questions: What would the participant's status be if the program did not exist? What is the participant's status given specific program changes? In the case of evaluating food assistance programs given the changes implemented by the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) however, the most important research question is much more complex. Future program evaluations need to address whether, in this era of welfare reform, the nation's food assistance programs will continue to serve as a national nutritional safety net for those in need, particularly children.

Program evaluations should be developed that go a step beyond the measurement of whether the programs simply meet their primary objectives, allowing for the evaluation of the programs in their role in the entire national social and health support system. A comprehensive research framework should encompass the diverse domains affected by the food assistance programs, including nutritional and health status, economic security, food security, and individual and family health and well-being. A list of all the issues identified by workshop participants that should be included in an evaluation effort appears in Box 5-1. Addressing all of these issues in a comprehensive manner may require several separate research efforts.

Workshop participants agreed that program interaction at all levels—both between welfare and food assistance programs and among the various food assistance programs—is a top priority when considering new research agendas to evaluate these programs. For instance, emergency food assistance programs, such as food banks, soup kitchens, and food pantries, may warrant further study to determine whether they substitute for or supplement each other. Also worthy of study is what happens when families and individuals run out of food at the end of the month and, in that connection, what demands are placed on emergency programs. The question of the extent to which these food sources serve an income-smoothing function gains importance as more low-income families move into the labor market and are subject to economic volatility. High priority should also be placed on studying the interactions of food assistance programs with the tax and income transfer systems. A variety of studies across disciplines—labor market studies on how food stamp usage relates to the minimum wage; nutritional studies on the link between program participation and nutritional status; the effect of housing costs on program participation; income security studies on, for example, what needs go unmet when families must spend more for food—all contribute to this broader understanding.

Interactions between outcomes are also important. When the impacts of a program cannot be determined directly, research can link those impacts to outcomes that can be measured directly or through proxies. For example, are children who live in low-income families receiving more food from school programs than through food stamps? If so, what are the implications? Which children are or are not reached? The school programs remain a source of food for some, such as illegal immigrants, who are no longer eligible for food stamps. What will happen to the children of illegal immigrants? Moreover, research on the school programs has focused on nutritional aspects; new analysis could be extended to the effects on family well-being and the role of these programs in supplementing family income. Other interactions may include the effect of possible fluctuations in food assistance program participation on the U.S. food commodities market, and, beyond it, to the entire economy.

Workshop participants agreed that the study of the dynamics of participation is critical and that complete data on program participation trends are needed. Modeling the determinants of program participation is a high priority in an era of devolution. Data are needed on multiple-program use—what programs people depend on for which periods of time and which ones they turn to when they lose eligibility for Temporary Assistance for Needy Families (TANF). Data collected

from such large surveys as the Survey of Income and Program Participation (SIPP) and the Survey of Program Dynamics (SPD) will help provide information on participation,¹ but assets are measured at only one time in SIPP, and food stamp participation is largely affected by asset changes. Moreover, assets are subject to much change among the population receiving public assistance. Study of the dynamics of participation can also reveal much about those that go in and out of the programs, those that are excluded from program eligibility, and high-risk populations. Special one-time, targeted surveys, such as one focusing on reasons for leaving a program, could be a useful strategy to supplement larger databases.

Long-term as well as short-term participation dynamics merit attention. It will be helpful to know how people ultimately reach self-sufficiency and what the long-run caseloads and costs will be. Participants suggested using SIPP data for learning about short-term dynamics; SPD data will permit examination over a longer term. Very long-term studies (e.g., several years or a decade) of participation dynamics could also provide insightful information. Microsimulation modeling was also suggested as a tool that can be very effective in predicting possible effects of new program changes. But workshop participants acknowledged that the models that are used must be transparent and widely accessible.

Actual participation rates, both in the food assistance programs and the other public assistance programs, can be a very useful tool in evaluating the effectiveness of these programs. Increases or decreases in participation rates, particularly in specific populations of recipients, are often good indicators of the effect that a particular change or event may be having on that program's effectiveness. Follow-up of participants who leave the food assistance programs—either due to the provisions of PRWORA or other circumstances, such as those accompanying a strong economy—can provide insight into a wide array of participation factors, the effectiveness of the programs themselves, and even a measure of the effects of PRWORA. Study of this population could benefit researchers by providing detailed information about specific questions, including the economic or social conditions that caused participants to no longer need assistance; how long they stay off the assistance; their quality of life after exiting the program, both financially and nutritionally; the factors leading to why they may need to return to the program; the form of assistance for which they apply first; and other factors.

The food assistance programs are very responsive to changes in the economy. For example, since its all-time peak participation rate in spring 1994, Food Stamp Program enrollment has declined continuously and is expected to continue to decline (Richardson, 1997). This decline is mainly due to the changes implemented under PRWORA and the growing economy and declining unemployment

since the passage of PRWORA. Provisions of the act, however, designate specific assistance programs with fixed amounts of money that will not increase, even during recessions. And it is in times of recession that poverty generally increases, as seen in 1989 through 1993, and the need for food assistance increases. A key issue for future program evaluations is whether, in times of depressed economic growth or recession, the food assistance programs continue to serve their purpose effectively. Participation rates and the trends in them could serve as useful tools to measure program effectiveness as the status of the economy changes.

Sample surveys and administrative databases will be called on to provide more and different information to support the evaluation of food assistance programs in this era of devolution. Existing surveys and datasets that provide useful baseline data will also be a necessary resource in future evaluation efforts. For evaluative purposes, workshop participants suggested that the optimal research tool would be a national, longitudinal, multiyear survey linked with administrative records across the country. In the absence of the ideal, however, efficient use of the existing data resources must be made. A large portion of the data currently being collected by the national surveys could be used in food assistance program evaluations. Surveys such as the Panel Study of Income Dynamics can provide quantitative information about program participation and qualitative information on the characteristics of the participants in a wide variety of contexts. Nevertheless, workshop participants acknowledged that the need to link national survey data to administrative data is a priority. National survey data and administrative data complement each other well, one often providing the information that is lacking in the other.

To take full advantage of the national surveys, as well as to keep research costs down, linkages between the national surveys were also suggested. Linkage across datasets could provide multiple sources of information on the same factor, thereby possibly improving the reliability of the data. The Food and Nutrition Service has produced food assistance program participation data by successfully linking databases with the National Health and Nutrition Examination Survey, the Continuing Survey of Food Intakes by Individuals, and the Quality Control data on food stamps. Methodological challenges of linking data, whether across programs or datasets, include the lack of a standard unit of analysis and a standard period of time for which the data are representative. These challenges would need to be considered to optimize the use of data that currently exist. In addition, the linking of survey data raises important issues of privacy.

Studies at the national level should continue to be conducted to examine the effects of TANF as shown in national databases, yet much more research at the state and local levels is needed. The Office of the Assistant Secretary for Plan-

ning and Evaluation in the Department of Health and Human Services has awarded grants to 14 states and counties to enhance their data systems by linking administrative datasets and by conducting surveys which will also be linked to administrative data, so that such data can be better used for research purposes. Furthermore, it was suggested that the states that have successfully implemented plans to handle the changes enacted under PRWORA could be studied and their experiences shared.

Workshop participants emphasized that, in addition to improving the national data collection resources, more state-level data are needed. To gain more information at the state level, workshop participants suggested that national surveys could be augmented by adding state supplements. Data collection at the state level should be supported by federal agencies by providing the states with necessary technical assistance in designing their own data systems. Because variation among programs across states is inevitable, since more responsibility now rests at the state level, more measurement needs to be conducted at the local level, including how localities select programs, a new area of research.

Given all of the effects of the changes caused by welfare reform, workshop participants emphasized the importance of collaboration among programs across the entire public assistance arena. In understanding the future demand for and consequences of food assistance program policies, it will be essential to examine what is going on in these other components of the welfare system and the interaction among programs and policies. The Urban Institute's New Federalism project was cited as an example that may offer data to address many of these more complex questions. Participants acknowledged the need for collaboration among research efforts, to promote communication between researchers and the sharing of information. Because the effects of PRWORA involve many federal agencies, collaboration among them should be encouraged so that they are able to monitor and take account of one another's policies and actions. Researchers, too, need the cooperation of the program staff of the agency funding the research efforts.

Workshop participants acknowledged that different evaluation methods will be needed to address different questions. Both experimental and nonexperimental methods should be considered. Given the extent of the structural changes made to the programs under PRWORA, however, states may be less likely to use experimental designs in evaluating public assistance programs. Nevertheless, recent welfare reform evaluations have effectively used multiple methods, including experimental evaluations focused on particular changes, with extensive outcome analyses tracking changes in participation rates, types of administrative actions, benefit payments, and costs; process analyses to understand what policies were implemented and how; ethnographic studies to understand the context

of client decisionmaking and the consequences of interactions between policy changes and client actions; and embedded studies to address possible secondary effects, such as the consequences of policy changes on the well-being of children in welfare households. Lessons from these evaluations could be applied to evaluations of the food assistance programs.

This workshop was convened to assist the Economic Research Service (ERS) in developing a comprehensive and collaborative agenda for federally funded food assistance research. The concluding session addressed what next steps ERS could take. Based on the discussions at the workshop, participants agreed that a program with broad scope examining food assistance as a whole is most consistent with ERS's mission, as well as meeting the needs of FNS, as the primary program agency. Participants envisioned additional ways a federal research program interacts with the research and policy communities. ERS could allow researchers complete access to data, in particular, to those data to answer the questions in which it is most interested. ERS could also provide funds for data collection to fill important gaps. Another important step is to support the linking of administrative databases, including food stamps, making these data available to the states. ERS should focus its data collection efforts both to meet program agency needs and create a foundation for future policy research.

Some participants suggested that the general research community be kept abreast of the efforts being conducted in research related to welfare reform, particularly food and nutrition issues, in order to stimulate interest and involvement in existing studies. Providing information at professional meetings about the food and nutrition programs is one way to do so.

The goals of the food assistance programs are relatively straightforward: to improve nutrition and reduce hunger and poverty. Devising appropriate measures of progress toward this goal, however, has proven to be difficult.

This workshop brought together many who work on the evaluation of food assistance programs, policy analysis, survey methods, nutrition, child nutrition and child development, outcome measurement, and state welfare programs, in order to provide ERS with information that could be used to develop a framework for its research program. The discussions indicated that nutrition and health status and income and poverty status are two important and necessary categories of issues that must be considered to effectively evaluate the food assistance

programs. Although the discussions at the workshop were designed specifically to help ERS design its evaluation program plans to continue to ensure that the basic goals of these programs are achieved, it is hoped that the discussion captured in this summary are of use to all who work in evaluation research.