Evaluating Food Assistance Programs in an Era of Welfare Reform: Summary of a Workshop (1999)

The effects of the changes made by the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) to the food assistance programs are likely to be reflected in many of the programs' outcomes. The outcomes likely to be affected include rates of participation in food assistance programs, the characteristics of participants, levels of food security, the dietary and health status of participants, the economic and social well-being of participants, and the actions of states and localities responsible for Temporary Assistance for Needy Families (TANF) and other welfare programs. The food assistance programs often go beyond meeting a set of specific objectives to provide an entire social and health support system, which consists of an interwoven fabric of programs and services provided by federal, state, and local government agencies, as well many other important local organizations and individuals. The extent to which the Food Stamp Program and other programs will continue to provide this kind of safety net to those in need is at issue. Also at issue in the evaluation of the programs is the greater heterogeneity in outcomes that is likely at the state and local levels as a result of the PRWORA provisions. Addressing this heterogeneity may prove to be a challenge to researchers, but it must be done in order to obtain accurate evaluation results. The discussions at the workshop identified specific research questions that can help guide the development of effective evaluation studies.

PRWORA contains strong work requirements, a performance bonus to states for moving welfare recipients into jobs, state maintenance-of-effort requirements, and comprehensive child support enforcement. The work requirements stipulated in the act require 25 percent of all families receiving assistance in each state to be engaged in work activities or have left the rolls in FY1997, increasing to 50 percent in FY2002; single parents must participate for at least 20 hours per week the first year, increasing to at least 30 hours per week by FY2000. Two-parent families must work 35 hours per week by July 1, 1997. A work activity is defined as paid or unpaid employment, on-the-job training, work study, internships, apprenticeships, participating in job search assistance for up to six weeks, or community service. PRWORA also provides resources for families moving from welfare to work, including $14 billion in child care funding over six years, which is an increase of $3.5 billion over current law, and the guarantee of coverage of medical care needs of individual family members on TANF at work or engaged in other required program activities. The act stipulates a time limit, imposed at the federal level on cash assistance: adult recipients will be allowed to receive cash assistance for a maximum of five years; after two years of receiving assistance, each adult recipient is required to participate in a work activity. Teenage parents must be attending school and living in an adult-supervised setting. States do have the option to exempt up to 20 percent of their caseload from the time limit for a variety of reasons, such as if the recipient has a disability or is the victim of domestic violence.

Workshop participants pointed to both the participation rates of food assistance programs as good indicators to use in evaluating the effectiveness of the

programs. The Food and Nutrition Service (FNS) has elaborate reporting systems in place that provide information about the distribution of participants and program costs over time and geographic areas. The participation rate is the ratio of the number of people who participate to the total number of people who are eligible. The number of actual participants is obtained from administrative data, and the number of people who are eligible is estimated from sample surveys; thus, the participation rates themselves are estimates. Typically, program participation rates are less than 100 percent of the eligible population. For example, in January 1992, the Food Stamp Program participation rate was 74 percent among income-eligible individuals; 89 percent among eligible households with incomes below the poverty line; 95 percent among eligible preschool children, and 86 percent among eligible children under 18 years of age (Trippe and Sykes, 1994). Hence, the Food Stamp Program is reasonably successful at reaching those who are eligible for benefits and very successful at reaching eligible children (Devaney et al., 1997). However, with the strict time limits on assistance for childless adults and the limits on participation of legal immigrants established by PRWORA, a possible decline in Food Stamp Program participation and that of other food assistance programs may occur. As more people leave the welfare rolls under the TANF rules, however, food stamp benefits may be the only assistance for which they qualify, a fact that may result in an increase in Food Stamp Program participation. The trends that will be observed in the near future can be very useful for program evaluation.

Trends in participation have been a major topic of past research. For example, FNS has conducted several studies using data from the Survey of Income and Program Participation (SIPP) and the Current Population Survey (CPS) on food assistance program participation (see Food and Nutrition Service, 1997). Trends in participation rates offer insight into factors influencing program effectiveness, such as the impact of outreach efforts, the effects of changes in program rules and eligibility requirements, and the status of the economy (U.S. Department of Agriculture, 1997a). Also important to note when evaluating trends in participation is that program changes could affect the number of those who participate or the number of those who may be eligible, either of which would lead to a change in the overall participation rate. Reasons for changes in participation are not always obvious: the causes of the rise in participation in the late 1980s and early 1990s, for example, are still not completely understood. Workshop participants agreed that the trends in participation rates should be monitored closely to ensure an accurate assessment of the factors that influence any fluctuations. Participation rates at state and local levels should also be kept in mind when examining trends to help evaluate program effectiveness. It was also suggested that the ability to forecast whether more or fewer people will participate in the programs in the future, as well as studies involving the follow-up of participants who leave the programs, would be very useful.

Participation rates can provide insight on state-to-state variations in program

change and implementation as well as the effectiveness of each state's program. Participation rates in food assistance programs, however, are likely to vary as a result of changes in other public assistance programs in the states, so considerable attention needs to be devoted to describing how these changes and other factors can affect participation.

Another area of research interest is the effect of outreach programs designed to inform those eligible but not participating in the program. Past research suggests that outreach programs that inform nonparticipants of their eligibility have the potential to increase participation in the Food Stamp Program, for example, where it has been estimated that approximately one-half of the eligible nonparticipants do not know that they are eligible (U.S. General Accounting Office, 1988, 1990; Levedahl, 1995). PROWORA, however, has reduced or eliminated funding for outreach activities in some of the food assistance programs, and, for some programs, educational materials may be available only in English.

Workshop participants identified other factors that can limit participation. Examples include the stigma that may be associated with participation in food assistance programs, lack of access to Food Stamp Program offices, potential benefits that are too small to warrant the investment of time and energy needed to apply, and reluctance to seek assistance from the government. The effects of stigma can be very serious, especially for specific populations such as children, the elderly, and those in the middle class who suddenly fall into poverty.

Some critics argue that there are negative impacts on society from a wide variety of public assistance programs. A common concern is that participation in food assistance programs gives participants incentives not to work, depending on the benefits provided by the programs rather than pursuing opportunities for earning income. Some labor supply studies suggest that, in fact, food stamp program participation does result in some work reduction effects (Fraker and Moffitt, 1988; Ohls and Beebout, 1993). Estimating the degree of work reduction associated with food stamp benefits, however, is very difficult, because the households receiving the food stamp benefits are often participating in other programs whose benefits also decline as household income increases (Devaney et al., 1997). Long-term use of food assistance that results in dependency is a concern, especially given the limitations on other forms of assistance enacted under TANF. A family is considered dependent if 25 percent or more of its household income comes from public assistance (Greenberg, 1993).

Another factor that may affect participation in food assistance is the level of food insecurity that participants perceive in their individual situations. Food insecurity, as defined by the USDA, is the limited or uncertain availability of nutritionally adequate and safe foods or limited or uncertain ability to acquire acceptable foods in socially acceptable ways (U.S. Department of Agriculture,

1997b). It is believed that participation in food assistance programs may account for the fact that many poor households show no or minimal evidence of food insecurity, but this hypothesis still needs to be analyzed (U.S. Department of Agriculture, 1997b).

Workshop participants suggested that food insecurity itself is a possible area of research. It is socially undesirable to have people in this country who are food insecure or hungry. Furthermore, food insecurity and hunger can lead to serious nutritional, health, and developmental problems. Because the elimination of food insecurity and hunger is a goal of the food assistance programs, accurate measurement of the extent of these conditions will allow for the assessment of the effectiveness of assistance programs—or changes in their effectiveness as reforms take hold—in meeting their intended objectives (Carlson, 1996).

Data were collected in the early 1990s that created the scientific basis for defining and measuring food security and hunger. Food security indicators were included in phase 3 of the National Health and Nutrition Examination Survey, known as NHANES III, however, issues concerning measurement of the full population were not fully addressed and adequate national data were not available (Bickel et al., 1996). In response to these issues, FNS and the National Center for Health Statistics sponsored the Food Security Measurement Project, a major national survey that provides the data needed for the first comprehensive measurement of food insecurity and hunger in the United States. The first Food Security Measurement Project was conducted in 1995 as a supplement for the month of April to the CPS. The CPS is a monthly survey of 50,000 households that provides information on labor force characteristics of the U.S. population, such as employment, earnings, hours of work, and other indicators. The U.S. Bureau of the Census, which conducts the CPS for the U.S. Bureau of Labor Statistics, conducted the Food Security supplement. Workshop participants agreed that efforts to examine food insecurity are important and should continue.

Because the primary goal of the food assistance programs is to provide lower-income families access to a nutritious diet, the nutritional well-being of program participants should be a primary outcome measure when evaluating the effectiveness of the programs. Almost all of the food assistance programs have been shown to have a positive impact on the nutritional well-being of the people they serve, but workshop participants agreed that more research still needs to be done. An additional reason for continued research is that, as states consider changing the eligibility and benefit provisions of the food assistance programs to conform to TANF, the potential risk of decreasing nutrient intake and thereby adversely affecting recipients' nutritional status, must continue to be monitored in assessments of program effectiveness.

This issue is perhaps most important when considering the nutrition, health,

and well-being of the children participating in food assistance programs. Since almost half of all program participants are children (age 18 or younger), and more than half of the food assistance programs are primarily child nutrition programs, the importance of measuring the effects of program changes on the particular needs of children is clearly evident, and this should be considered a fundamental area of study.

Studies have shown a correlation between food assistance program participation and the availability of nutrients to participants. For example, participation in the Food Stamp Program increases the availability of calcium, vitamin C, and iron to participants (Fraker, 1990). Although the nutrients are available to participants, they may not be consuming them. A review of existing studies of differences in nutrient intakes between food stamp participants and nonparticipants failed to show consistent statistically significant effects of program participation on actual nutrient consumption (Fraker, 1990). However, evidence indicates that poor children who participate in the Food Stamp Program are more likely to consume more than 70 percent of recommended dietary allowances (RDAs) of most nutrients than nonparticipating poor children (Cook et al., 1995). RDAs represent the amounts of nutrients that are adequate to meet the needs of most healthy people.¹

The Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) has also been instrumental in improving the nutritional health of its participants. The National WIC Evaluation, conducted in the 1980s, was the most comprehensive WIC program evaluation. Findings indicated that WIC participation is associated with higher intakes of iron and vitamin C for both infants and children (Rush et al., 1988). WIC played a strong role in reducing iron-deficiency anemia among low-income infants and children younger than 2 years old, by recommending breastfeeding and requiring purchase of iron-fortified formula. Because both formula and cereal purchased through WIC must be fortified with iron, formula and cereal producers began fortifying all their products with iron—creating a spillover effect from WIC participants to the general population. This raises the issue of whether program evaluation should take into account the effects of food assistance programs on nonparticipants as well as participants and whether there are other unintended consequences of the programs.

WIC participation has also resulted in cost savings at the national level. According to a study conducted by the General Accounting Office, the greatest cost savings associated with the WIC program are recognized during the first year of life, in the form of reduced medical costs (U.S. General Accounting

Office, 1992). The study also recognized long-term benefits of the WIC program, which include protection of a child's cognitive development, resulting in a possible savings for special education that may have otherwise been required had the child not received adequate nutrition during pregnancy. Although data on such outcomes as physical and cognitive development at older ages can be collected readily, measurement of all of the longer-term effects of WIC presents a methodological challenge.

Another issue discussed by workshop participants was the fact that the work requirement of TANF means that more children are being placed in day care, including unpaid care provided by relatives and others. Little is known about the effects of unlicensed care on child nutrition and family income, or how to go about collecting such data.

The majority of the research regarding the effectiveness of the National School Lunch and the School Breakfast Programs focuses on the dietary effects of program participation. Findings from the School Nutrition Dietary Assessment study show that participation in the school meal programs does have positive effects on participants' nutritional status. For example, participants in the breakfast program have higher breakfast intakes of calories, calcium, protein, and magnesium, when compared with nonparticipants who eat breakfast (Devaney et al., 1997). Similarly, participants in the School Lunch Program have higher lunch intakes of vitamin A, calcium, magnesium, and zinc than nonparticipants. The focus of past studies has been on diet quantity; more recently, studies have addressed diet quality. Additional findings from the School Nutrition Dietary Assessment study indicate that both school programs are fairly successful at achieving their meal-specific RDA goals; however, the meals are not successful at conforming to the Dietary Guidelines for Americans in that the meals tend to be high in fat (Devaney et al., 1997; U.S. Department of Agriculture, 1995). (The USDA Dietary Guidelines for Americans describes food choices that will help meet RDA goals.) Workshop participants noted the need for periodic evaluations of school feeding programs based on dietary quality, because dietary goals change (see Institute of Medicine, 1994). Other national studies, such as the American Academy of Pediatrics' Community Access to Child Health Program and the USDA's Team Nutrition, which show that the quality of school meals is improving, also contribute to our understanding of the availability of nutrients to children and other child health issues.

Workshop participants agreed that more research is needed to investigate whether the School Breakfast Program is providing breakfast to children who would not otherwise eat breakfast, or is replacing breakfast normally provided in the child's home. Current data indicate that less than 20 percent of eligible children participate in the breakfast program (Devaney et al., 1997), and the participation rate for school breakfast is about one-fourth that for school lunch. FNS is currently sponsoring a reanalysis of the data from the School Nutrition Dietary Assessment Study to better understand the relationship of the availability

of the breakfast program to the probability of eating a substantial breakfast. The National School Lunch Program is essentially universally available, however; so determining whether it increases the likelihood of eating lunch proves difficult. Reasons for not participating in either school meal program include dislike of the foods being served and the stigma associated with free or reduced-price program participation (Devaney et al., 1997). These factors should be considered when evaluating participation rates.

Another research area identified by workshop participants is the impact of participation in school meal programs on school performance. Factors typically measured when assessing school performance include cognition, attention, scores on standardized tests, problem-solving ability, memory, verbal fluency, and creativity. Factors affecting school performance, such as attendance and tardiness, are also often measured.

Data show, in general, that children who eat breakfast perform better in school. Specifically for food assistance program participants, preliminary studies have shown a positive correlation between school performance and participation in the school breakfast program. Although the effects were small, one study did find that participation in the school breakfast program was associated with increases in test scores and reductions in tardiness and absences (Meyers et al., 1989). The latter are important because school performance is positively associated with school attendance. Workshop participants agreed that further research is needed to substantiate the effects of participation in the school meal programs. Studies should also look at whether there are systemic differences in schools that do and do not offer the breakfast program; its availability may vary according to community beliefs and values, which can further influence participation and the outcomes being assessed. Similar to the other food assistance programs, the longer-term effects of participation in the school meal programs on school performance have not yet been assessed.

In discussing studies of nutritional intake, workshop participants raised the issue of selection bias that commonly affects studies that measure the nutritional status of participants in food assistance programs. For example, those who take part in these school meal programs may be more aware of the importance of a proper diet than those who do not participate, and their experience may not truly reflect that of others eligible for food assistance programs. Findings from the School Nutrition Dietary Assessment study indicated that students who participate in the National School Lunch Program are a self-selected group of students who differ from other comparable students in either food preferences, needs, or appetites (Devaney et al., 1997). Selection biases such as these, as well as others, should be addressed when designing studies to evaluate program effectiveness.

Another factor that has a bearing on nutritional status is the dramatic increase in recent years in the amount of nutrition information provided to food stamp recipients at the state and local levels, opening up a new set of mechanisms by which the food stamp program can be expected to influence such outcomes as

participation rates and nutritional status. However, PRWORA reduced funding for some of these education efforts, an action that may further influence results. The impact of nutrition education needs to be considered in specifying process and outcome measures for new evaluations of the programs.

The tools for measuring nutritional status currently include data on diet, anthropometry, and blood measures collected from program participants. Workshop participants noted that research should address the robustness of these tools and should confirm that they are providing the right data for accurately assessing the expected nutritional outcomes for the participants of the food assistance programs. Workshop participants also discussed the importance of identifying and adopting a standard set of factors to be addressed when evaluating nutritional status, as such factors can potentially confound the effects being evaluated. Examples of these other factors include exercise, physical activity, illness, and behaviors, such as smoking.

Functional outcomes as they relate to the food assistance programs are also important to consider for evaluating program effectiveness. Functional outcomes include health status measures, well-being, and quality of life, for example, including disabilities and the prevalence of disease. Measurement of these outcomes typically goes beyond the physiological measures that are used to assess nutritional status, providing the complete picture of the health of program participants by using a broad definition of health that addresses the physical, psychological, and social conditions affecting a person.

Workshop participants also discussed participation in food assistance programs as it affects uses of household income, particularly whether the receipt of food assistance induces families to shift spending from food to other goods, such as shelter and clothing. Kuhn et al. (1996) reported that funds previously spent on food are reallocated to other needs, such as rent, clothing, and medical care. Households receiving food stamps may substitute the stamps for funds ordinarily devoted to food expenditures, thereby resulting in no increase in food expenditures, or an increase that is less than the dollar amount of food stamp benefits (Devaney et al., 1997). For instance, studies have shown that for each dollar increase in food stamp benefits, food expenditures increase by 17 to 47 cents (Fraker, 1990). It would be reasonable to assume that, under PRWORA, which is expected to decrease food stamp benefits by more than 15 percent from baseline budget projections by FY2002, retail food spending will decline² (Smallwood et al., 1995). Workshop participants raised the question of what this change in

expenditures will do to the availability and intake of nutrients for food stamp recipients and encouraged more research to measure the effect of substitution.

The effect of substitution and the role of the Food Stamp Program as an income supplement program could become more significant under PRWORA as states opt to cash-out their Food Stamp program and substitute income assistance, or cash, in place of coupons that must be spent on food. This option is expected to be simpler and less expensive to administer, and proponents of cash-out argue that providing the benefits in the form of cash rather than coupons allows households to more efficiently allocate the funds among food, shelter, clothing, and medical care (Kuhn et al., 1996). Nevertheless, a program requiring funds to be spent on food may be preferred politically to one allowing funds to be spent on other goods and services.

Evidence suggests that cash benefits reduce the stigma associated with using food stamp coupons, so overall program participation may increase (Ranney and Kushman, 1987). Studies suggest, however, that replacing coupons with cash, which is more readily transferred to other uses, would result in a reduction in household food expenditures (Fraker, 1990; Fraker et al., 1995). The substitution effect is not yet fully understood, since households receiving food stamps typically spend more on food than their monthly allotment of benefits. Workshop participants suggested that more research is needed to examine the effect of cash versus coupon benefits on food expenditures and consumption.

With PRWORA's emphasis on labor force participation, workshop participants suggested that future research should examine the use of food assistance to smooth fluctuations in earnings. Such ''income smoothing" may have an effect on variation in benefit levels, length of program participation, and administrative costs. This effect should be analyzed with respect to its effect on participants' health and well-being. Another area for research is examining the interaction of food assistance programs with other income assistance programs, such as the Earned Income Tax Credit, Medicaid, Social Security Insurance, low-income housing assistance programs, and other in-kind programs, as well as health insurance programs, all of which can further add to the effects of income smoothing. Also important is the extent of efforts made to provide outreach and education to potential recipients of support.

The effects of welfare reform on the delivery of food assistance services are a major area for study. It is crucial to determine whether the appropriate services are being provided, whether access to these services is readily available, and whether costs are being kept in check. Changes in the food assistance program environment as a result of welfare reform may require compensating changes to the food assistance programs themselves. In any area of program management, it is useful to identify possible changes before they occur, so that plans can be

developed in advance to respond to those changes. Several areas of food assistance program delivery that need to be evaluated were discussed at the workshop, including profiles of program participants, program costs, program interactions, program operations at the local level, program effectiveness in rural areas, and program integrity.

Workshop participants agreed that analysis of the characteristics of those participating in food assistance programs is important because many policies address issues related to who is being served and because participation rates provide indicators of program access and program effectiveness. The extent to which eligible citizens participate, the characteristics of those who participate (and those who do not), the distribution of benefits among them, the length of time they stay in the programs, their reasons for entering or leaving the program, and the reasons why those who are eligible forgo their benefits are all useful measures of the extent to which benefits are or are not targeted in intended ways (Carlson, 1996).

Descriptive analyses of program participants could provide much-needed insight on the segment of the population known as the "working poor." According to the Bureau of Labor Statistics, the working poor are individuals who spend at least 27 weeks per year in the labor force (working or looking for work), but whose income falls below the official poverty threshold (U.S. Department of Labor, 1997). By that definition, the Bureau of Labor Statistics estimated that 7.4 million persons were classified as working poor in 1996, and, of that group, families with children had the highest poverty rate. In 1994, one study found that more than 8 percent of all children in the United States and more than 38 percent of children in poverty lived in working-poor families (O'Hare, 1996). Furthermore, the number of children in working-poor families increased 29 percent between 1989 and 1994.

Despite their prevalence, however, the working poor and their children are a group often overlooked or forgotten because they usually don't qualify for or receive any public assistance other than food assistance. According to the Center for Budget and Policy Priorities, under PRWORA, working-poor households, including those with and those without children, would absorb $5.4 billion of the food stamp cuts, or about one out of every five benefit dollars cut, over the next six years (Center for Budget and Policy Priorities, 1996).³ Stated another way, working-poor families would see their food stamp benefits cut an average of $356 annually in 1998. By 2002, these families would lose an average of $466

per year in food stamps. Much could be learned by monitoring the effect that these cuts will have on the quality of life for working-poor program participants. Profile studies of their participation in food assistance programs could provide more information about the characteristics of the working poor.

Although many of the provisions implemented under PRWORA will affect program participants from a wide range of population groups, immigrants, in particular, may experience consequences specific to that population. As mentioned earlier, PRWORA was amended in 1997 to allow states to purchase food stamps from the federal government for use in a state-funded food assistance program for legal immigrants, who were originally barred from participating in several of the food assistance programs without citizenship. Many states did adopt plans to continue to provide food stamps to immigrants, and the FY1999 federal budget eventually restored food stamp benefits to immigrants in the United States. Nevertheless, some immigrants experienced a temporary cessation of benefits during this time, making them vulnerable to possible ill effects, which could be identified through descriptive analyses of these participants. Illegal immigrants have been affected not only by the changes implemented under PRWORA as legal immigrants, but also by other recent immigration reform laws that threaten the provision of any form of public assistance to illegal immigrants.

Analytical studies of food assistance program costs can produce useful information, including evaluation of administrative costs, the value of benefits, and school meal costs. For example, will program administrative costs increase or decrease as a result of the program changes implemented in response to PRWORA, such as food stamp cash-out, program simplification, and the increased use of electronic benefits transfer? In fact, administrative cost analyses, such as recent FNS research on the cost-effectiveness of electronic benefits transfer and on cost saving in the WIC program, have provided insight into the efficiency of these programs. Since one goal is to reduce administrative costs, efficient strategies may be needed for program monitoring, which can be expensive. Under PRWORA, reductions in overall program costs could be indicative of general program effectiveness. With the exception of the first six months of FY1996, however, which witnessed a 0.5 percent decrease in food assistance program spending from the same period a year earlier, there has not been a decrease in overall program costs since FY1982. Workshop participants agreed that continued program monitoring and evaluation are needed.

The evaluation of food assistance programs must consider their interaction with other welfare programs, such as TANF, Medicaid, and Social Security

Insurance. With a similar goal of assisting those in need, these programs rely on similar components, such as income eligibility, family composition, and, possibly, administration (Kuhn et al., 1996). Possible effects of the interaction of these programs with the food assistance programs could be observed, for example, in changes in participation rates; either an increase in participation, as benefits from the food assistance programs are sought because of decreases in other forms of assistance; or a decrease in participation, as clients lose contact with the entire social welfare system. With the flexibility granted to states in establishing their welfare programs, however, state-to-state variations in eligibility requirements and administrative components could prove to be either an opportunity or an obstacle to the evaluation of food assistance programs. For example, one of the difficulties in evaluating the effects of the Food Stamp Program has been the lack of variation in benefits. State variation, if accurately measured, may help evaluation.

Workshop participants also discussed other aspects of program interactions that could be of importance. For example, individuals may come to a human service agency to sign up for one program, such as Medicaid, and, in the process, be enrolled in another, such as food stamps, resulting in the overrepresentation in the Food Stamp Program of persons with illnesses such as diabetes and hypercholesterolemia. The research question that arises is whether the Food Stamp Program can be expected to achieve the same kinds of outcomes in its population than in a relatively more healthy, low-income population.

Because most nutrition programs serve clients who are also served by other agencies, evaluations could contribute to more effective and efficient uses of resources. Interactions among the different food assistance programs should be evaluated. For example, some groups in the low-income population, such as the rural elderly, often prefer pantries and soup kitchens to food stamps as a way to meet food needs, usually because of the convenience of access and the opportunity for social interaction with others. Thus, the Emergency Food Assistance Program should be evaluated both as a food assistance program and as a substitute for food stamps.

As states devise their plans for addressing the changes that PRWORA mandates in both welfare programs and food assistance programs, the local offices will probably be the site where these changes are most immediately felt, and this impact is worth studying. At this level, contact is made with the individual benefit recipient, thus, measurement of program effectiveness at this level will be informative. Research would require field work in the program offices in order to gather measurements of the effectiveness of the program's operation. Factors to consider include the administrative changes made, such as application processes, notification of the changes in eligibility requirements, and appropriate training of

employees; variations caused by the simplified Food Stamp Program; and the effectiveness of employment assistance operations.

Because many areas of the country contain disproportionate numbers of poor or wealthy people, program outcomes depend on community and regional effects, as well as household factors. Increases in food assistance, in the form of net inflows of federal resources, to regions with larger proportions of people in poverty can lead to regional effects larger than the direct net inflow of funds. This "multiplier" effect results from a portion of the net inflow recirculating in regional labor and product markets before diffusing throughout the national economy. This added stimulus enhances the well-being of people in the region, probably including at least some poor people. This benefit is not recognized in current evaluations. An analogous argument applies to cutbacks in food assistance. A specific example that could occur in rural areas concerns the provisions for the Child and Adult Care Food Program set by PRWORA. They state that child care centers located in areas in which at least 50 percent of the children live in households with incomes less than 185 percent of the poverty level are eligible for the full reimbursements under the program.

Research should be directed toward assessing the importance of community and regional effects in determining the impacts of food assistance on the target population and on the overall economies of poorer communities. Rural areas and central cities tend to be places with larger proportions of poor people. Lagging economies in such areas are a concern of policymakers, as reflected by their enactment of programs such as empowerment zones. Small-area estimation techniques may be relevant because of state variation in welfare programs, the lack of a standardized way to set geographical parameters on programs, and the need to estimate subcounty data for large counties containing both a central city and other, less densely settled areas. Research would be facilitated by partitioning large counties of mixed rural-urban character into their rural and urban components, as well as by developing a better definition of rural and urban than the current concepts of metropolitan and nonmetropolitan areas. Some monitoring systems may be required in areas with greater need of public assistance.

Program integrity has to do with both appropriate procedural and administrative practices and issues of fraud and misuse. The majority of errors are at the administrative or procedural level; nevertheless, fraud and misuse at all levels of the food assistance programs have always been a concern. PRWORA has several provisions to protect the integrity of the Food Stamp Program. It imposes stricter procedures for authorizing stores that can participate in the food stamp program,

strengthens penalties for fraud, and requires the implementation of electronic benefits transfer systems, all of which will make previous methods of fraud more difficult to commit. Workshop participants agreed that additional research is needed to maintain and improve program integrity, although it was acknowledged that this is a very difficult issue on which to conduct research.

Need, as the basic determinant of eligibility for participation in the food assistance programs, has to be defined in a standardized manner to ensure a fair, yet effective program. The majority of the food assistance programs use the federal poverty measurement guidelines as the standard tool to determine eligibility. For example, children are eligible to receive free meals through participation in the National School Lunch Program if they live in a household with incomes at or less than 130 percent of the federal poverty line. The current poverty measure uses a set of lines, or thresholds, to assess families' before-tax money income to determine whether they are poor. The thresholds differ by the number of adults and children in a family and, for some family types, by the age of the family head. Because the poverty measure plays such a key role in establishing who receives assistance and who does not, its accuracy is crucial. As the poverty measure is widely used in many public policy arenas, this issue reaches into many realms. A Committee on National Statistics panel, in its report, Measuring Poverty: A New Approach, recommended that the current measure be revised and proposed the factors that the new measure should contain to more accurately capture all of the people in America that are truly in need (National Research Council, 1995).

Workshop participants emphasized the need for well-designed research and program evaluation, because the results of poorly designed studies could be damaging if data are conflicting or inaccurate. In any research program, the questions to be answered will determine the appropriate method. Characteristics of food assistance programs, however, can both help and hinder the design of effective evaluation studies.

Under welfare reform, evaluation methodology is undergoing a shift from large experiments to a combination of secondary analysis and small-scale experiments. Workshop participants suggested that randomized trials could be used to evaluate narrowly defined parts of programs. For instance, smaller-scale randomized trials can benefit such evaluation as whether food assistance programs are meeting the needs of children in day care settings. It is expected that the number of children in day care will increase as more parents move into the workforce. The Food Stamp Program, however, which is a national program, is

marked by less variation than other programs, and evaluation of changes on the smaller scale may prove more difficult.

Workshop participants discussed microsimulation as a useful method for evaluation efforts. Microsimulation is a modeling technique that simulates how a welfare program would operate under proposed changes and how participants would be affected. By comparing caseload and cost totals under proposed program rules with those under existing program rules, the impact of a proposed policy change can be seen. Thus, microsimulation becomes a very useful tool as states decide how and which new programs should be implemented to best achieve the desired outcomes set by PRWORA. The Panel to Evaluate Microsimulation Models for Social Welfare Programs of the Committee on National Statistics concluded that no other type of model can match microsimulation in its potential for flexible, fine-grained analysis of proposed policy changes (National Research Council, 1991). In this technique, the welfare population is represented by a micro database of administrative records, survey records, or a combination of both, and one record represents an individual or family. Thus, a major advantage of microsimulation models for social and economic policy analysis is that they produce results that can be analyzed at the individual level. The distributional impact of a policy measure across different types of families or different geographical regions can also be assessed.

Data that are used in microsimulation models often come from national surveys. For example, Mathematica's STEWARD program uses SIPP data to simulate responses to changes in food stamp policy and state welfare/child care policy. Microsimulation can be very effective when used for purposes such as descriptive analysis of program participants and program interactions. Although often regarded as more of a policy tool than a research tool, microsimulation could answer key research questions, if the goals are well defined. The assumptions made in the models should be clearly defined, and methods to test the sensitivity of the models need to be developed and assessed.

The importance of distinguishing between long-term and short-term policy research studies was emphasized. Long-term research studies usually involve primary data collection and take more than a year to conduct. Short-term policy research projects, usually carried out either to inform an ongoing policy or to provide information needed for program operations, typically require between two days and six months to conduct. These short-term projects usually rely on existing data, and they typically have a relatively narrow focus. If done well, they can be important in helping to focus policy discussion on larger, more substantive issues.

Although the distinction between these two types of research tends to blur, it is nevertheless important. Long-term research and short-term research have quite different requirements in terms of the resources needed, the amount of monitoring necessary, and the types of funding that are most appropriate. It was noted

that it is very important to keep this distinction in mind when planning future research; one model may not fit all requirements.

Implementation research was recommended by some workshop participants as a way to learn what information and guidance is being given to program participants. The implementation perspective tries to understand how the incentives offered to those asked to implement a program will affect the final shape of the program (Brady and Snow, 1996). This approach assumes that certain program elements, such as goals linked to funding, are more likely to be implemented than those without resources. Also, the tasks that are explicitly demanded may drive out those that are not, even if this detracts from achieving the goals of the program. Techniques for collecting implementation data that provide more than anecdotal evidence are not yet fully established; however, if implementation studies are carried out, they should be linked to participant outcomes. Research on program implementation and operation can provide information on caseworker behavior and the interaction of clients and workers. Much is known about participant behavior, but researchers have generally not adequately examined the behavior of those who administer the programs.

Workshop participants identified relatively inexpensive methodologies that produce good results: descriptive studies of who is being served by a program; use of existing datasets that can ask such questions as who qualifies for WIC; the use of administrative data; and syntheses of the existing literature, such as Fraker's synthesis of the effectiveness of food assistance programs (Fraker, 1990). Workshop participants emphasized the need to examine variation in the food assistance programs at the local level, such as variation in the organization and costs of school programs and their participation rates. Additional methodologies, such as stratification and oversampling to represent subgroups without using huge samples, were also discussed.