Applicability of Food Insecurity Outcomes for Assessment of Program Performance
This chapter examines the use of the annual prevalence estimates of food insecurity in the United States for assessing the performance of the U.S. Department of Agriculture’s (USDA) food assistance programs in accordance with the Government Performance and Results Act of 1993. Since specific recommendations and suggestions discussed earlier in the report may lead to changes in the current system, the objective in this chapter is to consider this issue from a broad perspective.
FOOD SECURITY AS A MEASURE OF PROGRAM PERFORMANCE
The Government Performance and Results Act of 1993 (GPRA) requires government agencies to account for the intended results of their activities. Although it provided for strategic planning and managerial accountability, the primary goal of the law was to force a shift in government programs away from process goals (like the number of grants made) to measuring outcomes—whether the intended results of the program were achieved. Federal government agencies were required to “improve program effectiveness and public accountability by promoting a new focus on results, service quality, and customer satisfaction.” The law requires that specific performance goals be established and that annual measurement of these output goals be undertaken to determine the success or failure of the program (Government Performance and Results Act, 1993, Section 2. Findings and Purposes, no. 2).
USDA’s Food and Nutrition Service (FNS) is the agency responsible for the major food assistance programs, including the Food Stamp Program,
the Special Supplemental Nutrition Program for Women, Infants, and Children, and the National School Lunch program. The 2000–2005 strategic plan for FNS states as a goal for the agency, in delivering the food assistance programs, to reduce the prevalence of “food insecurity with hunger” among households with incomes under 130 percent of the federal poverty standard.1 Currently, FNS uses trends in the prevalence of food insecurity with hunger based on the food insecurity module included in the annual Food Security Supplement (FSS) to the Current Population Survey (CPS) as a measure of its annual performance. The panel was asked to comment on the applicability of these data for this purpose.
The mission of FNS is “to increase food security and reduce hunger in partnership with cooperating organizations by providing children and low-income people access to food, a healthful diet, and nutrition education in a manner that supports American agriculture and inspires public confidence.” Legislative language authorizing the Food Stamp Program in the 1977 Food Stamp Act explicitly mentions the alleviation of hunger as a program goal. Therefore, it would seem reasonable to use trends in the prevalence of food insecurity and hunger in low-income populations as performance indicators to assess whether FNS is fulfilling its mission.
Estimates of the prevalence of food insecurity or hunger—when properly designed and measured—can be helpful in determining whether a population is experiencing more or less food insecurity over time. In principle such estimates could also serve as an important tool for identifying trends and levels in food insecurity for specific subgroups of households, such as the elderly, and for different geographic areas, such as rural areas and specific regions or states. Such monitoring efforts are important because they can help to identify where additional assistance may be needed, or where it can be reduced.
The methodological and conceptual issues regarding the measurement of food insecurity and hunger have already been reviewed in this report. There are many actual and possible limitations of the CPS and all other national household surveys in sampling the lowest income households in American society. As explained in Chapter 1, as with most national household population surveys, the CPS routinely excludes people who are institutionalized and those homeless people who cannot be found in households or other living quarters visited during household surveys. The panel recognizes the likelihood of relatively high rates of food insecurity among homeless, and the resulting negative bias resulting from their exclusion. At the
same time, it has serious questions about the operational and methodological issues. The panel concludes that until better methods to survey the homeless are developed, continuing to limit the target population to the household population seems appropriate. The panel, however, urges USDA to undertake research as part of its long-term agenda, leading to obtaining estimates of food insecurity in periodic or a one-time survey to get a sense of the negative bias of excluding this population in the household survey.
However, even an appropriate measure of food insecurity or hunger using appropriate samples would not be a definitive performance indicator of food assistance programs because their performance is only one of many factors that result in food insecurity. Consequently, changes in food insecurity could be due to many factors other than the performance of the food safety net. For example, if food insecurity declined because the price of food declined, the decline in food insecurity may not indicate better performance of the food safety net, because these programs would not have been responsible for all or part of the change. Conversely, if the reverse were to occur—that is, if food prices were to rise steeply or household income were to fall—the result might be an increase in the number of food-insecure households. But this, too, would not be the result of a decline in the performance of the food safety net. Developments like these could result in errors in assessing the performance of such programs as food stamps that are intended to reduce food insecurity and hunger.
USDA staff and colleagues have studied issues of this kind in a number of ways. They have compared the food security data with information collected in other surveys, such as the Survey of Income and Program Participation (SIPP) and the National Health and Nutrition Examination Survey (NHANES). They are currently using the panel feature of the CPS to look at the food insecurity of households as they approach the beginning of a food stamp spell—a period of one or more months during which a household received food stamps every month. Analysis is still in progress. A recent paper by Wilde and Nord (2005) used the food security data collected in 2002 and 2003 to estimate the effect of Food Stamp Program participation on food security. They used CPS Food Security Supplement data for December 2001 and December 2002 to determine the change in food insecurity with hunger status for food stamp participants who were in the survey both in December 2001 and in December 2002. They found that only 41 percent of the food-insecure households with hunger in 2001 had become food secure in 2002. They concluded that “it appears that unobserved hardships strike from time to time, with large effects on both program participation and food security. These hardships are sufficiently severe to swamp the presumably beneficial direct effect of food stamps on food security” (Wilde and Nord, 2005, pp. 430–431).
To better attribute changes in food insecurity or hunger to the food
safety net would require that trends in these outcomes be supplemented with estimates of changes in need. FNS could annually analyze trends in the prevalence of food insecurity, taking into account economic and other changes that might affect that trend. Ideally, the food safety net would be able to quickly respond to changes in need so as to prevent an increase in food insecurity when need changes. Realistically the food safety net probably cannot fully do this. For example, food stamps cannot help people when food prices increase unless their income declines to make them eligible for food stamps.
The GPRA intended that agencies develop assessment measures that would allow them to assess the efficiency of their programs as well as their effectiveness. Trends in food insecurity or hunger tell us little about changes in the efficiency of FNS. Imagine that changes in food prices or other factors reduce the need for the food safety net, so that food insecurity declines among the poor. The trend in the prevalence of food insecurity would suggest that the food safety net was doing a better job at meeting the need for food assistance. But it is conceivable that prices could drop so much that the poor actually may need less than the safety net offered.
The GPRA emphasis on performance indicators requires agencies to define their goals in quantitative standards related to the purposes of the programs. In this respect, it is important to distinguish performance measurement from program evaluation. Definitions of program measurement and program evaluation promulgated by the U.S. General Accounting Office (1998) identify performance measurement as the ongoing monitoring and reporting of program progress toward preestablished goals. Program evaluations, by contrast, are individual systematic studies conducted periodically or on an ad hoc basis to assess how well a program is working. Program evaluation examines achievement of program objectives in the context of other factors in the program performance that may impede or contribute to its success.
The difficulty facing FNS (and other government agencies)—once having found a way to quantify a program outcome—is that they often must take account of other factors that might affect the need for the program. This can be extremely difficult. For the food safety net programs, this suggests that simple annual trends in food insecurity and hunger would have to be supplemented with estimates of changes in need as well as other possible economic changes that could affect the program participants. It may be difficult, sometimes impossible, to come up with reliable assessment indicators that fulfill the broad intent of GPRA. For example, to assess the effectiveness of the Food Stamp Program, FNS each year would have to analyze trends in the prevalence of food insecurity and hunger among the poor, taking account also of changes in income, food prices and scarcities, and other relevant matters that might affect that trend. The staff of FNS has
already done a considerable amount of micro-level research as they have worked to understand the benefits achieved by the Food Stamp Program. They should continue this research.
The panel concludes that relying on trends in prevalence estimates of food insecurity as a sole indicator of program result is inappropriate. To assess programmatic results, better understanding is needed of the transitions into and out of poverty made by low-income households and the kind of unexpected changes that frequently bring about alterations—for good or bad—in households participating in food assistance programs.