Research Questions and Populations of Interest
The vast number of important questions about the effects of welfare reform on different populations is enough to keep evaluators in business for many years. Because resources for evaluation are limited, however, there is a crucial need to set priorities for which questions are addressed, which population groups are examined, and which outcomes are to be measured. This chapter describes the monitoring and evaluation priorities the panel believes must be pursued in order to understand the effects of welfare reform. Special circumstances and new developments may require future additions to this list, but the needs discussed here are likely to be long-standing.
We begin by first discussing what groups in the population are of interest, emphasizing that welfare recipients are too narrow a group for focus. We also provide a brief discussion of the types of outcomes that should be studied in welfare reform projects. Next, we provide a categorization of different types of welfare reform effects and, in so doing, outline the set of important research and evaluation questions that should be addressed. Because different individuals— those in policy-making positions and those in the outside research community, those in the federal government and those in the state government, and the public—sometimes are interested in different questions because of their own responsibilities and vantage points, we cast our net widely to include questions that are of interest to these different individuals. Finally, we assess whether the existing monitoring and evaluation efforts summarized in Chapter 2 adequately cover all the important groups and outcomes of interest for welfare programs and whether these efforts adequately address the priority research questions of interest.
POPULATIONS OF INTEREST
At the broadest possible level, the group of most concern to welfare reform is the low-income population in the United States. The welfare system exists to provide assistance to families, children, and individuals in this population and to help them raise their incomes, escape poverty, and avoid the negative consequences of poverty and low incomes. Thus, the landscape of evaluation efforts in the wake of PRWORA should include assessments of the effects of welfare reform on this broadly defined population.1
Virtually all low-income groups are covered by one type of program or another historically; however, particular subgroups have been given more attention by Congress and the public. Single mothers with children have received perhaps the greatest amount of programmatic assistance, although married couples with children have also been the recipients of major new programs in recent years (the Earned Income Tax Credit [EITC], Medicaid expansions, etc.). However, single individuals without children are also the targets of some programs, such as food stamps. Children have also been the focus of much recent policy attention, as in the expansions of the Medicaid program and the creation of the State Children’s Health Insurance Program (S-CHIP) and Early Head Start programs; children have also been the focus of more long-standing programs, such as Social Security Insurance and Head Start.
Recent welfare reform was primarily directed at changing the old AFDC program into the new TANF program. Although both the new and old programs served other subpopulations, both are concerned primarily with poor single mothers and their children. Consequently, trends in the well-being of that group are clearly a primary concern. However, the spirit of welfare reform in the 1990s, as embodied in the PRWORA legislation, works toward a broader focus. As the emphasis on leaving welfare continues, and as more families attempt to gain self-sufficiency without welfare, attention tends to shift to those families off welfare who are still having difficulties and need assistance. Consequently, policy discussions are shifting towards a focus on the nonwelfare poor population (the “working poor”), as well as the welfare population, or sometimes on the population in transition between welfare and nonwelfare. Programs that provide employment supports in the form of child care, job training, and transportation, for example, tend to be aimed at the nonwelfare population rather than the welfare population. In addition, much of the welfare reform debate over PRWORA and just prior to it focused more on low rates of marriage and high rates of nonmarital
childbearing, to a degree greater than in past decades. This, in turn, has led to an increased policy interest to discourage nonmarital childbearing by providing assistance to working, married families with children, a group that has also not traditionally been a focus of attention in welfare discussions.
To date, this shift toward a broader focus is only slight, for most attention continues to be given to families who are on welfare or who have had contact with the welfare system. These families have been the major group studied in most of the welfare reform projects conducted to date.
Even within the narrower population of those currently on welfare, however, groups of special interest can still be identified. One obvious group includes those women and children classified as welfare leavers, by some definition of that term, which have been the subject of the greatest number of recent welfare studies. However, as we detailed in our interim report, other groups of interest include those still on welfare (“stayers”), as well as those who are not on welfare but who have either applied and been diverted or rejected or who might have been discouraged from applying. Because these groups are also affected by welfare reform, their well-being should also be assessed.
Welfare stayers are usually presumed to be more disadvantaged than welfare leavers because they have been unsuccessful in leaving welfare. This expectation may be incorrect, however, because welfare sanctions lead to a considerable number of welfare exits. Those sanctions tend to affect the least educated and most disadvantaged families on welfare. Consequently, welfare leavers tend to be either quite well off or quite badly off in comparison to stayers. Welfare reform also sought to raise the employment levels of those on welfare. Families who are not on welfare but who are potential applicants are normally less disadvantaged than those on welfare, but they often experience job loss or other events that may cause them to seek assistance from time to time. Like welfare leavers, they may be more likely now than previously to turn to nonwelfare sources of support or return to employment. In addition, like welfare leavers, as they explore alternative sources of support, they are likely to have very diverse experiences. These three groups—leavers, stayers, and those who have not received welfare—together make up the total population eligible for welfare. A complete picture of the effects of welfare reform on the welfare-eligible population requires a study of the outcomes of all three groups.
Finally, there are many special-need subgroups within the low-income population that require special programmatic assistance. Families in particularly poor physical or mental health; the disabled; individuals with a history of substance abuse; women and men who experience domestic violence; men with a criminal history who have difficulty reintegrating into society, families with troubled adolescents; children with special physical, cognitive, or behavioral problems; and abused and neglected children are examples of groups with such special problems. Programmatically-defined groups, such as child-only cases, two-parent family cases, and poor but ineligible immigrants, should also be of interest in
welfare reform studies. Teenage mothers, who are at risk of not obtaining enough education or other adverse outcomes, are another group that many believe should be targeted for special attention. The subgroups of interest are likely to vary across areas, too, as states and localities may have specific populations that have specific service needs. These subgroups with special needs are important populations for monitoring and evaluation as agencies that serve them need to plan and devise programs to address their needs.
Conclusion 3.1 The primary population of interest for measuring the effects of changes in social welfare programs is the low-income population. The primary group of interest to the TANF program is the population of low-income mothers and their children.
Conclusion 3.2 Within the low-income population, those groups who have been on welfare or who are eligible for welfare are of particular interest. Within the population of welfare eligibles, there are four separate subgroups, each of which is of special interest for welfare reform studies: those who leave welfare, those who stay on welfare, those who are formally diverted from welfare through diversion programs, and those who are poor but have not applied for benefits or who have applied but been rejected.
Conclusion 3.3 The specific service needs of some low-income individuals and families also define subpopulations of interest for welfare reform research. First among these are families with special circumstances or characteristics that make the transition to employment and self-sufficiency difficult. Other subgroups of the low-income population have special needs that require assistance independent of their effects on employment, including: families with poor physical or mental health, substance abuse problems, or problems of domestic violence, as well as families with troubled adolescents or children with special physical, cognitive, or behavioral problems.
OUTCOMES OF INTEREST
A comprehensive list of all the outcomes of interest in welfare evaluations would be quite long and is not appropriate for this report. However, several distinctions among outcomes are worth making because these distinctions effect the evaluation methods and data needs that we discuss in subsequent chapters. How the specific outcomes are defined and operationally measured is a very important issue, but one that is beyond the scope of the panel’s charge. Many outcomes we discuss are measures of individual and family well-being. Well-being is a broad concept with many different dimensions, including health, safety,
economic, and social. Defining well-being, both conceptually and operationally, along these multiple dimensions is not easy, and the panel does not attempt to do so. Our discussion does not prescribe exact measurement of these concepts, only that they be measured according to current scientific standards.
The most important point to note is that there are a variety of different consumers of research and evaluation studies who are interested in different types of outcomes. The public and their elected representatives are, in part, interested in the well-being of the general low-income population as measured by many traditional outcomes: income levels; poverty rates; standard of living; and level of food, housing, and clothing consumption; as well as measures of health status and educational attainment, and traditional measures of child well-being, in terms of both educational attainment and health, and cognitive and affective competence. In a somewhat different category is employment, which might be considered an outcome of interest only inasmuch as it is an indirect indication of such outcomes as income, consumption, and a higher standard of living. However, employment is also important in itself in terms of traditional American values and hence it is legitimately included as an outcome of interest.
A rather different group in society, mostly government officials but also the public as taxpayers, is concerned with program expenditures and welfare caseloads. Welfare reform discussions have made it clear that many elected officials and a large segment of the public prefer that the well-being of families and children be achieved outside public programs. This is a long-standing public preference for independence from welfare through work and earnings. While some people believe that the government should limit its assistance for welfare and other programs, others believe that government should provide assistance outside welfare through more universal programs, such as EITC, child support enforcement, and universal health insurance. Others take a position between these. For all these groups, the sizes of caseloads and expenditures are themselves outcomes of interest.
A still different group of policy makers is interested in the administrative effects of welfare reform and how the agencies that provide welfare services have reorganized to better provide services. Because PRWORA has resulted in a greatly devolved administrative structure, there is now more room for differential implementation of programs across states and localities. Reorganizations of program administrative structures have been common throughout the country. For example, to offer a wider variety of services under TANF and related programs, many local welfare offices have merged with other social and economic program offices and now use front-line caseworkers as gatekeepers to these services. The way administrative systems have been reorganized to be more synchronized with the new TANF program is, thus, another outcome of interest to state and local officials.
Family structure and family formation outcomes have become more prominent and of direct interest in policy discussions than they once were (the pre-
amble to PRWORA explicitly discusses these, unlike prior welfare legislation). The emphasis on marriage and on childbearing within marriage reflects long-standing American values. Many reforms have been implemented specifically to discourage nonmarital childbearing (e.g., family caps, sanctions for not cooperating with paternity establishment, abstinence education programs, requirements that teenage parents live with their parents, etc.). PRWORA has also given an incentive to states to focus on nonmarital childbearing outcomes by implementing an illegitimacy bonus that will be given to the five states whose out-of-wedlock birth and abortion rates decrease the most over 2-year periods. As a consequence, family structure and family formation outcomes, such as marriage and divorce, out-of-wedlock birth, paternity establishment, and teenage pregnancy also belong on the list of outcomes of interest.
Thus, there is a broad range of outcomes that should be the focus of welfare reform evaluation and monitoring. Each of these broadly defined outcomes discussed here—individual adult and family well-being outcomes, such as income, poverty, consumption, employment, education and health; traditional measures of children’s physical, cognitive, emotional and behavioral well-being; family structure and family formation outcomes, such as marriage and divorce, childbearing, teenage pregnancy and out-of-wedlock births; and outcomes of governments themselves, such as expenditures, caseloads, and administrative structures—should be studied as part of a comprehensive welfare program research agenda.
Conclusion 3.4 The set of outcomes of interest for studies of welfare reform should be defined broadly to include all the outcomes that the different audiences of studies of welfare reform—the public, Congress and state legislators, and other governmental officials and program administrators—are concerned about.
QUESTIONS OF INTEREST
The number of important questions about the effects of changes in social welfare programs on the various populations of interest is virtually limitless. Thus, priorities have to be set in order to assess the needs for services among the low-income population, to make an assessment of whether PRWORA should be reauthorized, and to assess what future programmatic reforms may be needed. This section describes the questions that the panel believes are most important and should be used to guide evaluations and data collection efforts.
As was the case in detailing the diversity of audiences interested in different outcomes, there is diversity of audiences interested in different questions. It is important to realize that there are many different questions of interest and not all audiences are interested in all questions. The panel has taken a broad approach by considering the types of major questions in which significant segments of the public and government have an interest.
We classify the questions of interest into three general areas: (1) monitoring and describing trends in the well-being of the populations and subpopulations of interest; (2) determining the types of programs that states and localities have chosen and how the programs have been implemented and administered; and (3) conducting formal evaluations to assess how reform has changed outcomes relative to what they would have been in the absence of reform, and how future reforms will change outcomes relative to what they would be without further reform.
The questions of interest identified are both retrospective and prospective in nature. Some questions of interest concern the effects of the most recent changes in welfare policy, which is necessarily a retrospective question. Assessing the overall effect of PRWORA is necessarily a retrospective question, although we argue that the answers to this question may have implications for future systemwide changes. Questions of the effects of broad components and detailed strategies are retrospective in that they assess the effects of policies already in place. However, as we argue below, what is learned from these evaluations may have implications for the kinds of policy changes that are made in the near future within PRWORA’s overall, devolved framework. It is in this sense that they are also prospective. It is difficult to predict specific questions that will be of future policy interest. For example, policy priorities may change if there is a severe economic recession, perhaps causing a change in focus to job training for welfare recipients instead of immediate employment. Instead of trying to predict specific questions of interest for the future, and the methodological and data needs for answering these questions, we instead define the questions of interest broadly and, accordingly, keep our discussion of methods and data needed to answer these questions in later chapters at a more general level as well.
Monitoring the Well-Being of the Low-Income Population
As we have already noted at several points in this report, monitoring the well-being of the low-income population is, in and of itself, an important task that is essential to a complete and satisfactory study of major welfare policy changes for several reasons.
First and foremost, the well-being of the low-income population is just one part of a much broader effort to monitor the well-being of the nation’s population as a whole over a wide range of domains (e.g., poverty, educational attainment, unemployment, and health status) and over a range of subpopulations (e.g., children, the elderly, minorities, workers). Such monitoring is crucial in order to understand trends in the population and its well-being so that both private and public entities can gear production or policies to meet the nation’s needs. Understanding the well-being of the low-income population is part of this endeavor.
Second, monitoring studies can provide valuable information to guide the design and targeting of future program changes. Many federal programs are
targeted at the low-income population, and high-quality and consistent statistics can detect needs for service or policy change, early responses to changes in these programs, and responses to changes in macroeconomic conditions that might affect participation in these programs. For example, tracking program participation rates among poor families can help detect lack of use, changes in use, or critical gaps in coverage of programs. Monitoring efforts might also reveal subpopulations within a state that are doing especially well or especially poorly in comparison to the well-being of the larger population. This knowledge could be useful in helping to identify the types of services that the subpopulation uses or is in need of, which is important for budgetary planning purposes and for developing and planning future policy changes. Monitoring the well-being of the population can also be helpful in identifying areas where further policy evaluation and general behavioral research are needed.
State legislators and agency officials also benefit from knowledge of early responses to changes in policy and macroeconomic conditions in their states. Thus, it is valuable to monitor the well-being of the low-income population within each state. This is especially true as services for the low-income populations are increasingly blended together and caseworkers increasingly act as gatekeepers to an array of services. Agencies also benefit from knowing the service needs of their clients, such as needs for child care or needs for substance abuse or domestic violence counseling.
Monitoring also is important in directing research and evaluation activity. Research should focus on improving understanding of those groups in the population who are most in distress and most in need of assistance. Identifying those groups and characterizing the nature of their needs is a key function that monitoring can serve and that is therefore a necessary first step to effective evaluation and research.
Characterizing and Tracking Policies, Programs, and Administrative Practices
The need to understand policies, programs, and administrative practices in the TANF program is as great after devolution as it was before, but the task has grown enormously in size and complexity because of the need to cover a much larger number and range of programs. Under AFDC, there was some variation in benefit levels and a few other program characteristics across the states, but, there was considerable uniformity in the basic nature of the program nationwide. Under TANF, program authority has devolved to the states, who now have the discretion to devise programs of their own choosing (subject to a few requirements in the PRWORA legislation). States have responded by developing programs that differ in myriad ways, and the reporting requirements under PRWORA do not require the states to specifically describe all the details of their program features to the federal government, thus there is no automatic mechanism for knowing many details of state programs in the new era of welfare reform.
Documenting the nature of state programs is complicated by several additional factors. The PRWORA legislation and the regulations that have been issued by DHHS give the states the ability to develop noncash programs for TANF recipients as well as cash programs (e.g., transportation assistance).2 Noncash, service programs are more difficult to track than traditional cash programs. As states move toward greater provision of noncash services of multiple types, this problem will increase. Another factor is that states have in many cases given considerable discretion to the local offices in their states to implement variations on a common model, which results in even more variation in programs. Yet another aspect of this issue is that the way programs are administered and implemented may vary in significant ways from the way they are understood to operate from a formal description of the policy, which also creates the need for detailed descriptions. Finally, all of these developments pertain only to TANF. States are free, however, to develop non-TANF programs with their own funds or from other federal funds. These programs may serve either TANF recipients or, perhaps more often, those who have left the TANF program or been diverted from TANF, but who still need assistance. It is the entire programmatic environment in a state that has changed by welfare reform, and this total environment needs to be tracked and characterized.
Knowing what states and localities are actually doing—and making collection of this information a major research question in need of directed and focused investigation—is important for a variety of reasons. First, it is of interest in and of itself, because the flowering of different program types across different states and localities was one of the intentional outcomes of changing the program structure to a block-grant system. Indeed, PRWORA was not really designed to institute a new set of policies. Rather, it was designed to allow devolution of policy-setting to the states, under the belief that states know the needs of their populations better than the federal government and could better design programs to meet those needs. Evaluation of the PRWORA legislation, in this sense, necessarily requires knowing what types of programs actually exist. During the debate over PRWORA, many people were also concerned that some states, when freed from federal requirements, would adopt policies or program features that were inconsistent with the national interest in assisting the poor. A related concern at the onset of PRWORA was that states would limit the benefits they offered to recipients in a “race to the bottom,” fearing that if a state was relatively more generous, welfare recipients would migrate to that state from states that were relatively less generous. Determining the variation in programs that has resulted from the legislation is needed to address these concerns. Thus, once again,
tracking and monitoring state policies is an intrinsic and critical outcome of welfare reform and needs to be documented. This includes tracking and documenting how different state and local program agencies have been restructured.
Second, the U.S. federal structure has often been touted for its ability to create knowledge by allowing states to undertake variation in public policies and then to learn from each other (states as “laboratories of democracy” in Justice Brandeis’s famous phrase). These spillover and learning features require that states know what other states are doing. To some degree this has already occurred. States currently do consult with one another, and there are informal networks across states that communicate ideas and strategies. Nevertheless, a reasonable nationwide policy tracking system is needed to provide states with a more complete set of information.
Third, determining what states and localities are doing is a necessary intermediate step in evaluating outcomes, which is a main goal of formal evaluation. Policy evaluation that makes use of the program variation across areas is handicapped if only broad and superficial descriptors of policies are available. Evaluations of welfare reform would not be very useful if it were not clear what welfare reform actually is and what is actually being evaluated. The proliferation of different policies engendered by PRWORA was initially seen by evaluators as a potential benefit because it was presumed that it would allow many informative comparisons across areas. This view is the evaluator’s counterpart to the laboratories-of-democracy argument used in policy circles. However, capturing that benefit of policy variation requires that the policies be documented in sufficient detail for accurate characterization when conducting a formal evaluation analysis.
Formally Evaluating the Impact of Welfare Reform
A critical set of questions concerns the formal evaluation of the effects of welfare reform, where by formal evaluation we mean a rigorous assessment of the effects of a change in policy on outcomes relative to what would happen in the absence of that change. Forming the questions of interest for evaluating welfare reform in the 1990s and after PRWORA is difficult because of the structural change created by the reform. Unlike the incremental changes in the AFDC program that resulted from policy changes in 1967, 1981, and 1988, for example (see Chapter 2) —all of which changed a broad component of the program but not its fundamental character—welfare reform in the 1990s changed the entire system. The consequences of this type of systematic reform are many and influence all the conclusions drawn in this report in subsequent chapters on evaluation methods and data collection.
For defining the sets of evaluation questions that need to be addressed, systematic reform implies that there are several different levels at which the effects of reform need to be assessed. We have classified the questions in three
general categories, which are ordered with increasing specificity, in relation to outcomes:
What is the overall impact of welfare reform, taken as a whole and with all individual components bundled together?
What are the impacts of the individual, but still broad, components of welfare reform (work requirements, sanctions, time limits, family caps, etc.) on outcomes?
What are the impacts of individual, detailed strategies within each of the broad components (type of work strategy, specific cash assistance level, nature of sanction policy, etc.)?
This three-tier classification of the questions that need to be asked is useful because each type of question requires a different evaluation strategy and somewhat different data, as discussed in the next two chapters, and because there are different audiences for each of the three questions.
The overall effect takes into account not just the immediate effects on welfare recipients, but also the multiple systemic changes that result, such as changes in the nature of the welfare system, in the expectations that the individuals and the families in the low-income population (including current nonrecipients) have of the system, in their behavior, in the way the program is organized and administered locally, and in the types of assistance provided by other agencies. The overall effect can be measured at the national level or in individual states or localities.
The overall effect of welfare reform is of interest to many groups, including members of the public and members of Congress. Answering the question requires separating the effects of the economy and of other policy developments from the effects of welfare reform and requires estimating how outcomes in the low-income population (and in specific subgroups) would have differed in the absence of welfare reform. As we discuss in the next two chapters, answering this question has proven to be very difficult with existing evaluation methods and with the available data.
One reason for the interest in the overall effect of PRWORA stems from the fact that changes relative to the AFDC program were profound. PRWORA was one of the most important pieces of legislation affecting the AFDC program since 1935, and was surrounded by a great deal of public interest and media attention. A second reason for an interest in the overall effect is that the PRWORA legislation and the waiver reforms that preceded it were based on the presumption that changing many of the individual components of the program simultaneously—to enact an entire “bundle” of individual reforms—would have a synergistic effect that would equal more than the sum of its parts. In other words, the impact of the entire bundle would be greater than the cumulative impact of individual components had they been enacted separately. Welfare reform in the 1990s was in-
tended to change the culture of the welfare system and to change the basic expectations for the role that welfare plays in the lives of poor families, both the expectations held by families and those held by caseworkers and agency personnel. Separate estimations of the effect of individual components cannot capture this synergistic effect. A third reason for interest in determining the overall effect of welfare reform is to identify the evaluation and data needs required to answer this type of question should major, systemwide welfare reform take place in the future. Lessons can be learned from the attempt to answer this question that may be usefully applied to future rounds of reform, although there is little expectation of major change in welfare policy in the near future.
The question of the overall effect is of far less interest to many policy makers, particularly those at the state and local levels, who take the new welfare structure as given and are more interested in prospective questions concerning the incremental effects of future policy changes. As noted above, many states are actively considering how to assist low-income families in greatest need, including those not on welfare, and how to do so with noncash as well as cash assistance. These policymakers do not want to return to the AFDC system as it existed in the 1980s and, consequently, are not especially interested in what outcomes would have been today under that system, because it is no longer an interesting policy option. For this audience, the effects of broad components and detailed strategies (discussed below) are of more interest because they are more relevant to what future reforms might be—that is, incremental adjustments under the basic structure as it now exists. Nevertheless, even for this audience, determining the overall effect of welfare reform could be useful if it illuminated differential effects among subgroups of the population. Such evaluations could identify the subgroups toward whom new policies could be addressed.
The second category of question that many audiences find interesting is the effect of individual, broad components of welfare reform, such as family caps, time limits, work sanctions, and special provisions applicable to teenage parents. For example, one may ask how the introduction of time limits of any particular type has affected outcomes relative to not having any time limits.
As with the question of the overall effect, questions of the effects of individual, broad components has both a retrospective and a prospective formulation. One may ask what outcomes would have been under the old AFDC system had one component of welfare reform (e.g., family caps) been enacted while all the others had not been, for example. Or, one may ask what outcomes would be like in the future if that component were eliminated or modified and all other components were retained in their present form. The answers to these two questions are likely to be quite different, because the base policy from which the individual component is added or subtracted is different, and the base policy structure is likely to influence the effect of each component taken individually. Evaluations of the effects of individual program components is more likely to be of interest to policy makers interested in incremental reform from the current structure, espe-
cially if effects are considered prospectively. However, the evaluation and data problems that arise are the same whether the question takes a retrospective or prospective form, as we discuss in the next two chapters.
Currently, there is considerable dispute about the effects of individual broad components, which confirms that these questions are indeed of interest to many. For example, there is considerable disagreement on the effects of the time-limit policies that were contained in PRWORA and in many pre-PRWORA waiver plans. Some analysts have argued that time limits have not had the substantial effect that many anticipated, or feared (especially given that time limits were probably the most controversial feature of welfare reform). The evidence from states that set short time limits does not indicate that their effects on exit rates have been very big, on average, and the effects on post-welfare employment are unclear as well (Bloom, 1999). Whether this result is because states have used extensions and exemptions to avoid the time limits, because most recipients leave long before the limits are reached, because the work requirements have more effect, or for some other reason, is not clear. It may also be the case that not enough time has passed to assess the full effects that time limits will eventually have. Nevertheless, the importance that many give to determining the incremental contribution of time limits to the effect of welfare reform illustrates the policy importance of questions surrounding the effects of broad program components.
The third category of evaluation questions concerns the effects of detailed strategies and detailed programs and approaches within the broad components. The specific requirements for work activities, the relative effects of time limits of 2 years and 5 years, the relative emphasis on immediate work and job placement in comparison with training, the way in which sanctions are applied, strategies to encourage job retention among those who have already found work, the magnitude of family cap penalties, and the specific teenage parent requirements or programs to discourage teenagers from becoming pregnant are all examples of detailed strategies whose effect on outcomes is of interest. Many very detailed strategies are also administrative in nature. One concerns case management strategies, for which the issue is how best to guide families with particular needs to the programs and services that will best address those needs. A related issue is whether to house multiple programs in a one-stop shopping facility or parcel them out to different agencies. Indeed, the list of such detailed strategies is almost limitless.
While these examples are all easily distinguished from the broad components that we just mentioned—time limits, work requirements, and sanctions, for example—the distinction can be overdrawn because there are many policies that fit in a gray area between broad components and detailed strategies. There is, in truth, a continuum of policies ranging from the broadest archetype to the narrowest and most specific. Nevertheless, we make this distinction between different types of questions because the audience differs as one moves across the spectrum, and because, as we discuss in Chapter 4, the most desirable evaluation methods
change as one moves across that spectrum. Policy makers at the state and local level and at the federal level who are most actively involved in searching for incremental improvements within the existing welfare structure constitute the primary audience for answers to questions about detailed strategies. They are of critical importance and are also large in number. Priorities in isolating the most important issues and the most important alternative policies among the many that might be imagined must be established. The overall question is “what works, to what degree, and for whom.”
Conclusion 3.5 The monitoring questions of interest are the following: How has the well-being of the low-income population and key subgroups evolved subsequent to welfare reform? Which subgroups are doing well and which are doing less well? Which subgroups are in greatest need and deserve the attention of policy makers?
Conclusion 3.6 The descriptive questions of interest regarding program policy and implementation are the following: What policies, programs, and administrative practices have states and localities actually implemented as part of welfare reform? How wide is the variation across states and even within states in policy? How has implementation differed from officially described policy? How has the non-TANF programmatic environment changed?
Conclusion 3.7 The impact evaluation questions of interest are the following: What are the overall effects of the complete bundle of changes in policies, programs, and practices on the well-being of the low-income population, including the effects on both adults and children and on specific subpopulations of interest? What are the effects of the individual broad components of welfare reform on the well-being of the low income population and subpopulations of interest? What are the effects of specific detailed strategies within each of the broad program components on the well-being of the low-income population and the subpopulations of interest—what works and for whom?
NATIONWIDE VERSUS INDIVIDUAL STATE ASSESSMENTS
A final issue in formulating the questions of interest concerns whether an overall nationwide assessment of the effect of welfare reform is needed or whether a set of state-specific results, perhaps not even for a complete set of all the states, would be sufficient. Although an overall nationwide estimate of a major piece of legislation is usually appropriate, the devolution inherent in current welfare reform and the proliferation of different types of reform programs across the states and localities makes a nationwide estimate of somewhat lesser interest than usual,
because such an estimate does not represent the effect of any one type of program and so is not very instructive for what policies work best and which do not. However, to the extent that PRWORA is not so much a law leading to specific policies but a law giving states freedom to formulate the programs they desire, the effect at the national level is of interest. Estimating the effect at the national level does answer the question of the effects of devolution in that it answers questions about the effects of the many programs that states have implemented with their new discretion over program design. A nationwide estimate of the effect of welfare reform (either overall or broad components) is therefore of interest, even if it is not particularly helpful for assessing what to do next.
Conclusion 3.8 The effect of welfare reform is a question of interest for the nation as a whole as well as for individual states.
The set of welfare reform projects under way at the present time is impressive in its scope, volume, and diversity. The number of projects is unprecedented compared to any prior era of welfare reform evaluation—such as, for example, the evaluation efforts following the landmark 1981 and 1988 legislation referred to in Chapter 1. A large number of capable researchers in the private and public sectors are devoting major efforts toward welfare reform research and have been producing a great number of valuable and informative studies. Both ASPE and the Administration for Children and Families (ACF) have substantial agendas for welfare reform research and have supported much high quality work. The role of private foundations is also notable for its scope and magnitude of funding to support high-quality welfare reform research. The knowledge and information base that has been established and continues to develop is broad and deep.
Taken as a whole, the studies have been reasonably successful in addressing the correct populations, outcomes, and questions of interest. However, we find that there are still gaps and areas of overemphasis and underemphasis. This section assesses the projects described in Chapter 2 in light of the populations and questions of interests discussed in this chapter.
Several studies and a number of government agencies have focused on monitoring the general well-being of adults and children in the low-income population over time. ASPE has sponsored and conducted many studies that extensively document trends in the low-income population, both adults and children, and in the welfare recipient population. These are perhaps the best and most comprehensive examples of monitoring studies. These studies should be continued and expanded. The other monitoring studies described in Chapter 2 are also addressing the key monitoring questions of interest for the low-income population, even if only for localized areas.
However, while most of the important monitoring questions are being addressed in one place or another, we find considerable imbalance in the amount of attention paid to different groups. As noted in Chapter 2, the current landscape of research overemphasizes welfare leaver studies and underemphasizes studies of stayers, applicants, divertees, and discouraged nonapplicants. Leaver studies serve a valuable monitoring function in documenting the outcomes of leavers, identifying what types of leavers are doing well and what type are not, and assessing the degree to which self-sufficiency has been attained after leaving the rolls. But the other groups deserve equal attention because they are also affected by welfare reform. A complete picture of the trends in well-being of the welfare-eligible population as a whole is needed for a satisfactory monitoring effort. Such a picture has not been completed to date, 4 years after the passage of PRWORA. ASPE has made a start in the right direction in its funding of state studies of divertees and applicants, but much more needs to be done.
While monitoring questions are being addressed with reasonable frequency, questions concerning the determination of what states and localities have actually been doing has until recently received much less attention and fewer resources than they deserve. As a consequence, there are large gaps in knowledge about what governments have done in the wake of PRWORA. Documentation of welfare rules at the Urban Institute, at the Center for Law and Social Policy (CLASP) and the Center for Budget and Policy Priorities (CBPP), and at other organizations are laudable, but represent only a small fraction of what needs to be done. These documentation efforts do not cover all programs and still lack much important detail about local implementation. They are also coming at a very late date after the passage of PRWORA. Studies of process and implementation at the Rockefeller Institute, the Urban Institute, at the Manpower Demonstration Research Corporation as part of the Urban Change Project, and in other evaluation projects, are valuable and will provide useful information in the future. But more process and implementation studies need to be conducted at the local level and in a more comprehensive set of areas around the country. We return to these issues in our discussion of data needs in Chapter 5.
With respect to the evaluation questions of interest, most are being addressed by one or more research studies in the set we discussed in Chapter 2. The overall effects of PRWORA and welfare reform have been addressed in the caseload and econometric modeling literature, in which the effects of individual broad components have also been studied. The ACF waiver experiments also seek to determine the impacts of entire bundles of reforms, albeit in a pre-PRWORA environment. While we have findings and recommendations in Chapter 4 on the strength of the particular evaluation methods used in these studies, and in Chapter 5 on the strength of the data bases used, the questions themselves, as we have posed them, have been addressed in at least some studies. Likewise, studies of detailed strategies have begun, particularly at ASPE and ACF, although most are in their
beginning stages and few results are yet available. These efforts should be greatly expanded and strengthened.
Although most of the evaluation questions are being addressed in one study or another in the total constellation of projects under way, future efforts should invest more in identifying emerging issues of importance, defining the key questions of interest, and in providing a framework for welfare reform studies that the evaluation community can follow. Our review of existing studies shows a large number of studies funded by different public or private organizations, addressing a wide variety of questions, but in an unfocused and uncoordinated fashion. A clear delineation of populations, outcomes, and evaluation questions of interest as we have laid them out in this chapter has not been made. The panel has concluded, therefore, that there is a need for some organization to take a leadership role in defining the questions and populations of interest for the welfare research community as a whole and in setting the agenda for what types of research is needed. Setting that agenda should involve consultation with a broad set of groups, including the states, private foundations supporting research on welfare programs, and others. Research conferences like the ones ASPE and ACF have previously held are a possible venue for such consultation. We view such an agenda-setting task as appropriately a federal responsibility because a full view of the effects of reform across the nation is needed and because no one state will be able to do this on their own. Because ASPE is the branch of DHHS with the specific mission to advise the secretary of DHHS on policy research and evaluation, economic analysis and more generally, policy development, it is the most appropriate agency to provide that leadership. We recommend therefore that ASPE take a more proactive and public role in this regard.
Recommendation 3.1 The panel recommends that ASPE take primary responsibility for publicly defining the questions of interest for welfare reform research and evaluation, identifying emerging issues for social welfare programs, and defining alternative detailed strategies and policies that address the what-works-and-for-whom questions. In doing so, ASPE should expand its current activities in seeking input from states, private foundations, and other stakeholders on emerging policy and evaluation issues.
Finally, the panel believes that defining the questions of interest is sufficiently important that they should be subject to congressional and public review. The best way for this review to take place is for ASPE to document its list of important questions in the form of an annual report to Congress. Such an annual report should also review the degree to which existing welfare reform studies, both those funded by ASPE and those funded by other government agencies and private entities, are addressing the necessary questions. The annual report should relate its own agenda of research to those questions, and should place itself in the constellation of welfare reform research and discuss how it sees its own role. In
doing so, it will need to coordinate with ACF and other programmatic branches in DHHS that administer and evaluate specific welfare programs. In subsequent chapters, we suggest other topics for inclusion in such an annual report.
Recommendation 3.2 ASPE should produce an annual report to Congress that, among other things, presents a comprehensive list of the important questions to be addressed in welfare reform research, describes how those questions are being addressed in the overall landscape of welfare reform studies, and explains how its own research agenda relates to those questions and to other studies under way.