4
Recommendations

Based on its work to date, the panel offers several recommendations directed to the U.S. Department of Health and Human Services (DHHS) regarding its immediate research agenda. These recommendations are general in nature and are aimed at ensuring that adequate data and research are available to measure the effects of recent sweeping changes in social welfare programs. Five recommendations address steps that the Office of the Assistant Secretary for Planning and Evaluation (ASPE) and the department can take to address data and research needs generally; two recommendations specifically address the ASPE grant programs for studies of families who leave the welfare system. The recommendations indicate broad areas in which the panel believes action is needed in the near term to develop new initiatives or to expand initiatives that are already under way. In its final report, the panel will consider a broader range of issues and present more detailed recommendations for data and research that can provide the basis for informed policy making in an era of continuing social welfare program change.

BROAD ROLE FOR THE DEPARTMENT

The 1996 PRWORA legislation and earlier DHHS waiver programs embodied the viewpoint that the federal government should accord substantial discretion to states in designing and administering social welfare programs. In turn, many states have chosen to give considerable discretion to their counties and cities. Undergirding this stance is the assumption that decisions about social welfare programs are best made by governments that are closer to the people



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 79
Evaluating Welfare Reform: A Framework and Review of Current Work 4 Recommendations Based on its work to date, the panel offers several recommendations directed to the U.S. Department of Health and Human Services (DHHS) regarding its immediate research agenda. These recommendations are general in nature and are aimed at ensuring that adequate data and research are available to measure the effects of recent sweeping changes in social welfare programs. Five recommendations address steps that the Office of the Assistant Secretary for Planning and Evaluation (ASPE) and the department can take to address data and research needs generally; two recommendations specifically address the ASPE grant programs for studies of families who leave the welfare system. The recommendations indicate broad areas in which the panel believes action is needed in the near term to develop new initiatives or to expand initiatives that are already under way. In its final report, the panel will consider a broader range of issues and present more detailed recommendations for data and research that can provide the basis for informed policy making in an era of continuing social welfare program change. BROAD ROLE FOR THE DEPARTMENT The 1996 PRWORA legislation and earlier DHHS waiver programs embodied the viewpoint that the federal government should accord substantial discretion to states in designing and administering social welfare programs. In turn, many states have chosen to give considerable discretion to their counties and cities. Undergirding this stance is the assumption that decisions about social welfare programs are best made by governments that are closer to the people

OCR for page 79
Evaluating Welfare Reform: A Framework and Review of Current Work being served and thereby more cognizant of local needs, resources, and concerns. It is also assumed that program devolution is more likely to lead to innovative cost-effective approaches to reducing welfare dependency than a centralized approach. (Not all social welfare programs have experienced such extensive devolution as AFDC/TANF; food stamps, for example, remains largely centralized in design and regulation.) The PRWORA legislation laid out several reporting requirements to states, including a mandate to produce quarterly reports on a sample of TANF cases, a sample of cases from separate state programs that use federal maintenance of effort funds (SSP-MOE), financial data on TANF and state programs, and an annual report on TANF and SSP-MOE program characteristics. Beyond these reports, the federal government has very little ability to mandate the collection of data or the conduct of research to track the effects of the changes in social welfare programs. However, the lack of a mandate does not mean that it is desirable for DHHS to assume a passive role. On the contrary, there are clear benefits—to the states as well as to the federal government—for the department to be proactive in facilitating the collection of needed data and research for social welfare program analysis. The federal government has an obvious interest in the availability of data and research results that can provide a nationwide picture of the effects of social welfare program changes. Congress and the administration will require such a picture to make informed decisions about the renewal of PRWORA (which expires at the end of fiscal 2002) and related legislation and to determine if modifications to PRWORA are needed. Perhaps somewhat less obviously, the states also have a strong interest in producing data and research results that can support state-level policy making not only by analysis of what is happening in a state, but also by analysis of a state's experience in comparison with other states and in comparison to national-level results. Many of the representatives from states and counties receiving ASPE leaver grants who attended the panel's November 1998 workshop expressed the view that governors and state legislators are very interested in evaluations of the policy changes and how the results of these evaluations compare with results from other areas. States have an interest in learning the most cost-effective methods for data collection and analysis of the low-income population, including cross-state (and within-state) comparisons, and in order to meet the federal data reporting requirements. Making cross-state comparisons will require not only an understanding of other states' policies and implementations, but also an understanding of the data, definitions of outcomes, and methods used to make these evaluations. The state interest came across clearly to the panel in the discussions at the November 1998 workshop, as state researchers were eager to share their experiences, listen to the experiences of others, and acquire additional data collection and evaluation skills. In Chapter 3 we specified some limitations of the current round of leavers

OCR for page 79
Evaluating Welfare Reform: A Framework and Review of Current Work studies. We noted, with the recognition that it was not the sole purpose of the studies, that there are data and definitional barriers to achieving cross-state comparability. We also noted that although the studies will be quite useful for monitoring the outcomes of those who have left welfare, studies that monitor the outcomes of those who have applied for the program but have been diverted, as well as the outcomes of those who have not applied, are also needed. We noted that outcomes of former recipients need to be stratified between long-term and short-term recipients, which requires that enough historical data on recipient histories be available to identify those types. Comparisons of leavers with stayers is another priority item. Another need with implications for data collection is to carefully account for the local socioeconomic environment during the time periods of the studies, so that the cohort comparison designs intended for program evaluations can be reliably assessed. Consequently, major efforts toward capacity-building for research and data collection, toward making research and data comparable across states, and toward examining program entry as well as exit decisions will need to be made in order to understand the effects of welfare reform. To ensure that data and research needed for evidence-based federal and state policy making are collected and conducted, there are benefits to having DHHS assume a leadership role. There are five areas where a leadership role would yield benefits: (1) identifying key policy issues that will emerge over the next 3–5 years and using those issues to guide priorities for data collection and research; (2) taking steps to ensure that federal and state-level research addresses all the key populations of interest for social welfare policy analysis; (3) fostering improved capabilities for data collection and research on social welfare programs at the federal and state levels; (4) encouraging efforts to make data and research results comparable across jurisdictions; and (5) comprehensively documenting state policies and program rules. The department already has initiatives under way in some of these areas. Coordinating these activities at the federal level will ease the burden on the states to conduct such coordination on their own and will provide a resource to the states wishing to participate in coordination activities. We discuss each of these areas for DHHS leadership in turn. Identifying Key Policy Concerns Because resources across all levels of government for data collection and research on social welfare programs are limited, it is important to set priorities so that resources are used in ways that are most likely to inform key policy questions. While individual states may have specific policy concerns that differ from each other and from federal concerns, there is likely a core set of issues that will be the focus of federal and state policy making over the next 3–5 years and on which it makes sense to concentrate scarce resources for data collection and research. An initiative by DHHS to identify key policy issues of concern to

OCR for page 79
Evaluating Welfare Reform: A Framework and Review of Current Work federal and state governments will help all governmental levels make best use of their resources to ensure that data and research are available to inform policy making. Some examples of key issues of broad concern in the near term are the effects of the time limit provisions in PRWORA; outcomes not only for families who leave welfare, but also for families who remain on the program (up to the limits) and who may represent a particularly hard-to-serve population; and outcomes for families who might have applied for assistance in the past but who are diverted or choose not to apply under the new policy. Besides these groups that are eligible for or currently receiving cash assistance, the population of low-income families as a whole may be of interest in the next few years, as their circumstances and behaviors may change and move them in and out of eligibility for cash assistance. Some specific outcomes may require a longer time frame for observation and may be of great policy interest in the next few years as well. Earnings, employment, and the self-sufficiency of program participants, which are major goals of the legislation, as well as child outcomes, are examples of such longer term outcomes. As we discuss in Chapter 3, the department has already initiated studies to aid research on these long-term outcomes. ASPE and ACF, along with other federal agencies and with private foundations, have funded studies in five welfare waiver states to help the states measure the impact of welfare reform on children. ACF is also funding the development of programs for employee retention and advancement for welfare recipients, which will have a program evaluation component as well. We encourage the department to continue to fund and expand studies for key welfare outcomes that are of policy relevance for the next few years. To determine other issues that will be critically important over the next 3–5 years, the department should make special efforts to maintain regular and formal contact with groups likely to identify such issues. For example, it could be useful for department staff to talk regularly not only with state welfare agency staff, but also with state legislators and their staffs. It would also be useful to maintain close relationships with interstate coordinating mechanisms that the states have already established (e.g., WELPAN, the Midwest Welfare Peer Assistance Network, which is a network of senior welfare officials from seven Midwest states who meet to discuss the policy and administrative issues of welfare reform). ASPE has already invited researchers and administrators from other states to join the Internet-based list-serve for the welfare leaver study grantees. Expanding this means of communication between jurisdictions will be helpful for the purposes of tracking key policy concerns. The department should also maintain close relationships for information-gathering purposes with the Congress and other relevant federal agencies, such as the Departments of Agriculture, Housing and Urban Development, and Labor. We recognize that the department already has many ties of these kinds,

OCR for page 79
Evaluating Welfare Reform: A Framework and Review of Current Work particularly with Congress and other federal agencies, with research organizations that are active in social welfare program analysis, and with states. In addition, the department has begun a series of reports to provide a national-level assessment of the use of cash assistance programs, caseload composition, and welfare dependency over time. We recommend that the department expand its ties with all levels of government to focus on obtaining early warning of key policy concerns. Such efforts can assist in a long-range planning perspective that, in turn, can guide the department's current efforts in data collection and research at the federal and state levels. (1) The panel recommends that the Department of Health and Human Services be proactive in identifying important current and emerging issues for welfare policies at both federal and state levels that, in turn, can guide priorities for investment in data and research. Defining Key Populations Attention in the media and, to some extent, among policy makers, about the effects of changes in social welfare programs has tended to focus on what happens to families who leave welfare. While leavers are an important group to track, policy decisions—for example, the decision to renew or modify PRWORA—will not be adequately informed if other key populations are not also considered. Important underlying goals in the PRWORA legislation are to strengthen low-income families and to reduce their dependence on welfare for support. To understand the extent to which PRWORA and other social welfare program changes are achieving such goals, it is important to look at the entire low-income population, including the key groups of families who are diverted from applying for cash assistance and families who are potentially eligible for benefits but do not apply. At issue is whether families have not applied or have been diverted because they have achieved self-sufficiency off welfare, and whether, more generally, they are finding adequate support through other means. Defining broader populations could also provide information on participation decisions for other social welfare programs. A confluence of factors makes it difficult in the new program environment to clearly define such key population groups as leavers, divertees, and eligible nonapplicants. These factors include: the devolution of program responsibility to state and substate jurisdictions, which means that eligibility provisions for assistance differ; the redefinition of ''assistance'' to include not only cash, but also a variety of noncash benefits and services, which makes it difficult to determine when someone has "left" the program or been diverted; similarly, the blurring of lines of responsibility among program agencies (e.g., "welfare" case-workers may now serve as brokers to provide a variety of services to clients); the

OCR for page 79
Evaluating Welfare Reform: A Framework and Review of Current Work increase in child-only cases, some of which are the result of partial family sanctions; and, finally, the striking declines in cash assistance caseloads that have occurred in many states, which mean that the population of "leavers" is not likely the same across time or across jurisdictions. The Department of Health and Human Services can make a major contribution to improving the analytical rigor and cross-area comparability of data and research on social welfare program effects by addressing the definition of the key population groups, including leavers, divertees, and eligible nonapplicants, and proposing standard definitions for use in research and data collection. The department should also take steps to ensure that grant and contract research programs adequately cover key groups. We note that the department has already taken steps in this direction. For example, through its Internet-based list-serve, ASPE and the 14 leaver studies reached a consensus in defining a welfare leaver as someone who has stopped receiving cash assistance for 2 consecutive months. The next round of ASPE grants for leaver studies will include studies of divertees as well as leavers. Also, ASPE is planning a research program to study entrance as well as exit effects. In addition to these planned studies of a broader range of leavers, ASPE and ACF have already funded a set of studies on specific groups of welfare recipients, including studies of recipients who are Native Americans, who are from rural areas, who have disabilities, who are victims of domestic violence, or who are child-only cases. (2) The panel recommends that the Department of Health and Human Services address the definition of key populations of interest for welfare policy analysis in its research agenda and take steps to ensure that its grant and contract research programs adequately cover all important population groups for welfare reform. In particular, to consider the effects of changes in welfare policies on the outcomes of the low-income population, it is important to study not only leavers, but also stayers and potential applicants who are diverted from programs or who do not apply. Capacity Building Obtaining high-quality, relevant data and analyses for measuring the effects of changes in social welfare programs is increasingly difficult in the new and changing program environment. Both federal and state governments face growing challenges to keep surveys and administrative records data systems current and to obtain data by cost-effective means. Improving capabilities for cost-efficient, policy-relevant data collection and research at both the federal and state levels should be a priority for DHHS. The department can usefully undertake a range of initiatives and activities to build capacity. Being proactive in this regard

OCR for page 79
Evaluating Welfare Reform: A Framework and Review of Current Work will enable all levels of government to make better use of scarce resources for data collection and research to better serve social welfare program policy needs. There are several examples of the activities that we encourage: DHHS should facilitate efforts by states and localities to form networks for exchanging information and technical assistance on such topics as low-cost methods for tracking former welfare program participants, definitions of key population groups, analytical methods to compensate for nonresponse bias in surveys, standard formats for administrative records tracking systems, and others. Some interstate groups of this nature already exist. Examples are WELPAN (described above), the National Institute for Child Health and Human Development's Family and Child Well-Being Research Network, and the MacArthur Network on the Family and the Economy. The department could work with and learn from these groups and use their experience to help other states form similar networks. DHHS should encourage recipients of departmental grants and other state and local jurisdictions to invest in developing the appropriate staff skills for conducting surveys and analysis. The department could bring states together with academic survey organizations to develop appropriate short courses in survey and analysis methods that specifically address data collection and analysis issues for the low-income population. For example, the Institute for Social Research at the University of Michigan, which offers summer courses in survey methods, and the University of Maryland-University of Michigan Joint Program in Survey Methodology could be training venues, as could local colleges and universities. DHHS should facilitate lower-cost survey development and more comparable data and analysis by compiling questionnaires from telephone and in-person surveys of the low-income or welfare population that have had good results and making them available not only to departmental grantees, but also to other state and local jurisdictions. For example, ASPE has provided lists of questions on a range of topics to its leaver study grantees, including tested questions on child outcomes that were developed by Child Trends, Inc., and used in various forms in other programs (see Child Trends, Inc., 1999). Making available translations of tested survey questionnaires in Spanish and other languages would also be helpful. In future grant programs, such as subsequent rounds of the ASPE-funded leaver study grants, DHHS should consider hiring one or more contractors with research and survey expertise to serve as consultants to all the state and local grantees. With regard to federal household surveys on income and welfare, ASPE has historically represented the department on interagency committees for content development and related issues. For example, ASPE has played a key role in the

OCR for page 79
Evaluating Welfare Reform: A Framework and Review of Current Work development of welfare program-related questions for the March CPS, SIPP, and SPD. ASPE should continue to play a leadership role in this regard. Such a role is critically important to ensure that national household surveys continue to provide relevant, high-quality data for social welfare program analysis. (3) The panel recommends that the Department of Health and Human Services make the improvement of capabilities for data collection and research on social welfare programs at both federal and state levels a priority. The department should include capacity-building initiatives in its grant and contract programs for welfare research and evaluation. Cross-Area Comparability Although devolution was undertaken in part to reduce the federal role in social welfare program design and administration, the fact of devolution makes the need for federal-level facilitation and coordination of data and research across the states even more important. Congress and the administration require an overall picture of the effects of program changes, and state analysts assert that they are often asked by state legislators and others about how their state compares to other jurisdictions and to national-level measures. Hence, for both state and federal purposes, there is a premium on cross-area comparability of data and research, and DHHS is best positioned to encourage and facilitate such comparability in a cost-effective manner. The steps the department should take to foster cross-area comparability, which are similar to those outlined above for capacity building, include: working with existing networks of states and localities and facilitating efforts to form new networks for exchanging information and technical assistance on such topics as comparable definitions of key population groups, standard formats for administrative records tracking systems, and others; compiling questionnaires (and translations in other languages) from surveys of the low-income or welfare population that have had good results and making them available not only to departmental grantees, but also to other state and local jurisdictions; and in subsequent rounds of grants for welfare data collection and evaluation, hiring one or more contractors with research and survey expertise to serve as consultants to all the state and local grantees. The provision in ASPE's current round of leaver study grants requiring grantees to provide public-use files of their data for research use is another mechanism for working toward greater cross-area comparability that should be

OCR for page 79
Evaluating Welfare Reform: A Framework and Review of Current Work continued. Researchers who work with such files will undoubtedly have suggestions for more comparable formats and variables that can be considered in subsequent rounds of grants. In subsequent rounds of leaver study grants, ASPE can also foster cross-area comparability of research analyses and results by such actions as: encouraging grantees to collect information on the educational level and employment history and experience of leavers and use these characteristics as stratifiers in analysis; encouraging grantees to collect information on the welfare recipiency history of leavers and to stratify them in their analyses as short-termers, cyclers, and long-termers (see Chapter 2); more generally, encouraging grantees to stratify their analyses by characteristics that will allow their caseloads and leaver populations to be compared with those of other states; and encouraging grantees in their reports, as a matter of routine good practice, to include full descriptions of the welfare programs and economic environments in effect over the life of the cohorts studied. In subsequent rounds of leaver studies, ASPE could also consider asking grantees to focus their data collection and research on key outcomes, such as employment and income. A benefit of such an approach is that it is likely more cost-effective to measure fewer outcomes well and comparably across areas than to try to cover too many outcomes in one grant program. (4) The panel recommends that the Department of Health and Human Services take the lead in working with states, localities, and research organizations to achieve cross-state and within-state comparability of data and research on welfare program effects to the greatest extent possible. Documenting State Policies A fundamental necessity of making cross-state comparisons of the effects of welfare reform is knowing the program rules in each state. Keeping up-to-date information about state and substate program rules and how the rules are implemented is also essential in order to improve the capabilities for national data sets to estimate program eligibility and to evaluate policy changes. Before PRWORA, states were required to report the rules, benefit levels, and a wide variety of information on their AFDC programs to the federal government. In addition, the federal government previously shared the costs of reporting with states. Under PRWORA, states are required to report annually on the characteristics of their TANF program rules. However, as we discuss in Chapter 2, there is less stan-

OCR for page 79
Evaluating Welfare Reform: A Framework and Review of Current Work dardization of how these characteristics are reported. Furthermore, there are few reporting requirements for separate state programs, even though understanding these rules is necessary for conducting cross-state comparisons. The department can take steps to improve the capabilities of national data sets and to help make studies comparable across states by comprehensively documenting state policies and changes to policies. The department is in the best position to do so because it has contact with each state's program administrators and because it will already be enforcing the reporting requirements under PRWORA. Although efforts to document state program rules and implementations are under way at the Urban Institute, and ACF has funded the National Governor's Association and the National Conference of State Legislatures to develop common descriptions of TANF program components, the department should be committed to ensuring that systematic and comprehensive data-gathering efforts to document state programs and changes to state programs take place within the federal government. (5) The panel recommends that the Department of Health and Human Services take an active and direct role in documenting and publishing the TANF policies enacted in every state and in every substate area where relevant. The panel also recommends that the department document and publish any changes to state and substate area policies on a regular and ongoing basis. ASPE LEAVER STUDY GRANTS The current round of ASPE leaver study grants is well under way. Most grantees are completing data collection and are beginning data analysis. Hence, some of the ideas suggested above for ways to further enhance the cross-area comparability of data items—beyond the significant efforts that ASPE has made in this regard with the current grantees through its technical workshops and list-serve—must await subsequent rounds of grants. There is still time, however, to foster greater cross-area comparability in the last stage of the current round of grants by encouraging comparability in the data analysis phase and in the documentation of methods and results. ASPE should ask grantees to put their documentation plans on the list-serve and encourage a dialogue to promote comparability for key aspects of the plans. For example, grantees should be encouraged to use standard tabulation and reporting categories for key stratifiers, such as education, employment, and welfare recipiency history reporting—to the extent that the data are available. They should also be strongly encouraged, as a matter of good practice, to include information on the welfare program provisions in effect for the cohorts studied and key features of the economic environment (e.g., state and local unemployment rates). As another matter of good practice, they should be strongly encour-

OCR for page 79
Evaluating Welfare Reform: A Framework and Review of Current Work aged to fully document their data collection and analysis methods. For surveys, ASPE should foster agreement on a standard definition of response rates and other indicators of survey quality. (6) The panel recommends that ASPE encourage the leaver study grantees to achieve the greatest possible comparability of analysis and results by asking grantees to share their tabulation, analysis, and reporting plans and by facilitating a dialogue to work toward comparability of analysis methods, reporting categories used, and documentation of methods and results. We recommended above (Recommendation 2) that the Department of Health and Human Services address the definition of key populations of interest for welfare policy analysis and take steps to ensure that grant and contract research programs adequately cover key population groups. As important steps in this direction, we noted ASPE's plans to include divertees as well as leavers in its next round of leaver studies and its plans to develop a research program to study entrance as well as exit effects. For subsequent rounds of studies of welfare leavers, we recommend that ASPE ask grantees to specify a broad definition of leavers that includes the widest possible set of families. For example, in the current round of leaver studies, most grantees are excluding child-only cases in which the children are eligible for cash assistance but the adults, for one or another reason, are not (e.g., they may be foster parents or ineligible immigrants). The exception is that some grantees are including child-only cases that result because the adults in the families are sanctioned and thereby lose benefits for themselves. While there are reasons to exclude child-only cases, we believe the arguments are stronger to include them, so that it is possible to assess the circumstances of families that are receiving reduced cash assistance as well as those that are receiving no cash assistance. If administrative records systems are redesigned to track recipient families, it may be possible to analyze child-only cases at a relatively low cost. (7) For subsequent rounds of grants for studies of welfare program leavers, ASPE should broaden the population of leavers to include the widest possible set of families.