Evaluating Welfare Reform: A Framework and Review of Current Work, Interim Report (1999)

Abt Associates, Inc.

Arizona's EMPOWER Program

AZ

Longitudional survey of participants

Experimental design; random assignment

Evaluation of Alabama ASSETS program

3 counties in AL

Administrative data

Quasi-experimental design; each of 3 counties matched with another county in Alabama running standard AFDC programs

Evaluation of Delaware's A Better Chance program

DE

Administrative data; survey data; aggregate cost data

Experimental design; random assignment

Evaluation of Electronic Benefits Transfer in Maryland

MD

Transaction data

Evaluation of Indiana Manpower Placement and Comprehensive Training Program

IN

Administrative data; client survey; interviews of administrators and staff

Experimental design; random assignment

Evaluation of NY State Child Assistance Program

7 counties in NY

Administrative; survey data

Experimental design at 3 localities

Evaluation of New York Community Access Network (process and cost study too)

New York City after sample entry

Telephone surveys—3 months

Pre-, post-design

Evaluation of Ohio Transitions to Independence demonstration

15 counties in OH

Administrative data

Experimental design; random assignment

Evaluation of to Strengthen Michigan's Families Welfare Reform program

MI

Administrative data

Experimental design; random assignment

National Study of Low Income Child Care

25 communities in 5 to 10 states

American Public Human Services Association and National Conference of State Legislatures

State efforts to track and follow-up on welfare recipients

All states

APHSA and NCSL, with the National Governors' Association, are keeping track of leavers studies

Center for Law and Social Policy and Center for Budget and Policy Priorities

State Policy Documentation Project

All 50 states and DC

Monitor, document and analyze state welfare, health, and family support programs

Center for Urban Poverty and Social Change at Case Western Reserve University

Study for Federation for Community Planning

Cleveland, OH

Study to aid county Department of Human Services

Cuyahoga County, OH

Transportation barriers to welfare recipients

Urban Change project with MDRC

Cuyahoga County, OH

Neighborhood indicators study

Urban Institute's National Neighborhood Indicators project

Cleveland

Will assist other communities in developing similar studies of inner-city welfare recipients' barriers to employment

Chapin Hall Center for Children

The dynamics of AFDC, Medicaid and food stamp use

IL

State-level administrative data

Description and event-history analysis

The impact of welfare reforms on children's well-being

IL

Administrative data to track outcomes of families and children

Technical assistance to Illinois Department of Public Aid

Integrated Data Base on Children's Services

IL

Administrative data from child welfare, TANF, Medicaid, food stamps, special education, corrections, and mental health

Massachusetts Longitudinal Database for Research on Child Support Enforcement and Social Service Agencies

MA

Constructing a longitudinal database of administrative data from TANF, Medicaid, food stamps, child enforcement, wage reporting, and new hires

Also developing outcome indicators

Monitoring child and family social program utilization: Before and after welfare reform in 4 states

CA, IL, MA, NC

Individual-level administrative

Monitoring data

The State of the Child

IL

Monitoring study of child well-being

Child Trends

Measuring child outcomes under state welfare waivers (support for project from DHHS, NICHD, and other private sources)

CA, CT, FL, IL, IN, IA, MI, MN, OH, OR, VT, VA

Aggregate state-level data on child well-being indicators

Monitoring

Welfare-to-Work (JOBS) National valuation of Child Outcomes Study with MDRC (funded by DHHS)

Fulton County, GA; Grand Rapids, MI; Riverside, CA

3,000 families; 790 in Fulton County, GA; survey of mothers and children 2 years after enrollment in JOBS program; administrative records

Mothers randomly assigned to program or control group

JOBS Observational Study

Atlanta, GA

In-home observational study of 250 mothers and children

Mothers randomly assigned to program or control group

New Chance Observational Study of Teen-Mothers with MDRC

16 locations in 10 states

Observational study of 290 teen mothers and children who are on welfare

Voluntary program participation

Measurement of the impacts on children in evaluations of state welfare reforms (with DHHS funding)

CT, FL, IN, IA, MN

Technical support to states to develop measures of child outcomes

Assessing the New Federalism (part of Urban Institute's Project)

AL, CA, CO, FL, MA, MI, MN, MS, NJ, NY, TX, WA, WI

Child Trends is responsible for conceptualizing and designing ways to measure changes in child well-being as a result of policy changes

Department of Health and Human Services: Office of the Assistant Secretary for Planning and Evaluation, and Administration for Children and Families

Analyzing the employment and wage patterns of welfare recipients (see MPR)

CA, IL, OR, TX

NLSY and post-employment services demonstration

Assessing effective Welfare-to-Work strategies for domestic violence victims and survivors in the Options/Opciones Project (Taylor Institute)

North Lawndale community of Chicago

Study effective strategies of addressing needs of abused women

Assisting states to design and conduct follow-up studies of recipients who leave welfare (NGA, NCSL, and APHSA)

Conference in 1998

Report as issue brief—Tracking Welfare Reform: Designing Followup Studies of Recipients who Leave Welfare

Baseline data on Aid to Families with Dependent Children

All states

Descriptive historical tables of families using AFDC

Child Care Research Partnerships

5 research partnership sites

Field-initiated research on child care policies especially for low-income families

Child outcomes study of the National Evaluation of Welfare-to-Work Strategies (see Child Trends)

Domestic Violence and Welfare: An Early Assessment (see Urban Institute)

Implementation study of states; implementation of the TANF Family Violence Option and child support enforcement responses

Employment Retention and Advancement Project

CA, IL, MD, NJ, NC, OH, RI, SC, TN, TX, VA, WA, WI

Planning grants

Programs are required to utilize random assignment, experimental designs

Evaluation of community-based job retention programs (the Pittsburgh Foundation)

Pittsburgh, PA

Implementation study; monitor outcomes

Evaluating the feasibility of using food stamp administrative data to track welfare leavers

Examining the possibility of tracking welfare leavers using linked federal quality control and state automated data systems

Evaluation of Los Angeles Jobs-First GAIN (see MDRC)

Los Angeles, CA

Evaluation of New Jersey Substance Abuse Research

NJ

Will compare 2 models for providing services for substance abusing welfare recipients

Front-Line Management and Practice Study (see Rockefeller Center-SUNY Albany)

Part of 20-state study

In-depth observations of three local offices in four states

Implementation study

Improving States' Capabilities to Evaluate Child Care Policy Options as Components of Welfare-to-Work Strategies (Urban Institute/Mathematica Policy Research)

Development of expanded simulation model for state welfare administrators to consider interactions between child care assistance and welfare policies

Integration of Welfare and Workforce Development Systems

5-8 sites

Case studies

Jobs-Plus Community Revitalization Initiative for Public Housing Families (see MDRC)

Mandatory review and modification of child support orders in TANF cases. (see IRP)

Fiscal impact on state and federal governments of optional child support case modification of child support review under PRWORA

Measurement of impacts on children in evaluations of state welfare reforms (see Child Trends and Chapin Hall Center for Children)

Phase I: CA, CT, FL, IL, IN, IA, MI, MN, OH, OR, VT, VA

Phase I: Planning phase;

Technical assistance to states to develop data collection ability

Phase II: CT, FL, IN, IA, MN

Phase II: Large-scale data collection activities to expand states' abilities to measure and track child outcomes for impact analyses

Modified State Welfare Reform Projects -13 states funded to continue evaluations of welfare reform demonstrations in place prior to TANF

CA, IL, IA, MD, MN, NE, NH, NC, ND, OH, SC, VA

Sources of data vary at each site

Some experimental designs; site some nonexperimental designs

Multi-Site Evaluation of Welfare to Work Grants

Sites to be selected

4 study components: descriptive assessment of all grantees; net impact and cost-effectiveness analyses with process/implementation analyses; process/implementation analyses only; and study of Tribal Welfare to Work

National Evaluation of Welfare-to-Work Strategies (see MDRC)

National Study of Low Income Child Care (see Abt Associates)

National study

National Longitudinal Study of Children and Families in the Child Welfare System

Longitudinal survey data; sample representative of children and families who enter the child welfare system; over 6,000 children; 1st interview in Spring 1999; 3 annual follow-up rounds planned

Neighborhoods, Service Providers and Welfare Reform in Los Angeles County (see RAND)

Los Angeles County

Partner and Father Involvement in the Lives of Low-Income First Time Mothers-- Children's Hospital, Denver, CO

Elmira, NY; Memphis, TN; Denver, CO

Longitudinal study

Experimental design; random assignment

Policy Implications of Welfare Reform: Technical Assistance to States for Serving People with Disabilities (with Urban Institute and SSA)

All states

Review of state efforts to provide welfare services to those with disabilities; also case study series

The Role of Child Care in Low Income Families' Labor Market Participation (Urban Institute/Mathematica Policy Research)

Project will develop optional research designs to identify and address child care services needed; working paper series

The Role of Labor Market Intermediaries in Welfare-to-Work

All of U.S.

Data on intermediaries participating in welfare-to-work project

Description and implementation study

Rural Welfare to Work Strategies

IL, IA, LA, MD, MN, MS, MO, NY, VT, WA

Implementation and evaluation studies

State Welfare Waiver Demonstration Projects-- Waivers for 9 states to continue their pre-TANF demonstration projects

AZ, CT, FL, IN, IA, MN, TX, VT, WI

Sources of data vary at each site

Experimental design

Study of Nurse Home Visitation (University of Colorado Health Sciences Center)

Memphis, TN

696 women and their children; 5 years of data to be collected

Experimental design—random assignment

Supporting state efforts to link administrative data systems dor the purpose of studying the effects of welfare reform on other state and federal public assistance programs

MD, MA, Mecklenburg County, NC, SC, WI

Provides funding for sites to link administrative program data for monitoring and evaluating purposes

Typology of welfare policy decisions at the local level (contract with IRP)

Study of feasibility of collecting information on local administrative climate of county welfare offices

Understanding the Impact of TANF and Other Laws on Immigrant Families (Urban Institute)

Reviewing information on immigrants gathered in several major longitudinal data sets

Welfare leavers project (see Appendix A, Table A-1)

14 states, counties, or county groups

Linked administrative data and survey data

Monitoring and some non experimental impact studies

Welfare Policy Typology Project (contract with Urban Institute)

All 50 states

Establish groundwork for data base of state welfare policy information

Welfare Reform and Its Impact on Persons with Disabilities (part of Three-City Study)

Boston, MA; Chicago, IL; San Antonio, TX

Will be initial part of 5-year longitudinal study

Welfare to Work: Monitoring the Impact of Welfare Reform on American Indian Families (Washington University School of Social Work)

American Indians in Arizona

Descriptive and impact analysis

Welfare Reform Studies and Analyses (rural TANF)—Eastern Washington University

3 rural counties in Washington State, WI

Process evaluation

Wisconsin Data Project on Former AFDC Recipients (see Institute for Research on Poverty)

Fragile Families and Child Well-Being Study (Columbia University and Princeton University

20 cities randomly selected from all cities over 200,000

Birth cohort sample of 4,000 kids born to unwed parents and 1,000 kids born to wed parents. Longitudinal survey of parents annually for 4 years after birth of child

Birth cohort design

Institute for Research on Poverty, University of Wisconsino

Development and use of merged administrative data

Addresses issues related to use and organization of administrative data

Evaluation of Wisconsin Works Child Support Demonstration

WI

Data from 1997 of families on AFDC—allowed to retain all of child support payments made

Administrative data, policy documents, formal surveys of program staff and participants, and field research

Examining the labor market impacts of W-2

Wisconsin

State administrative data

How Teen Mothers are Faring under Welfare Reform

One Michigan county

Paired interviews of teen mothers with their mothers with whom they are required to live if given benefits; longitudinal study

The Impact of Welfare Reforms on Families

Dane County, WI

Baseline and 1-year follow-up interviews of 200 participants

Compare those who left welfare to those who stayed on welfare

Immigrant Health and Welfare Reform

Hmong and Mexican immigrants in WI and nonimmigrants in same area

Child and maternal health information; also state policy data

Job Holding and Earnings Dynamics for Low-Wage Workers

Milwaukee, WI

Part of New Hope Demonstration project; (see MDRC) pay stub data, household survey of participants. Experimental design

Monitoring State Efforts to Reduce Out-of-Wedlock Childbearing: The Impact of Welfare Reform

WI; other states

State program and expenditures data; in WI—county data; federal expenditure and nativity data also

Monitoring study of state efforts to reducing out-of-wedlock births; integrated administrative data

The Welfarization of Family Law

U.S.

Examines how welfare law has influenced family law

What Happens to Families Who Leave AFDC

WI

8,000 families on welfare in Who Leave AFDC? July 1995; linked administrative data

What Happens to Families Under W-2

Milwaukee, WI

Interview of families who apply or are diverted from program; interviewed twice: first as they first make contact with welfare agency and then 12 months later

Compare those who left welfare to those who stayed on welfare; of those who left, compare those who return to those who did not return

Wisconsin Works: A Case Study in Evaluating Comprehensive Welfare Reform

Wisconsin

Examines methods for evaluating state-based welfare reform program

Conferences:

Process Evaluation: Workshop, Papers, Conference

Efforts to improve quality and utility of process evaluations

Conference on Rethinking Evaluation Strategies under TANF

Examine role of federal government in public assistance policy

Foundations of Anti-Poverty Policies

Examine public support of antipoverty policies

Joint Center for Poverty Research, Northwestern, and University of Chicago

Administers small grants program funded by ERS/USDA, ASPE/DHHS, Census Bureau

Supports research on food assistance, domestic poverty and policy using the SIPP data set

Advisory Panel on Research Uses of Administrative Data

Working Papers Series on various poverty-related topics

Lewin Group

Analysis of the determinants of AFDC caseload growth

All 50 states and DC

Quarterly state-level data from 1979–1994

Models effects of changes in demographics, the economy, and programs to changes in the caseload, participants, and expenditures per case

Disaggregating the TANF child-only cases in three states

3 states

Administrative records and case file records

Describes composition and trends in child-only TANF cases

Employment Retention and Evaluation Development Project (also with Johns Hopkins)

13 states

Help states develop program interventions and prepare sites for a possible multisite evaluation

Evaluability assessment of Responsible Fatherhood programs

Develop evaluation designs for impact analysis

Evaluation of Maryland's Primary Prevention Initiative (with Johns Hopkins)

MD

Cost-benefit analysis of health components of intervention

Policy evaluation of the overall effects of welfare reform and SSA programs

Literature review and analysis design; analysis of pre-reform data; review of state-level reform assessments; site visits to 5 states; development of evaluation design

Success in the New Welfare Environment (with ICF Kaiser Consulting Group)

Review of locally implemented HUD employment and training programs and their linkage to other employment, training, and human services programs

Temporary Assistance for Low-Wage Workers: Evolving Relationships among Work, Welfare, and Unemployment Insurance (for the NGA)

Analyzed roles of TANF and state UI programs, including a look at participation patterns across UI and AFDC

Manpower Demonstration Research Corporation

California GAIN Analysis

6 Counties in CA

33,000 applicants of AFDC when GAIN was mandatory; data from automated employment earnings, welfare records, registrant survey, county level administrative data

Experimental design; randomly assigned to GAIN or not

Canada's Earnings Supplement Project

9 sites in Canada

Administrative data supplemented with ''mini-survey'' of sample of participants

Experimental design; random assignment to program

Canada's Self-Sufficiency Project

British Columbia and New Brunswick, Canada

Single parents on welfare 11/92-3/95

Experimental design; random assignment to program

Connections to Work Project

Various sites

Case studies of communities that are developing innovative approaches to connecting welfare recipients with jobs

Connecticut's Jobs First Program

a) Evaluation and Post-Time-Limit Study

New Haven and Manchester, CT

Experimental design; random assignment to program

b) Tracking study

6 sites in CT

3-month and 6-month follow-up surveys after time limit on TANF expired

Monitoring study

Cross-state study of time-limited welfare

CT, FL, VT

Focus groups and in-person interviews of 100 current welfare recipients

Implementation component and monitoring component

Florida's Family Transition Program

2 counties in FL

2,800 computerized records of monthly AFDC/TANF, food stamps and quarterly earnings; follow-up survey to 600 recipients 2 years later

Randomly assigned to AFDC or FTP; longitudinal for part of sample

Jobs-Plus Revitalization Initiative for Public Housing Families—Employment Demonstration Program

Public housing developments in 7 cities

Individual household study plus public housing project study

Los Angeles Jobs First GAIN Program

Los Angeles, CA

Administrative data for almost 21,000 households

Experimental design with random assignment to program

Minnesota Family Investment Program

MN—3 urban and 4 rural counties.

1994–1996 baseline data, administrative data, 12 -and 36-month client surveys, staff attitude surveys, and field research

14,639 families randomly assigned to one of 4 research groups

National Evaluation of Welfare-to-Work Strategies

7 sites nationwide

55,000 sample members; in some sites telephone and in-person interviews

Random assignment; process, implementation, and impact study components

New Chance Demonstration and Observational Study

16 locations in 10 states

Young moms in 10 States (n = 2,322); observational study has videotapes of 290 mother/child combinations

Individual data and the coding of mother/child interactions

New Hope Project

Milwaukee

Linked administrative data; 2-year follow-up survey of all applicants

Experimental design; applicants randomly assigned to program

Ohio's Learning, Earning and Parenting Program

OH—12 counties

Teen mothers on welfare: survey of 1, 118 teens 1 year after randomly assigned to program; review 263 participant cases; survey of 913 teens 3 years after randomly assigned

Case reviews, survey, random assignment

Oregon's Evaluation of Welfare-to-Work program

OR

Administrative data on 5,547 single-parent AFDC applicants and recipients aged 21+ who attended orientation between 10/93 and 10/94

Experimental design

Parent's Fair Share

7 counties across the country

Administrative data and survey data

Experimental design; randomly assigned to program

Project on Devolution and Urban Change

L.A., Miami, Cleveland, Philadelphia

Survey of households—two cohorts of AFDC/TANF single mothers.; in-depth interviews; Welfare office site visits; Interviews with community institutions and service agencies, neighborhood-level indicators

5 components: individual impact study, implementation study, neighborhood indicators study, ethnographic study, institutional study

ReWORKing Welfare

Guidance to planning and implementing welfare reform

Vermont's Welfare Restructuring Project

6 districts in VT

Winning New Jobs

3 California sites

Mathematica Policy Research, Inc.

Addressing Barriers to Employment for Welfare Recipients

Early Head Start Research and Evaluation Project

17 diverse communities

3 rounds of site visits, program documents, parent services follow-up interviews, child care observations, staff surveys, parent report, direct assessments of children, observations of children, videotape coding of parent-child interactions, interviews of parents

Experimental design: implementation study, impact evaluation, local research studies, policy studies, guide to EHS programs

Evaluation of Iowa's Family Investment Program and Limited Benefit Plan

Iowa

Surveyed 137 families; case records of over 4,000 families

Experimental design; process study; impact study; cost-benefit study; client focus-group discussions

Evaluating Welfare Reform: New Freedom, New Challenges for States

CA, CO, MI, MN, WI

Advice to states in evaluating welfare reform

Expanding Health Insurance Coverage for Low Income People: Experiments in 5 States with Urban Institute

HI, MD, OK, RI, TN

Current Population Survey; interviews with local and state officials, managed care reps, health care providers, consumers; focus groups with consumers and providers

Implementation study

National Evaluation of Welfare to Work Grants Program:

State and local sites

(a) Descriptive assessment of program designs

All sites

Surveys to grantees; about 35 site visits

(b) Impact and cost-effectiveness

8–10 sites that agree to participate

Experimental design; random assignment

(c) Process and implementation study

12–15 sites

Site visits, discussion with staff, focus groups with participants, program observations

Post-Employment Services Demonstration

Chicago, Portland, Riverside, San Antonio

Survey of about 300 in each site; participants administrative data for full sample of participants

Experimental design: random assignment

Teenage Parent Demonstration

Camden, Newark, and South Chicago

Site observations, interviews and case reviews with program staff, program data, state records data, baseline and follow-up interviews with teens, focus groups, in-depth semi-structured interviews

Experimental design; n= 6,000; late 1987–1991

Welfare Reform: New Requirements for Teen Parents

Welfare waiver states

Youth Fair Chance Program

17 sites in high-poverty areas

Telephone survey of youth in target areas

Michigan Women's Employment Survey

An urban county in MI

Simple random sample of 753 single mothers with children who received cash assistance in Feb. 1997; face-to-face interviews; in total, 3 waves of data to be collected

National Center for Children in Poverty—Columbia University

Research Forum on Children, Families and the New Federalism

Three purposes: promote monitoring and evaluation research; promote collaboration among key stakeholders; information exchange that includes a clearinghouse for welfare research projects

National Governors' Association

Tracking Welfare Reform: Designing Follow-up Studies of Recipients Who Leave Welfare

All states

NGA, with National Conference of State Legislatures and American of Public Human Service Association, is keeping track of leavers studies

Summaries of Selected Elements of State Programs for Temporary Assistance for Needy Families (TANF)

All states

Keeping track of state welfare policies

Working Out of Poverty: Employment Retention and Career Advancement for Welfare Recipients

Explores promising welfare-to-work programs and practices of states and localities

Summary of state welfare-to-work plans

All States

Summarizes state plans

RAND

Los Angeles Survey of Families and Communities

65 neighborhoods (census tracts) in L.A. County

Stratified random sample of neighborhoods with oversample of poor neighborhoods and of households with children under age 18; 4-year panel study

Annual household survey and annual neighborhood survey: household survey contains program participation questions and questions on child outcomes; neighborhood survey collects administrative data and interviews key neighborhood informants

Statewide CalWORKS evaluation — 5 components to study: State-level process study; county-level process study; implementation study; statewide impact and cost-benefit study; county-level impact and cost-benefit study

58 California counties

For the corresponding 5 components: (1, 2, and 3) all-county implementation survey with county and state site visits; (4, 5) administrative data from the state and counties, household survey in 6 focus counties, first in 9/99 and again a year later, plus data from components 1–3 of study.

Observational study—standard regression approach and case-control design; process and implementation analyses

Rockefeller Institute of Government Federalism Research Group

State Capacity Study & Implementing PRWORA (SUNY Albany)

20 sample states

Implementation and process study of institutions administering social welfare programs

Field research evaluation

University of California Data Archive and Technical Assistance (UC-Data)

CA Work Pays Demonstration Project (with California Department of Social Services Research)

California

State-level administrative records for AFDC, Medical, UI, other state and federal assistance programs, and employment tax files; county-level administrative records for AFDC and food stamp programs; nonautomated client records at county

welfare offices; and telephone interviews with AFDC recipients

Advisory Panel for Research Uses of Administrative Data—with JCPR

Published in January 1998

CAL—Learn Program Evaluation

CA—4 counties

3 cohorts of AFDC teens n = 4,900

Random assignment; 2-way factorial design of the 2 program elements.

Inventory of state effort to use administrative data for welfare research (funded by ASPE)

28 states

Interviews with state-level data system managers, administrators, and researchers to determine how each state use administrative data records for monitoring evaluation and research

University of Maryland, School of Social Work, Welfare and Child Support Research and Training Unit

Life After Welfare Study

Maryland

Matched administrative records; 5% random sample of closed cases taken every month for a year (2,156 cases); survey of former recipients

Examining Customer Pathways and Assessment Practices

Maryland

In-person staff interviews, site visits, case record review, observation of worker-customer interactions

Process analysis of Maryland's welfare program, which is state supervised, locally administered

Urban Institute: Assessing the New Federalism

State indicators

All 50 states

Compiled data on income security, social services, health, child and youth well-being, taxes, etc.

Aggregate state-level data

State case studies of policy and programs

AL, CA, CO, FL, MA, MI, MN, MS, NJ, NY, TX, WA, WI

Collecting data in base year (1996) and again in following years

Development and implementation of policies

National Survey of America's Families

AL, CA, CO, FL, MA, MI, MN, MS, NJ, NY, TX, WA, WI

Survey of over 40,000 households in 1997; follow-up survey of second cross-section in 1999

Well-being changes of two cross-sections to be analyzed

Child Well-Being, with Child Trends (see above)

All 50 states

Tracks legislation at federal and state level. Child Trends is developing ways to measure changes in child well-being

Understanding the Impact of TANF and Other Laws on Immigrant Families

Los Angeles and New York City

Large-scale study of immigrants and communities

Welfare Children and Families: A Three-City Study

Various researchers across country

Boston, Chicago, and San Antonio

Longitudinal data on 2,800 households; developmental study of 800 children in 2,800 households; comparative ethnographies of 170 families

Cohort design