NATIONAL ACADEMY PRESS
2101 Constitution Avenue, N.W. Washington, DC 20418
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This study was supported by Contract No. HHS-100–98–0011 between the National Academy of Sciences and the U.S. Department of Health and Human Services. Support of the work of the Committee on National Statistics is provided by a consortium of federal agencies through a grant from the National Science Foundation (Number SBR-9709489). Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.
Library of Congress Cataloging-in-Publication Data
Evaluating welfare reform in an era of transition/Panel on Data and Methods for Measuring the Effects of Changes in Social Welfare Programs ; Robert A.Moffitt and Michele Ver Ploeg, editors.
“Committee on National Statistics, Division of Behavioral and Social Sciences and Education, National Research Council.”
Includes bibliographical references and index.
ISBN 0-309-07274-3 (hardcover)
1. Public welfare—United States—Evaluation. 2. Public welfare administration—United States—Evaluation. 3. Social surveys—United States—Evaluation. I. Moffitt, Robert. II. Ver Ploeg, Michele. III. National Research Council (U.S.). Panel on Data and Methods for Measuring the Effects of Changes in Social Welfare Programs.
HV95 .E958 2001
Additional copies of this report are available from
National Academy Press,
2101 Constitution Avenue, N.W., Lockbox 285, Washington, D.C. 20055; (800) 624–6242 or (202) 334–3313 (in the Washington metropolitan area); Internet, http://www.nap.edu
Printed in the United States of America
Copyright 2001 by the National Academy of Sciences. All rights reserved.
Suggested citation: National Research Council (2001) Evaluating Welfare Reform in an Era of Transition. Panel on Data and Methods for Measuring the Effects of Changes in Social Welfare Programs, Robert A.Moffitt and Michele Ver Ploeg, Editors. Committee on National Statistics, Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.
THE NATIONAL ACADEMIES
National Academy of Sciences
National Academy of Engineering
Institute of Medicine
National Research Council
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M.Alberts is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. William A.Wulf is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I.Shine is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M.Alberts and Dr. William A.Wulf are chairman and vice chairman, respectively, of the National Research Council.
PANEL ON DATA AND METHODS FOR MEASURING THE EFFECTS OF CHANGES IN SOCIAL WELFARE PROGRAMS
ROBERT A.MOFFITT (Chair),
Department of Economics, Johns Hopkins University
RAND, Santa Monica, California
Institute for Research on Poverty, University of Wisconsin, Madison
Mathematica Policy Research, Inc., Washington, D.C.
Institute for Policy Research, Northwestern University
School of Social Work, Columbia University
Chapin Hall Center for Children, University of Chicago
Hoover Institution, Stanford University
Departments of Economics and Policy Studies, University of California, Los Angeles
Statistics, Health, and Social Policy, Research Triangle Institute, Research Triangle Park, NC
Graduate School of Education, University of Pennsylvania
Department of Family Studies, University of Maryland
California Department of Social Services, Sacramento
MICHELE VER PLOEG, Study Director
CONSTANCE F.CITRO, Senior Program Officer
JAMIE CASEY, Senior Project Assistant
COMMITTEE ON NATIONAL STATISTICS 2000–2001
JOHN E.ROLPH (Chair),
Marshall School of Business, University of Southern California
Department of Economics, Northwestern University
Department of Statistics, University of Pennsylvania
RAND, Santa Monica, California
Joint Program in Survey Methodology, University of Maryland, College Park
Department of Economics, University of Iowa
Statistics Division, United Nations
Department of Biostatistics, University of North Carolina
School of Public Health, University of Michigan
RAND, Arlington, VA
AT&T Laboratories-Research, Florham Park, NJ
Division of Statistics, University of California, Davis
Sloan School of Management, Massachusetts Institute of Technology
ANDREW A.WHITE, Director
The Panel on Data and Methods for Measuring the Effects of Changes in Social Welfare Programs wishes to thank the many people who contributed to the preparation of this report.
The study was sponsored by the Office of the Assistant Secretary for Planning and Evaluation in the U.S. Department of Health and Human Services (ASPE-DHHS) through a congressional appropriation. Throughout the study, the panel was assisted by the dedicated staff of the agency. In particular, we thank Patricia Ruggles, Susan Hauan, Julie Isaacs, and Don Oellerich for briefing the panel and providing background information. The panel also thanks Howard Rolston of the Administration for Children and Families (ACF) in DHHS and his staff for meeting with panel members to discuss the reporting requirement data for the Temporary Assistance for Needy Family (TANF) program and efforts to catalogue state welfare programs.
The panel is greatly indebted to those who presented and discussed papers at the Workshop on Data Collection for Low-Income and Welfare Populations sponsored by the panel. The papers comprehensively discussed the current state of knowledge for surveying low-income populations; preparation and use of and access to welfare program-relevant administrative data systems; and measuring important outcomes for welfare studies. The panel would also like to acknowledge Kristin Moore of Child Trends, Inc., and Janet Norwood of the Urban Institute for their remarks at the workshop. The panel is also grateful to Greg Acs and Pamela Loprest of the Urban Institute, who prepared a paper summarizing welfare leavers studies and to Leyla Mohadjer and Hussain Choudhry of Westat, who prepared a paper on adjusting for missing data on surveys. These papers will
appear in the panel’s companion volume entitled “Data Collection and Research Issues for Studies of Welfare Populations” to be published later this year.
Many other individuals participated in panel meetings or briefed the panel on data and methodological issues that provided background information for the panel’s deliberations. Kenneth Bryson, Patricia Doyle, Enrique Llamas, Stephanie Shipp, and Edward Welniak, of the Census Bureau, briefed the panel on the Bureau’s low income and welfare related data sets. The panel is also grateful to participants of the panel’s Seminar on Evaluation Methods: Burt Barnow of Johns Hopkins University, Randy Brown of Mathematica Policy Research, Gary Burtless of the Brookings Institution, Robert LaLonde of the University of Chicago, Charles Metcalf of Mathematica Policy Research, Bruce Meyer of Northwestern University, Charles Michalopoulos of the Manpower Demonstration Research Corporation, and Robert Schoeni of RAND. The panel also thanks David Stevens of the University of Baltimore for preparing and presenting a paper and a follow-up memorandum on the composition of the Maryland caseload. We thank Sheila Zedlewski and Linda Giannarelli of the Urban Institute for preparing a paper on microsimulation methods and for briefing panel members and staff on the Welfare Rules Database. The panel also thanks Gary Cyphers and Kathleen Kinsella of the American Public Human Services Association for meeting with panel members and staff to discuss state efforts to develop administrative databases for use in welfare program research. Finally, we thank the Institute for Research on Poverty and its former director, Barbara Wolfe, for providing hospitality to Michele Ver Ploeg to conduct a study of Wisconsin welfare leavers.
The panel is indebted to the efforts of the National Research Council (NRC) staff in the work of the panel and in the preparation of this report. Michele Ver Ploeg, the study director for the panel and coeditor of this report, provided essential support throughout the life of the panel and in drafting the report. She organized the panel’s discussions into coherent topics, identified issues and formulated questions and conclusions, and supervised the general activities of the panel by organizing meetings, gathering materials and interviewing outside experts, and serving as liaison to the sponsor and other outside groups. She also assisted the panel by drafting major sections of this report and by supervising the redrafting process as part of the NRC internal review process. Shelly also conducted an independent study of welfare leavers at the same time, which is included in the panel’s companion volume. Constance Citro, senior project officer, provided experienced and wise counsel throughout the life of the panel and on this report, and contributed important insights in panel discussions. Miron Straf, former Director of the Committee on National Statistics under whose auspices the panel operated, is acknowledged for his assistance in organizing the panel and working out the charge of the panel with the sponsor. The panel is very appreciative of the work of Jamie Casey, senior project assistant, for arranging the logistics of panel meetings and for her excellent work on the production of this
manuscript. We also thank Eugenia Grohman, associate director for reports of the Division of Behavioral and Social Sciences and Education, for her careful and thorough technical editing, her commentary on report drafts, and for overseeing the review process.
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making the published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.
We thank the following individuals for their participation in the review of this report: Kenneth Arrow, Department of Economics, Stanford University; Greg J.Duncan, Institute for Policy Research and School of Education and Social Policy, Northwestern University; David Illig, California Health and Human Services Agency, Sacramento; Michael Pergamit, National Opinion Research Center, Washington, DC; Frank Samaniego, Department of Statistics, University of California, Davis; Fritz Scheuren, Urban Institute, Washington, DC; and Deanna Schexnayder, Lyndon B.Johnson School of Public Affairs, University of Texas, Austin.
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations nor did they see the final draft of the report before its release. The review of this report was overseen by John Bailar, Department of Health Studies, University of Chicago (emeritus), and Robert Michael, Harris Graduate School of Public Policy, University of Chicago. Appointed by the National Research Council, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring panel and the institution.
Finally, I thank my fellow panel members for giving their time and expertise so generously towards the completion of this report. Every member of the panel brought a critical perspective and expertise to the discussions, without which the report would have been missing important elements. Their hard work and self-less dedication in the service of a public benefit was exemplary.
Robert Moffitt, Chair
Panel on Data and Methods for Measuring the Effects of Changes in Social Welfare Programs
The Statistical Power of National Data to Evaluate Welfare Reform