ADVANCING THE POWER OF
Economic Evidence
to Inform Investments in
Children, Youth, and Families
Committee on the Use of Economic Evidence to
Inform Investments in Children, Youth, and Families
Eugene Steuerle and Leigh Miles Jackson, Editors
Board on Children, Youth, and Families
Division of Behavioral and Social Sciences and Education
THE NATIONAL ACADEMIES PRESS
Washington, DC
www.nap.edu
THE NATIONAL ACADEMIES PRESS500 Fifth Street, NWWashington, DC 20001
This activity was supported by Contract No. 10002411 from the Jacobs Foundation, Contract No. 10002006 from the MacArthur Foundation, and Contract No. 10002289 from the Robert Wood Johnson Foundation. Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.
International Standard Book Number-13: 978-0-309-44059-2
International Standard Book Number-10: 0-309-44059-9
Digital Object Identifier: 10.17226/23481
Additional copies of this report are available for sale from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2016 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Cover credit: Jay Christian Design, LLC.
Suggested citation: National Academies of Sciences, Engineering, and Medicine. (2016). Advancing the Power of Economic Evidence to Inform Investments in Children, Youth, and Families. Washington, DC: The National Academies Press. doi: 10.17226/23481.
The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Ralph J. Cicerone is president.
The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. C. D. Mote, Jr., is president.
The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.
The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.
Learn more about the National Academies of Sciences, Engineering, and Medicine at www.national-academies.org.
This page intentionally left blank.
COMMITTEE ON THE USE OF ECONOMIC EVIDENCE TO INFORM INVESTMENTS IN CHILDREN, YOUTH, AND FAMILIES
EUGENE STEUERLE (Chair), Urban Institute, Washington, DC
RICARDO BASURTO-DAVILA, Office of Health Assessment and Epidemiology, Los Angeles County Department of Public Health, CA
JENNIFER BROOKS, Early Learning, U.S. Program, Bill & Melinda Gates Foundation, Seattle, WA
JEANNE BROOKS-GUNN, Teachers College and the College of Physicians and Surgeons, Columbia University, New York City, NY
BARBARA CHOW, Education Program, William and Flora Hewlett Foundation, Menlo Park, CA
PHAEDRA CORSO, Department of Health Policy and Management, University of Georgia, Athens
DANIEL MAX CROWLEY, College of Health and Human Development, Pennsylvania State University, University Park
JODY L. FITZPATRICK, School of Public Affairs (retired), University of Colorado, Denver
LYNN A. KAROLY, Pardee RAND Graduate School, RAND Corporation, Philadelphia, PA
MARGARET KUKLINSKI, Social Development Research Group, School of Social Work, University of Washington, Seattle
RACHEL NUGENT, Chronic Noncommunicable Diseases Global Initiative, RTI International, Seattle, WA
OLGA ACOSTA PRICE, Center for Health and Health Care in Schools, George Washington University, Washington, DC
TED MILLER, Public Services Research Institute, Pacific Institute for Research and Evaluation, Calverton, MD
ANNE SHERIDAN, Sheridan & Associates, Potomac, MD
LEIGH MILES JACKSON, Study Director
BRIDGET KELLY, Senior Program Officer
TARA MAINERO, Associate Program Officer
NOAM KEREN, Research Associate
STACEY SMIT, Senior Program Assistant
PAMELLA ATAYI, Administrative Assistant
ALIA SANI, Intern
BOARD ON CHILDREN, YOUTH, AND FAMILIES
ANGELA DIAZ (Chair), Departments of Pediatrics and Preventive Medicine, Icahn School of Medicine at Mount Sinai
SHARI BARKIN, Monroe Carell Jr. Children’s Hospital, Vanderbilt University
THOMAS F. BOAT, College of Medicine, University of Cincinnati
W. THOMAS BOYCE, Faculty of Medicine, University of British Columbia
DAVID A. BRENT, Western Psychiatric Institute and University of Pittsburgh School of Medicine
DAVID V.B. BRITT, Sesame Workshop (retired)
DEBBIE I. CHANG, Nemours Health and Prevention Services
PATRICK H. DELEON, F. Edward Hebert School of Medicine and the Graduate School of Nursing Uniformed Services University of the Health Sciences
ELENA FUENTES-AFFLICK, University of California, San Francisco, and San Francisco General Hospital
EUGENE E. GARCIA, Mary Lou Fulton Teachers College, Arizona State University
J. DAVID HAWKINS, School of Social Work, University of Washington
JEFFREY W. HUTCHINSON, Uniformed Services University of the Health Sciences
JACQUELINE JONES, Foundation for Child Development
ANN S. MASTEN, Institute of Child Development, University of Minnesota
VELMA McBRIDE MURRY, Peabody College, Vanderbilt University
BRUCE S. McEWEN, The Rockefeller University
MARTIN J. SEPULVEDA, IBM Corporation
TAHA E. TAHA, Johns Hopkins University, Bloomberg School of Public Health
NATACHA BLAIN, Director (beginning December 2015)
KIMBER BOGARD, Director (through July 2015)
BRIDGET KELLY, Acting Director (July-December 2015)
Preface
Almost nothing drives the development of society more than investments in the nation’s children. Accordingly, public and private policy makers, funders, and others have in recent years called for and sponsored the production and use of economic evidence to inform decision making on how to make such investments. The rationale for these efforts appears straightforward: better evidence should enable higher returns from such investments. Yet to date, the use of such evidence has been limited. Why? Many reasons might be ventured: politics, special interests, power of the status quo, the limits on which evidence can be quantified, and the relative adolescence of the field. Some of these reasons can be interesting from an historical viewpoint, but the more compelling question for future investments is how to improve the development of economic evidence so it can better inform those investments.
Two answers to this latter question stand out and serve as the two principles around which this report is organized: quality counts and context matters. The better the quality of the research, the better it is received, and the more likely it is to generate demand for future economic evidence even on unrelated investments. At the same time, if high-quality evidence is to be used well, it must be suited to the context in which decisions are made. It must be timely and relevant to the decisions at hand and account for many other needs of the consumer. It was in teasing out the many ramifications of these two principles that the committee convened to conduct this study responded to a 2014 charge from its sponsors—the Jacobs Foundation, the MacArthur Foundation, and the Robert Wood Johnson Foundation—to
study how to improve the use of economic evidence to inform investments in children, youth, and families.
The committee focused its attention on economic evaluation, a type of economic analysis that is commonly performed to provide economic information related to investments in children, youth, and families. Economic evaluation encompasses cost analysis, cost-effectiveness analysis, benefit-cost analysis, and related methods used in an effort to quantify program costs and outcomes and potentially to make comparisons among programs. These methods are commonly employed in randomized controlled trials, but by no means does the committee discount the value of other approaches—ranging from theory to qualitative analysis to other forms of statistical analysis—and indeed, it encourages researchers reporting on economic evaluation to acknowledge what might be learned through such other means.
Perhaps not surprising, the committee identified many instances in which the quality of economic evidence was low, such as failure to account for many types of costs, or was reported in ways that could mislead by failing to acknowledge limitations of the analysis, often forced by restricted budgets. Accordingly, a major goal of this study was to recommend a number of ways in which current practices in the production of economic evidence could be improved. Likewise, this report suggests that producers and consumers of economic evidence can gain by giving considerable attention before, during, and after economic evaluations are performed to the context or broader system within which investment decisions are made. Setting and organizational capacity matter—as do politics and values, culture and management practices, and budget. This report includes a roadmap outlining a multipronged strategy for fostering multi-stakeholder partnerships to address these issues and for improving incentives for the use of economic evidence for various stakeholders, ranging from publishers of economic research results to program evaluators.
Needless to say, the topic of this study is of such breadth that the committee makes no pretense of having covered every angle. In some cases, moreover, it was necessary to apply lessons from related literatures because the literature on the actual use of economic evaluations was scant.
The committee members brought to this study a wide range of experience and expertise, as well as common sense. Their energy was unbounded; their enthusiasm strong; and their dedication to the public good through solid, professional, and unbiased research paramount. This report was truly a collaborative effort, with multiple authors and mutual editors and wide acceptance of critiques. It was my pleasure to serve with this esteemed group.
The committee’s talents would have been sorely tried without the superlative efforts of the study staff, led by Leigh Miles Jackson, study director;
Tara Mainero, associate program officer; Noam Keren, research associate; and Stacey Smit, senior program assistant. Wonderful guidance and encouragement also were provided by Natacha Blain, current director of the Board on Children, Youth, and Families; Bridget Kelly, former acting director; and earlier, Kimber Bogard, then serving as director. The report also benefited greatly from the efforts of our editor, Rona Briere. They kept us on track, organized our disparate thoughts, and made extraordinary organizational and other tasks look ordinary. The committee extends its profound thanks and indebtedness to them.
Of course, this study is not about us; it is about the children, youth, and families whose lives are touched, often in crucial and profound ways, by the investment decisions that were this study’s focus. It is our hope that we have advanced their well-being by describing ways to inform these decisions through better use of economic evidence. If producers and consumers of this evidence devote greater attention to its quality and the context in which it is used, we believe we will have succeeded in that task.
Eugene Steuerle, Chair
Committee on the Use of Economic
Evidence to Inform Investments in
Children, Youth, and Families
This page intentionally left blank.
Acknowledgments
This report reflects contributions from numerous individuals and groups. The committee takes this opportunity to recognize those who so generously gave their time and expertise to inform its deliberations. To begin, the committee would like to thank the sponsors of this study. Support for the committee’s work was provided by the Jacobs Foundation, the MacArthur Foundation, and the Robert Wood Johnson Foundation. We wish to thank Valerie Chang, Kerry Anne McGeary, and Simon Sommer for their guidance and support.
The committee greatly benefited from the opportunity for discussion with individuals who made presentations at and attended its workshops and meetings (see Appendix A). The committee is thankful for the many contributions of these individuals.
The committee could not have done its work without the support and guidance provided by the National Academies of Sciences, Engineering, and Medicine project staff: Leigh Miles Jackson, Tara Mainero, Noam Keren, and Stacey Smit. The committee is also grateful to Lisa Alston, Pamella Atayi, and Faye Hillman for their administrative and financial assistance on this project, and gratefully acknowledges Kimber Bogard, Bridget Kelly, and Natacha Blain of the Board on Children, Youth, and Families for the guidance they provided throughout this important study.
Many other staff within the Academies provided support to this project in various ways. The committee would like to thank the executive office reports staff of the Division of Behavioral and Social Sciences and Education (DBASSE), especially Kirsten Sampson-Snyder, who managed the report review process. Thanks are due as well to the staff in the DBASSE Office
of Communication and Reports (Patricia L. Morison, Douglas Sprunger, Eugenia Grohman, Viola Horek, and Yvonne Wise), Janice Mehler of the Report Review Committee, the Academies Research Center staff (Victoria Harriston, Daniel Bearss, Rebecca Morgan, and Ellen Kimmel), and the National Academies Press staff.
We thank Richard Cookson, Donald P. Moynihan, Spyros Konstantopoulos, and Jeffrey Valentine for their valuable commissioned work. We are grateful to Lauren Tobias and Steve Olson for their work as communications consultants for this study, as well as to Jay Christian, Francesca Moghari, and Michael Dudzik for their creative efforts in our graphic design projects. We also wish to thank Justin Ingels, Nathaniel Taylor, Rebecca Walcott, Laura Wiese, and the project’s intern, Alia Sani, for the superb research assistance they provided. Finally, Rona Briere and Alisa Decatur are to be credited for their superb editorial assistance in preparing this report.
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the Report Review Committee of the Academies. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: Richard P. Barke, School of Public Policy, Georgia Institute of Technology; Jere R. Behrman, Department of Economics, University of Pennsylvania; Janet Currie, Department of Economics and Center for Health and Wellbeing, Princeton University; Paula M. Lantz, Research and Policy Engagement, Gerald R. Ford School of Public Policy, University of Michigan; Henry M. Levin, Economics and Education, Teachers College, Columbia University and Education and Economics, (emeritus), Stanford University; Rebecca A. Maynard, Education and Social Policy, University of Pennsylvania; Lawrence A. Palinkas, Department of Child, Youth and Families and Behavior, Health and Society Research Cluster, School of Social Work, University of Southern California; Dan T. Rosenbaum, Economic Policy Division, Office of Management and Budget; Charles Sallee, New Mexico Legislative Finance Committee, Santa Fe.
Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the report’s conclusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by Robert A. Moffitt, Department of Economics, Johns Hopkins University, and Greg J. Duncan, School of Education, University of California, Irvine. Appointed
by the Academies, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.
This page intentionally left blank.
Contents
Study Scope and Key Definitions
Methods for Economic Evaluation
Stakeholders of the Production and Use of Economic Evidence
Current Uses of Economic Evaluation to Inform Investments in Children, Youth, and Families
Challenges in the Use of Economic Evaluation to Inform Investments in Children, Youth, and Families
3 PRODUCING HIGH-QUALITY ECONOMIC EVIDENCE TO INFORM INVESTMENTS IN CHILDREN, YOUTH, AND FAMILIES
Determining Whether an Intervention Is Ready for Economic Evaluation
Defining the Scope of the Economic Evaluation
Determining Intervention Impacts
Getting to Results: The Development and Reporting of Summary Measures
Handling Uncertainty in Economic Evaluation
Addressing Equity Considerations
Recommendations for Best Practices for Producing and Reporting High-Quality Economic Evidence
Alignment of Evidence with the Decision Context
Other Factors in the Use of Evidence
Factors That Can Facilitate the Use of Economic Evidence
Examples of Efforts to Improve the Use of Evaluation Evidence
5 A ROADMAP FOR IMPROVING THE USE OF HIGH-QUALITY ECONOMIC EVIDENCE
Overview of the Preceding Chapters
List of Boxes, Figures, and Tables
BOXES
S-1 Methods Used to Produce Economic Evidence
S-2 What Consumers and Producers of Economic Evidence Want Each Other to Know
1-2 National Academies Efforts Relevant to the Study Charge
1-3 Information-Gathering Process
1-4 Summary of Key Definitions
2-1 Illustrative Example of Cost Analysis (CA)
2-2 Illustrative Example of Cost-Effectiveness Analysis (CEA)
2-3 Illustrative Example of Benefit-Cost Analysis (BCA)
2-5 The Role of Economic Evidence in Promoting Publicly Funded Home Visiting Programs
2-6 Issues Affecting the Use of Economic Evidence
3-1 Assessing Approaches to Measuring Quality-of-Life Gains
4-1 Building Capacity to Seek and Use Evidence: An Example
4-4 Illustrative Example of Accountability: No Child Left Behind
4-5 Knowledge Translation Strategies
5-1 Recommendations from Chapter 3
5-2 Recommendations from Chapter 4
5-3 What Consumers and Producers of Economic Evidence Want Each Other to Know
FIGURES
3-1 Different types of economic evaluation can be conducted to answer different types of questions
4-1 Opportunities for the use of administrative data in economic evaluations
TABLES
1-1 Types of Interventions Relevant to This Study
1-2 Outcome Domains of Interventions Relevant to This Study
2-1 Types of Economic Evaluation Methods and Associated Information Requirements and Outputs
2-2 Examples of Evidence-Supported Legislation/Programs and Resulting Impacts
3-1 Illustrative Valuation of Fixed and Variable Cost in Cost Analysis
3-2 Examples of Direct and Linked Economic Impacts in Three Benefit-Cost Analysis Studies