Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

DESIGN CONSIDERA TIONS FOR EVALUATING THE IMPACT OF PEPFAR W O R K S H O P S U M M A R Y Clara Cohen, Michele Orza, and Deepali Patel, Rapporteurs Board on Global Health 

THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Govern- ing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineer- ing, and the Institute of Medicine. This study was supported by Contract No. SAQMPD05D1147 (STAT-7394) be- tween the National Academy of Sciences and the Department of State. Any opin- ions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the organizations or agencies that provided support for this project. International Standard Book Number-13:  978-0-309-11672-5 International Standard Book Number-10:  0-309-11672-4 Additional copies of this report are available from the National Academies Press, 500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap. edu. For more information about the Institute of Medicine, visit the IOM home page at: www.iom.edu. Copyright 2008 by the National Academy of Sciences. All rights reserved. Printed in the United States of America The serpent has been a symbol of long life, healing, and knowledge among almost all cultures and religions since the beginning of recorded history. The serpent adopted as a logotype by the Institute of Medicine is a relief carving from ancient Greece, now held by the Staatliche Museen in Berlin. Suggested citation: IOM (Institute of Medicine). 2008. Design considerations for evaluating the impact of PEPFAR: Workshop summary. Washington, DC: The National Academies Press. 

“Knowing is not enough; we must apply. Willing is not enough; we must do.” —Goethe Advising the Nation. Improving Health. 

The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Acad- emy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding en- gineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineer- ing programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is presi- dent of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Insti- tute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sci- ences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Coun- cil is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council. www.national-academies.org 

Reviewers This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Research Council’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: Ties Boerma, Measurement and Health Information System, World Health Organization Fred Carden, International Development Research Centre Helen Gayle, CARE Ruth Levine, Center for Global Development Phillip Nieburg, HIV/AIDS Task Force, Center for Strategic and International Studies Nancy Padian, Research Triangle Institute Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the final draft of the report before its release. The review of this report was overseen by  

vi REVIEWERS Dr. Frederick A. Murphy, Department of Pathology, the University of Texas Medical Branch at Galveston. Appointed by the National Research Council, he was responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authors and the institution. 

Contents Preface xi Overview 1 1 Introduction to Impact Evaluation for PEPFAR 23 Meaning and Uses of Impact Evaluation, 23 PEPFAR’s Evaluative Approach, 27 Evaluative Approach and Major Findings of the IOM PEPFAR Evaluation Committee, 34 2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting 37 Cost-effectiveness, 38 Conceptual Approach, 39 Health Impacts, 42 Impacts Beyond Health, 46 Impacts on Sustainability, Capacity Building, and Health Systems Strengthening, 47 Coordination and Harmonization, 52 Sustainability Impacts, 57 Equity and Fairness Impacts, 58 Unintended Impacts, 62 vii 

viii CONTENTS 3 Designing an Evaluation That Incorporates the Guiding Principles of Coordination, Harmonization, and Capacity Building 67 Benefits, Costs, and Opportunities of Coordination and Harmonization in Evaluation, 67 Benefits, Constraints, and Opportunities of Building Capacity in Evaluation, 73 4 Designing an Impact Evaluation with Robust Methodologies 77 Conceptual Models and Methodological Approaches: Case Studies, 77 Methodological Challenges and Opportunities in Evaluating Impact, 86 Themes Common to Evaluation Methodologies and Approaches, 107 APPENDIXES A Agenda 111 B Abbreviations and Acronyms 119 C List of Participants 121 D References 125 

List of Tables, Figures, and Boxes Tables O-1 Challenges and Opportunities in Measuring HIV/AIDS-Specific and General Impacts, 12 2-1 Possible Effects of ART on HIV Transmission, 43 Figures 1-1 PEPFAR strategic information budget, 2004–2007, 29 1-2 Structure of PHE, 33 4-1 Impacts of alternative HIV/AIDS education strategies on girls’ behavioral outcomes, 82 4-2 Private-sector attrition data show evidence of early ART impact on mortality, 92 Boxes O-1 Summary of Impact Evaluation Questions as Identified by Workshop Participants, 5 1-1 Introduction to PEPFAR, 28 1-2 Main Recommendations from IOM Evaluation of PEPFAR, 34 2-1 Comanagement of PEPFAR in Rwanda: A Case Study, 55 2-2 The Elements of Fairness, 59 ix 



Preface Human immunodeficiency virus/acquired immunodeficiency syndrome (HIV/AIDS) has become one of the largest epidemics in history, with more than 33 million people living with the disease, over 2 million deaths, and more than 2 million new infections estimated last year (UNAIDS and WHO, 2007). Developing countries—where the epidemic has caused not only loss of life, but also major social and economic dislocations—have borne a disproportionate share of the HIV/AIDS disease burden. The United States has become a major player in the global response to HIV/AIDS through the President’s Emergency Plan for AIDS Relief (PEPFAR). Legislated in 2003 through the United States Leadership Against HIV/AIDS, Tuberculosis, and Malaria Act of 2003 (The Leadership Act), PEPFAR is a 5-year effort that seeks to prevent 7 million new AIDS infec- tions, treat 2 million people with AIDS, and care for 10 million orphans and other vulnerable groups, with a focus on 15 target countries. About the Workshop Also included in The Leadership Act legislation was a mandate to the Institute of Medicine (IOM) to appoint an expert committee to conduct an evaluation of the implementation of PEPFAR. The IOM committee began work on the evaluation very early in PEPFAR’s implementation because the evaluation was mandated to be delivered 3 years after the legislation was passed. It was possible to evaluate only the first phase of the implementa- xi 

xii PREFACE tion, and at the close of the committee’s evaluation, PEPFAR had been sup- porting programs in the focus countries for less than 2 years. As the final element of its project on PEPFAR, the IOM convened a workshop, “Design Considerations for Evaluating the Impact of PEPFAR,” on April 30 and May 1, 2007. The workshop focused on developing meth- odological, policy, and practical design considerations for a future evalua- tion of PEPFAR’s impact at a point when the program is sufficiently mature to fairly judge its impact. The workshop underscored what the evaluation committee and workshop participants would have liked to have evaluated in the long term and sought to outline more meaningful questions about the true impact of the program. Largely because of the mandated timing, the implementation evaluation that the IOM committee provided in its recently published report, PEPFAR Implementation: Progress and Promise (IOM, 2007), could not answer the questions that deeply interest the U.S. Congress and others about the impact of the program. Although the PEPFAR report was limited in looking at early indicators—inputs, processes, and a few outputs—the workshop was able to address what longer term outcomes and impacts could be evaluated in the future.  Three main perspectives on accountability were sought at the workshop: “upward” accountability to the U.S. Congress, “horizontal” accountability to global partners, and “downward” accountability to country partners and intended beneficiaries. The workshop was widely consultative and brought together a range of interested parties—including staff of the U.S. Congress; PEPFAR officials and implementers; major multilateral organizations such as The Global Fund to Fight AIDS, Tuberculosis, and Malaria (The Global Fund), the Joint United Nations Programme on HIV/AIDS (UNAIDS), and the World Bank; evaluation experts experienced with similar types of evalu- ations; and representatives of partner countries, particularly the PEPFAR focus countries. The first day of the workshop was devoted to outlining broader issues and hearing from relevant perspectives; the second day focused on technical and methodological issues in evaluating the impact of PEPFAR. Is PEPFAR helping partner countries to succeed, and how could it do better? What are the right specific questions to ask, and what are the best ways to get the answers? How can PEPFAR coordinate and harmonize to get the most from its evaluation resources? These are among the basic questions that the workshop discussions addressed. Organization of the Workshop Summary This workshop summary is divided into four chapters, preceded by an Overview. The Overview puts forth core messages that arose from the workshop presentations and discussions that may be of greatest interest to 

PREFACE xiii decision makers. Chapter 1 introduces the PEPFAR program and includes workshop discussions on the definition of impact evaluation and previous evaluation efforts, including those internal to the PEPFAR program and that by the IOM committee. In Chapter 2, workshop participants provided their vision for the questions of interest that could be addressed in a future impact evaluation of PEPFAR. These questions include those about both the process and the results of program implementation. Chapter 3 describes the benefits, costs, and opportunities for conducting the impact evaluation in a way that incorporates guiding principles of coordination, harmonization, and capacity building. Finally, Chapter 4 describes workshop participants’ discussions of the methodologies and approaches that can be used in im- pact evaluation, lessons learned from previous evaluations of HIV/AIDS programs, and specific methodological challenges and opportunities. The meeting agenda, list of acronyms, list of participants, and bibliographic references are included in the report’s appendixes. The authors prepared this summary on the basis of attendance at the workshop, associated materials, and a transcript, webcast, and audio- recordings of the meeting, presentations, and discussions that took place during the workshop. Chapters have been edited and organized around major themes to provide a more readable summary and to eliminate dupli- cation of topics. The material presented reflects only the views and opin- ions of those participating in the workshop and not the consensus view of a formally constituted study committee. The summary reflects only what was covered at the workshop and is not intended to be a comprehensive examination of the subject matter. Acknowledgments We are grateful to the many people who contributed to making the workshop a success. Many thanks to Ruth Levine for superbly moderating the meeting and to Phil Nieburg for his able assistance in moderating. We appreciate the continued service of the IOM Committee for the Evaluation of PEPFAR Implementation in serving as the steering committee for the workshop, especially those members who also participated in the meeting: Drs. Stefano Bertozzi, Geoff Garnett, Bill Holzemer, Carl Latkin, and Jim Sherry. All of the speakers and discussants were admirably generous with their considerable expertise but limited time, particularly Jonathan Mwiindi and Dr. Agnes Binagwaho, whose travel from Africa required an invest- ment of several days in addition to the workshop. We also appreciate the continued hard work of the staff—Dr. Michele Orza, Kimberly Scott, and Angela Mensah—for whom the preparations for this workshop followed immediately upon the release of the committee’s report, leaving them no time for even a brief respite. They could not have accomplished it without 

xiv PREFACE the kind and expert assistance of Hellen Gelband. Deepali Patel’s editorial support in revising the draft workshop summary in response to the com- ments of the external reviewers is also greatly appreciated. Thank you to the PEPFAR evaluation team, led by Drs. Kathy Marconi and Paul Bouey, for their support in planning the meeting and participation throughout. We are grateful to the Kaiser Family Foundation for its webcast of the work- shop proceedings, allowing people who were not able to join us to see and hear the proceedings. Last, but most definitely not least, special thanks to our lead author, Dr. Clara Cohen, who somehow managed to condense 2 full days of detailed presentations and involved discussions into this coher- ent and useful summary. The Road Ahead The workshop and this summary are intended to be helpful to the U.S. Congress in developing expectations for the evaluation of PEPFAR’s impact as well as to those involved in implementing and evaluating the PEPFAR program. Because the law authorizing PEPFAR will expire in September 2008, I hope the report will contribute to developing a compelling, in- formed, and expanded vision for building on PEPFAR’s initial success. In her remarks to conclude the workshop, moderator Ruth Levine used a colorful and creative analogy comparing PEPFAR to a car with a full tank of gas—several billions of dollars worth—with instructions to go as fast as possible. She described back-seat drivers in the car who admonish the driver about speed, direction, and number of passengers and a road that is also moving at the same time. Where is the car relative to where it wants to be? Where is the car relative to where it was? Is the car moving in the most direct way to where it should be? Should we stop driving or continue driving in the same direction without looking at the signals along the way? This workshop has demonstrated that the best option is to ask key ques- tions, look for signs that can help orient us all, and keep moving. Jaime Sepúlveda, Chair IOM Committee for the Evaluation of PEPFAR Implementation