EVALUATION

DESIGN

FOR COMPLEX

GLOBAL

INITIATIVES

WORKSHOP SUMMARY

Steve Olson, Rapporteur

Board on Global Health

INSTITUTE OF MEDICINE
        OF THE NATIONAL ACADEMIES

THE NATIONAL ACADEMIES PRESS

Washington, D.C.

www.nap.edu



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
Steve Olson, Rapporteur Board on Global Health

OCR for page R1
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001 NOTICE: The workshop that is the subject of this workshop summary was approved by the Governing Board of the National Research Council, whose mem- bers are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. This activity was supported by contracts between the National Academy of Sciences and the Bill & Melinda Gates Foundation, the Doris Duke Charitable Foundation, the Wellcome Trust, and the William and Flora Hewlett Foundation. The views presented in this publication do not necessarily reflect the views of the organizations or agencies that provided support for the activity. International Standard Book Number-13:  978-0-309-30258-6 International Standard Book Number-10:  0-309-30258-7 Additional copies of this workshop summary are available for sale from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu. For more information about the Institute of Medicine, visit the IOM home page at: www.iom.edu. Copyright 2014 by the National Academy of Sciences. All rights reserved. Printed in the United States of America The serpent has been a symbol of long life, healing, and knowledge among almost all cultures and religions since the beginning of recorded history. The serpent adopted as a logotype by the Institute of Medicine is a relief carving from ancient Greece, now held by the Staatliche Museen in Berlin. Suggested citation: IOM (Institute of Medicine). 2014. Evaluation design for com- plex global initiatives: Workshop summary. Washington, DC: The National Acad- emies Press.

OCR for page R1
“Knowing is not enough; we must apply. Willing is not enough; we must do.” —Goethe Advising the Nation. Improving Health.

OCR for page R1
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Acad- emy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineer- ing programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. C. D. Mote, Jr., is presi- dent of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Insti- tute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sci- ences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Coun- cil is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. C. D. Mote, Jr., are chair and vice chair, respectively, of the National Research Council. www.national-academies.org

OCR for page R1
PLANNING COMMITTEE FOR THE WORKSHOP ON EVALUATING LARGE-SCALE, COMPLEX, MULTI-NATIONAL GLOBAL HEALTH INITIATIVES1 ANN KURTH (Chair), New York University, New York, NY GEORGE ALLEYNE, Pan American Health Organization, Washington, DC KARA HANSON, London School of Hygiene & Tropical Medicine, London, United Kingdom DOROTHY MUROKI, FHI 360, Nairobi, Kenya JONATHON SIMON, Boston University, Boston, MA MARTIN VAESSEN, ICF International, Rockville, MD IOM Staff BRIDGET B. KELLY, Project Co-Director KIMBERLY A. SCOTT, Project Co-Director KATE MECK, Associate Program Officer CHARLEE ALEXANDER, Senior Program Assistant (from November 2013) JULIE WILTSHIRE, Financial Officer PATRICK W. KELLEY, Senior Board Director, Board on Global Health 1  Institute of Medicine planning committees are solely responsible for organizing the work- shop, identifying topics, and choosing speakers. The responsibility for the published workshop summary rests with the workshop rapporteur and the institution. v

OCR for page R1

OCR for page R1
Reviewers This workshop summary has been reviewed in draft form by individu- als chosen for their diverse perspectives and technical expertise, in accor- dance with procedures approved by the National Research Council’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published workshop summary as sound as possible and to ensure that the workshop summary meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the process. We wish to thank the following individuals for their review of this workshop summary: RICHARD BERK, University of Pennsylvania ANN KURTH, New York University DOROTHY MUROKI, FHI 360, SANJEEV SRIDHARAN, University of Toronto HOWARD WHITE, International Initiative for Impact Evaluation (3ie) Although the reviewers listed above have provided many constructive comments and suggestions, they did not see the final draft of the workshop summary before its release. The review of this workshop summary was overseen by Enriqueta Bond, President Emeritus, Burroughs Wellcome Fund. Appointed by the Institute of Medicine, she was responsible for vii

OCR for page R1
viii REVIEWERS making certain that an independent examination of this workshop sum- mary was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this workshop summary rests entirely with the rapporteur and the institution.

OCR for page R1
Acknowledgments The planning committee and project staff deeply appreciate many valu- able contributions from individuals who assisted us with this project. First, we offer our profound thanks to all of the presenters and discussants at the workshop, who gave so generously of their time and expertise. These individuals are listed in full in the workshop agenda in Appendix B. We are also grateful to the many participants who attended the workshop both in person and via the live webcast. The engagement of all those in attendance was robust and vital to the success of the event. We are also particularly appreciative of the thoughtful and creative contributions of Mary Ellen Kelly and James Kelly, who applied their many years of experience to help generate the fictional initiative used for the hypothetical design exercise at the workshop. In addition, we thank the sponsors of this project for their support: the Bill & Melinda Gates Foundation, the Doris Duke Charitable Foundation, the Wellcome Trust, and the William and Flora Hewlett Foundation. We are also grateful to the Wellcome Trust for hosting the workshop, with a special thanks to Zoe Storey, Danielle Taplin, and all of the staff there for their gracious assistance in support of every aspect of the event. We also extend many thanks to Anthony Mavrogiannis and the staff at Kentlands Travel for supporting the travel needs and requirements of this project. We appreciate LeAnn Locher’s creative work in designing the report cover. Finally, we convey our gratitude for the hard work of the many staff of the Institute of Medicine and the National Academies who supported the proj- ect at every stage, from its inception to the workshop to the final production of this workshop summary report. ix

OCR for page R1

OCR for page R1
Contents 1 INTRODUCTION AND OVERVIEW OF THE WORKSHOP 1 Terms and Objectives, 2 Messages from the Workshop, 4 Organization of the Workshop Report, 9 2 OVERVIEW FRAMEWORK FOR COMPLEX EVALUATIONS 11 The BetterEvaluation Initiative Rainbow Framework, 12 Navigating the Framework, 15 3 FRAMING THE EVALUATION 17 Reflections from the Experience of the Evaluation of the U.S. President’s Malaria Initiative, 18 Evaluation at the United Nations, 19 An Evaluation Funder’s Perspective on Where Problems Arise, 21 Lessons Learned from the Global Fund to Fight AIDS, Tuberculosis, and Malaria, 22 Comparing Evaluations of Selected Global Initiatives, 24 Impact of an Evaluation, 25 Other Topics Raised in Discussion, 27 4 DEVELOPING THE EVALUATION DESIGN AND SELECTING METHODS 29 Institute of Medicine Evaluation of PEPFAR, 30 Global Fund Evaluation, 32 xi

OCR for page R1
xii CONTENTS Affordable Medicines Facility–Malaria Assessment, 35 The Search for Good Practice in Complex Evaluations, 38 Other Topics Raised in Discussion, 40 5 MAPPING DATA SOURCES AND GATHERING AND ASSESSING DATA 43 Data Mapping in the IOM Evaluation of PEPFAR, 44 Data Issues in the Global Fund Evaluation, 44 Data Approach and Challenges in the Affordable Medicines Facility–Malaria Evaluation, 46 President’s Malaria Initiative Evaluation, 48 Using Routine Program Data, 49 Working with Financial Data, 51 Developing a Large-Scale Data Infrastructure, 52 Other Topics Raised in Discussion, 53 6 APPLYING QUALITATIVE METHODS TO EVALUATION ON A LARGE SCALE 55 Rigor and Credibility in Qualitative Design, 56 The Value of Qualitative Methods, 58 Other Topics Raised in Discussion, 60 7 APPLYING QUANTITATIVE METHODS TO EVALUATION ON A LARGE SCALE 61 Outcomes Matter, 62 Mathematical Modeling as a Tool for Program Evaluation, 63 Extended Cost-Effectiveness Analysis, 66 Other Topics Raised in Discussion, 67 8 ANALYSIS THROUGH TRIANGULATION AND SYNTHESIS TO INTERPRET DATA IN A MIXED METHODS EVALUATION 69 Triangulation in the Global Environmental Facility, 70 Developing a Deeper and Wider Understanding of Results, 72 Triangulation in Practice, 73 Approaches to Triangulation, 74 Lessons from the IOM’s PEPFAR Evaluation, 75 Other Topics Raised in Discussion, 77 9 EVOLVING METHODS IN EVALUATION SCIENCE 79 Use of Realist Methods to Evaluate Complex Interventions and Systems, 79

OCR for page R1
CONTENTS xiii Innovative Designs for Complex Questions, 81 Comparative Systems Analysis: Methodological Challenges and Lessons from Education Research, 84 Other Topics Raised in Discussion, 86 10  LESSONS FROM LARGE-SCALE PROGRAM EVALUATION ON A NOT-QUITE-AS-LARGE SCALE 87 Saving Mothers, Giving Life Strategic Implementation Evaluation, 87 Avahan—Reducing the Spread of HIV in India, 90 EQUIP—Expanded Quality Management Using Information Power, 91 Other Topics Raised in Discussion, 94 11  USING EVALUATION FINDINGS AND COMMUNICATING KEY MESSAGES 95 The U.S. President’s Malaria Initiative, 96 Roads to a Healthy Future, 97 The South African Perspective on Influencing Policy and Performance, 99 Communicating Results from the PEPFAR Evaluation, 100 Other Topics Raised in Discussion, 102 12 ENVISIONING A FUTURE FOR EVALUATION 105 Framing the Evaluation, 107 Building an Ecology of Evidence, 109 Capacity Building for Future Evaluators, 113 REFERENCES 115 APPENDIXES A STATEMENT OF TASK 117 B WORKSHOP AGENDA 119 C PARTICIPANT BIOGRAPHIES 129 D EVALUATION INFORMATION SUMMARY FOR CORE EXAMPLE INITIATIVES 149 E EVALUATION DESIGN RESOURCES HIGHLIGHTED AT THE WORKSHOP 169

OCR for page R1