National Academies Press: OpenBook

Evaluation Design for Complex Global Initiatives: Workshop Summary (2014)

Chapter: Appendix B: Workshop Agenda

« Previous: Appendix A: Statement of Task
Suggested Citation:"Appendix B: Workshop Agenda." Institute of Medicine. 2014. Evaluation Design for Complex Global Initiatives: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/18739.
×

Appendix B

Workshop Agenda

Workshop on Evaluation Methods for Large-Scale,
Complex, Multi-National Global Health Initiatives
January 7–8, 2014
Wellcome Trust, Gibbs Building, 215 Euston Road, London

GOALS OF THE WORKSHOP

The goal of the workshop was to elucidate the decision making needed and options available to develop and implement a credible, rigorous evaluation that is also feasible, affordable, and maximally matched to the priority evaluation questions, aims, and audiences. Workshop sessions identified the resultant gains and trade-offs from different choices across the key elements that make up a large-scale evaluation.

The workshop convened relevant stakeholders, including evaluators and users of large-scale evaluations as well as others interested in evaluation methods, to present and discuss insights gained and transferrable lessons learned from the conduct of recent evaluations of the U.S. President’s Emergency Plan for AIDS Relief; the Global Fund to Fight AIDS, Tuberculosis, and Malaria; the U.S. President’s Malaria Initiative; the Affordable Medicines Facility–malaria; and other large-scale global initiatives.

For the core examples chosen, large-scale refers to initiatives with a total cumulative budget over multiple years in at least the hundreds of millions of U.S. dollars. Multi-national means implementation on a global scale, including multiple countries and regions or subregions of the world. Complexity refers to several dimensions of the initiative. One is that the initiative encompasses multiple different components, such as varied types of interventions and programs implemented in varied settings; systems-strengthening efforts; capacity building; efforts to influence policy change; and utilization of heath diplomacy to achieve the aims of initiative. Another

Suggested Citation:"Appendix B: Workshop Agenda." Institute of Medicine. 2014. Evaluation Design for Complex Global Initiatives: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/18739.
×

dimension of complexity is implementation at varied levels within partner countries through a large number of diverse, multi-sectoral partners, including an emphasis on local governments and nongovernmental institutions.

While the core examples discussed reflected complex initiatives on this global scale, the intent of the workshop was to also elucidate how the same considerations and decision making can be applied to enhance the credibility, rigor, feasibility, and utility for evaluations that may be on a smaller scale yet share similar features of complexity.

This workshop was sponsored by the Bill & Melinda Gates Foundation, the Doris Duke Charitable Foundation, the Wellcome Trust, and the William and Flora Hewlett Foundation.

AGENDA

Tuesday, January 7, 2014
8:00A – 8:30A Registration and Breakfast
 
8:30A – 8:45A Welcoming Remarks and Overview of Workshop Agenda

Ann Kurth, New York University, Planning Committee Chair

 
8:45A – 9:15A Setting the Stage: Workshop Framing and Crosscutting Issues

Simon Hearn, Overseas Development Institute/BetterEvaluation Initiative

 
9:15A – 10:45A Panel 1: Framing the evaluation, choosing the evaluator, and establishing the governance structure for the evaluation
 
  OBJECTIVES OF THE PANEL:
  At the end of this session, workshop participants will be able to identify and understand the key considerations and decisions needed to:
 
  • Identify the evaluation goals, develop and prioritize the evaluation questions, and define the audience and the intended uses.
  • Understand how decisions and trade-offs in
Suggested Citation:"Appendix B: Workshop Agenda." Institute of Medicine. 2014. Evaluation Design for Complex Global Initiatives: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/18739.
×
 
  • choosing the type of evaluator and in setting up the governance structure for the evaluation may be affected by the different aims/audiences and the degree to which they are aligned.

  • Understand how the options and trade-offs in choosing the type of evaluator and in setting up the governance structure contribute to designing and conducting the best evaluation for the available funds.
  • Identify how to understand, ensure, and assess the value and utility of the evaluation and its findings.
  PANELISTS:
  Overview Speaker/Moderator: Jon Simon, Boston University, Planning Committee Member

Discussants: Chris Whitty, UK Department for International Development

Ryuichi Komatsu, Global Fund to Fight AIDS, Tuberculosis, and Malaria

Deborah Rugg, United Nations Inspection and Evaluation Division

Carmela Green-Abate, Country Coordinator, PEPFAR Ethiopia

Robert Black, Johns Hopkins University

 
10:45A – 11:15A BREAK
 
11:15A – 1:00P Panel 2: Developing the evaluation design and methodological approaches
 
OBJECTIVES OF THE PANEL:
At the end of this session, workshop participants will understand the following:
  • The importance of mixed methods approaches to evaluate complex initiatives and to understand not only whether an effect was achieved but also how and why.
Suggested Citation:"Appendix B: Workshop Agenda." Institute of Medicine. 2014. Evaluation Design for Complex Global Initiatives: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/18739.
×
  • The importance of strategically thinking through different options for methods and how they can be matched to the evaluation aims/questions, the available data, and the feasibility of implementing the methods with appropriate rigor (design that is “fit for purpose” and feasible).
  • How to recognize, understand, and grapple with the complexity of the initiative being evaluated and the context in which that initiative is implemented—this affects, for example, attribution versus contribution designs and expectations for findings.
  • The importance of planning for and building flexibility into the design.
PANELISTS:
Moderator: Kara Hanson, London School of Hygiene and Tropical Medicine, Planning Committee Member
 

Presentation: Evaluation of PEPFAR

Deborah Rugg, United Nations Inspection and Evaluation Division

Presentation: Evaluation of Global Fund

Daniel Low-Beer, Global Fund to Fight AIDS, Tuberculosis, and Malaria

 

Presentation: Evaluation of the Affordable Medicines Facility–malaria

Catherine Goodman, London School of Hygiene and Tropical Medicine

Presentation: Illustration of Good Practices for Complex Evaluation Design

Elliot Stern, Lancaster University

 
1:00P – 2:15P LUNCH
 
2:15P – 4:00P Panel 3: Mapping data sources and gathering and assessing the data
Suggested Citation:"Appendix B: Workshop Agenda." Institute of Medicine. 2014. Evaluation Design for Complex Global Initiatives: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/18739.
×
OBJECTIVES OF THE PANEL:
At the end of this session, workshop participants will understand the following:
  • The importance of and strategies for identifying and assessing potential data sources and mapping these to the evaluation questions and the methodological approaches being considered (data that is “fit for purpose”).
  • The key elements of assessing data sources:
  • Availability
  • Accessibility (including data ownership issues and data sharing agreements)
  • Quality
  • Feasibility of gathering/accessing the identified data
PANELISTS:
Overview Speaker/Moderator: Ann Kurth, New York University, Planning Committee Chair

Discussants: Martin Vaessen, ICF, Planning Committee Member

Kara Hanson, London School of Hygiene and Tropical Medicine, Planning Committee Member

Jon Simon, Boston University, Planning Committee Member

Batya Elul, Columbia University

Victoria Fan, Center for Global Development

Peter Elias, University of Warwick

 
4:00P – 4:30P BREAK
 
4:30P – 6:00P Panel 4: Analyzing data using triangulation in a mixed methods evaluation to reach overall findings, conclusions, and recommendations
Suggested Citation:"Appendix B: Workshop Agenda." Institute of Medicine. 2014. Evaluation Design for Complex Global Initiatives: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/18739.
×
OBJECTIVES OF THE PANEL:
At the end of this session, workshop participants will understand the following key considerations in data analysis and interpretation for a complex, mixed methods evaluation:
  • Data analysis and interpretation within each data type/method according to the methodological standards of rigor for that method (i.e., avoiding “mixed-up” methods).
  • Triangulation among the findings from mixed methods to achieve an overall, integrated analysis and interpretation to arrive at major conclusions and recommendations.
  • Triangulation among investigators.
  • Among members of the evaluation team
  • Use of expert/advisory panels
  • Use of internal or external reviewers
PANELISTS:
Overview Speaker/Moderator: Carlo Carugi, Global Environment Facility
 

Discussants: Jon Simon, Boston University, Planning Committee Member

Catherine Goodman, London School of Hygiene and Tropical Medicine

Sangeeta Mookherji, George Washington University

Bridget Kelly, Institute of Medicine

 
6:00P – 6:15P Day 1 Wrap-Up

Ann Kurth, New York University, Planning Committee Chair

Wednesday, January 8, 2014
8:00A – 8:30A Registration and Breakfast
 
8:30A – 8:45A Welcoming Remarks: Brief Reflection on Day 1 and Overview of Day 2 Agenda

Ann Kurth, New York University, Planning Committee Chair

Suggested Citation:"Appendix B: Workshop Agenda." Institute of Medicine. 2014. Evaluation Design for Complex Global Initiatives: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/18739.
×
8:45A – 10:15A Panel 5: Using the evaluation findings and communicating the key messages
 
OBJECTIVES OF THE PANEL:
At the end of this session, workshop participants will understand the following:
  • Complexity of diverse uses and audiences for large-scale evaluations (e.g., accountability to funders/legislators; program improvement at headquarter, implementer, and country program levels; advocacy to continue initiative; public awareness).
  • Importance of matching the message, the messenger, and the audience (communication that is “fit for purpose”).
  • Challenges with tracking the use of large-scale evaluation findings.
PANELISTS:
Moderator: Sir George Alleyne, University of West Indies, Planning Committee Member

Discussants: Bernard Nahlen, President’s Malaria Initiative

Dorothy Muroki, FHI360, Planning Committee Member

Ian Goldman, Department of Performance Monitoring and Evaluation, Government of South Africa

Kimberly Scott, Institute of Medicine

 
10:15A – 10:45A BREAK
 
10:45A – 12:15P Concurrent Sessions—Block 1
 
1A: Lessons from Large-Scale Program Evaluation on a Not-Quite-as-Large Scale
 
Moderator: Dorothy Muroki, FHI360, Planning Committee Member
Suggested Citation:"Appendix B: Workshop Agenda." Institute of Medicine. 2014. Evaluation Design for Complex Global Initiatives: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/18739.
×

Presenters: Margaret Kruk, Columbia University

Padma Chandrasekaran, The Chennai Angels

Tanya Marchant, London School of Hygiene and Tropical Medicine

 
1B: Evolving Methods in Evaluation Science
Moderator: Sir George Alleyne, University of West Indies, Planning Committee Member

Presenters: Geoff Wong, Queen Mary University of London

Emmanuela Gakidou, Institute for Health Metrics and Evaluation

Caine Rolleston, University of London

 
1C: Strategic Evaluation Design Troubleshooting: “Bring Your Own” Evaluation Challenge
Workshop participants are invited to bring their own current evaluation design challenges for a roundtable discussion of design options and solutions with evaluation expert panelists and colleagues.
 
Facilitator: Kimberly Scott, Institute of Medicine
 

Evaluation experts: Tessie Catsambas, EnCompass LLC

Sharon Knight, East Carolina University

 
12:15P – 1:30P LUNCH: Workshop Feedback Session
Facilitator: Sharon Knight, East Carolina University
 
1:30P – 3:00P Concurrent Sessions—Block 2
 
2A: Applying Qualitative Methods to Evaluation on a Large Scale
 
Moderator: Kate Meck, Institute of Medicine
 

Presenters: Sharon Knight, East Carolina University Tessie Catsambas, EnCompass LLC

Suggested Citation:"Appendix B: Workshop Agenda." Institute of Medicine. 2014. Evaluation Design for Complex Global Initiatives: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/18739.
×
2B: Applying Quantitative Methods for Evaluation on a Large Scale
 
Moderator: Martin Vaessen, ICF International
 
Presenters: Eran Bendavid, Stanford University

Charlotte Watts, London School of Hygiene and Tropical Medicine

Rachel Nugent, University of Washington

 
2C: Strategic Evaluation Design Troubleshooting: “Bring Your Own” Evaluation Challenge
Workshop participants are invited to bring their own current evaluation design challenges for a roundtable discussion of design options and solutions with evaluation expert panelists and colleagues.
 
Facilitator: Bridget Kelly, Institute of Medicine
 
Evaluation experts: Batya Elul, Columbia University

Sangeeta Mookherji, George Washington University

Joanna Schellenberg, London School of Hygiene and Tropical Medicine

 
3:00P – 3:30P BREAK
 
3:30P – 4:45P Panel 6: Envisioning a future evaluation
 
OBJECTIVE OF THE PANEL:
Synthesize key messages of the workshop by applying them in a hypothetical design exercise.
 
PANELISTS:
Moderator: Elliot Stern, Lancaster University
 

Presenters: Sanjeev Sridharan, Evaluation Centre for Complex Health Interventions, University of Toronto/St. Michaels’s Hospital

Charlotte Watts, London School of Hygiene and Tropical Medicine

Suggested Citation:"Appendix B: Workshop Agenda." Institute of Medicine. 2014. Evaluation Design for Complex Global Initiatives: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/18739.
×
4:45P – 5:15P Summary Highlights of the Workshop and Reflections on Key Outcomes
  Ruth Levine, William and Flora Hewlett
  Foundation
Additional Reflections
  Mary Bassett, Doris Duke Charitable Foundation

Gina Dallabetta, Bill & Melinda Gates Foundation

Jimmy Whitworth, Wellcome Trust
5:15P – 5:30P Closing Remarks and Adjournment
  Ann Kurth, New York University, Planning Committee Chair
Suggested Citation:"Appendix B: Workshop Agenda." Institute of Medicine. 2014. Evaluation Design for Complex Global Initiatives: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/18739.
×
Page 119
Suggested Citation:"Appendix B: Workshop Agenda." Institute of Medicine. 2014. Evaluation Design for Complex Global Initiatives: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/18739.
×
Page 120
Suggested Citation:"Appendix B: Workshop Agenda." Institute of Medicine. 2014. Evaluation Design for Complex Global Initiatives: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/18739.
×
Page 121
Suggested Citation:"Appendix B: Workshop Agenda." Institute of Medicine. 2014. Evaluation Design for Complex Global Initiatives: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/18739.
×
Page 122
Suggested Citation:"Appendix B: Workshop Agenda." Institute of Medicine. 2014. Evaluation Design for Complex Global Initiatives: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/18739.
×
Page 123
Suggested Citation:"Appendix B: Workshop Agenda." Institute of Medicine. 2014. Evaluation Design for Complex Global Initiatives: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/18739.
×
Page 124
Suggested Citation:"Appendix B: Workshop Agenda." Institute of Medicine. 2014. Evaluation Design for Complex Global Initiatives: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/18739.
×
Page 125
Suggested Citation:"Appendix B: Workshop Agenda." Institute of Medicine. 2014. Evaluation Design for Complex Global Initiatives: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/18739.
×
Page 126
Suggested Citation:"Appendix B: Workshop Agenda." Institute of Medicine. 2014. Evaluation Design for Complex Global Initiatives: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/18739.
×
Page 127
Suggested Citation:"Appendix B: Workshop Agenda." Institute of Medicine. 2014. Evaluation Design for Complex Global Initiatives: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/18739.
×
Page 128
Next: Appendix C: Participant Biographies »
Evaluation Design for Complex Global Initiatives: Workshop Summary Get This Book
×
Buy Paperback | $50.00 Buy Ebook | $39.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Every year, public and private funders spend many billions of dollars on large-scale, complex, multi-national health initiatives. The only way to know whether these initiatives are achieving their objectives is through evaluations that examine the links between program activities and desired outcomes. Investments in such evaluations, which, like the initiatives being evaluated, are carried out in some of the world's most challenging settings, are a relatively new phenomenon. In the last five years, evaluations have been conducted to determine the effects of some of the world's largest and most complex multi-national health initiatives.

Evaluation Design for Complex Global Initiatives is the summary of a workshop convened by the Institute of Medicine in January 2014 to explore these recent evaluation experiences and to consider the lessons learned from how these evaluations were designed, carried out, and used. The workshop brought together more than 100 evaluators, researchers in the field of evaluation science, staff involved in implementing large-scale health programs, local stakeholders in the countries where the initiatives are carried out, policy makers involved in the initiatives, representatives of donor organizations, and others to derive lessons learned from past large-scale evaluations and to discuss how to apply these lessons to future evaluations. This report discusses transferable insights gained across the spectrum of choosing the evaluator, framing the evaluation, designing the evaluation, gathering and analyzing data, synthesizing findings and recommendations, and communicating key messages. The report also explores the relative benefits and limitations of different quantitative and qualitative approaches within the mixed methods designs used for these complex and costly evaluations.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!