An ad hoc committee will plan and conduct a 2-day public workshop on “Lessons Learned in the Conduct of Large-Scale, Complex, Multi-National Global Health Program Evaluations.” The scope of the workshop will encompass key points learned in the execution of the IOM’s second evaluation of PEPFAR (to be released in February 2013), the 2009 evaluation of the Global Fund, the 2011 evaluation of the President’s Malaria Initiative (PMI), the 2012 evaluation of the Affordable Medicines Facility–malaria (AMFm), and 1–2 examples of non-health development assistance programs. The workshop will focus on transferable insights gained across the spectrum of choosing the evaluator, framing the evaluation, developing the methodological approach, implementing the study plan, gathering, assessing and analyzing the data, synthesizing the findings and recommendations, and communicating the key messages. The workshop will illuminate the relative benefits and limitations of quantitative, qualitative, and mixed methodologies used for these complex and expensive evaluations.
The workshop will be planned by an appointed, international planning committee and take place in Washington, DC; London, UK; Geneva, Switzerland; or South Africa. The participants will include individuals involved with the execution of the previously mentioned evaluations, representatives of the agencies that commissioned the evaluations, experts in a range of evaluation methods, and diverse users of the published evaluations. The discussions will reflect the reality of executing these kinds of evaluations, rather than evaluation theory. The goal is to extend evaluation methodologies by capturing lessons learned. Representatives of foundations, bilateral
and multilateral governmental donors, and NGOs also will be invited to attend and participate as will representatives of the academic, international organization, and scientific communities involved with the execution of evaluations of this type.
The format of the workshop will include presentations, moderated panel discussions, and small group consultations for participants to explore the application of lessons learned to future work. The workshop may be webcast and archived if resources allow and a rapporteur-authored book length report will summarize the proceedings and be published for a global audience by the U.S. National Academies Press.
Specific issues to be explored in the workshop include
- Perspectives on terminology, levels of evidence, and the nature of evidence.
- Issues in program evaluation design, including determination of appropriate evaluation questions to be addressed with appropriately matched methods; prospective and retrospective approaches; longitudinal versus cross-sectional versus comparison approaches; and purposeful sampling to maximize feasibility and representativeness.
- Methodological issues in evaluating complex and dynamic programs while they are evolving.
- Data sources, data quality assurance, and post-evaluation data access issues.
- The complementary utility of quantitative and qualitative mixed methods approaches to evaluation; creating an effective interdisciplinary scientific team.
- Managing the challenge of program contribution versus attribution.
- Component analysis.
- Framing and communicating key findings and recommendations in terms that are useful to stakeholders.
- Using evaluations to guide future program implementation and program improvement.
- Using evaluations to guide or influence policy and funding decisions.