In summary, the committee applied a mix of methods and layers of investigation and analysis using a range of available primary and secondary data sources, summarized in Figure 2-3. Using this approach, the committee was able to arrive at findings that could be triangulated to draw conclusions about the performance and impact of PEPFAR even when any one data source was not sufficient or any one methodological approach was not feasible. Building on the interpretation of the available data, the conclusions and recommendations presented in this report represent the consensus reached through the deliberations of the evaluation committee.
There were a number of challenges to carrying out this evaluation. The overarching challenges and limitations are described here, while more specific challenges and limitations are described in Appendix C and in the subsequent chapters of this report.
Limitations to Evaluation Design
The Strategic Approach included a robust discussion of different evaluation designs and the limitations to applying these designs to this study (IOM and NRC, 2010). In summary, one major limitation for the design of this evaluation of PEPFAR is that it was not feasible to identify an appropriate comparison or control, which would typically be the approach used to answer the underlying question of what would have happened if the program had not existed or if it had been implemented differently. A main reason for the lack of an appropriate comparison is that PEPFAR is widely implemented across many partner countries, which were not selected at random but rather for specific strategic reasons.
In addition, an ideal evaluation would use a prospective design, in which data for both intervention and comparison groups would be collected from the beginning of the evaluation. When it is not possible for ethical reasons or practical considerations to have a comparison group, a prospective design can at least allow for the planning of a before-and-after comparison of the intervention group. However, the timeframe of this mandated evaluation begins with the initiation of PEPFAR-supported programs, which took place before the evaluation was mandated, planned, and carried out. It was not feasible to carry out complex intervention and evaluation designs or new data collection in order to make prospective comparisons within the time period and resources for this evaluation. Therefore, the questions asked in this evaluation can only be answered retrospectively.