Limitations on Data Availability
A primary and very concrete challenge to the evaluation was the limited availability of data to address health outcomes and impact across the whole of PEPFAR, a limitation that was revealed by the data mapping and data collection process for this evaluation. The lack of relevant available measures made it difficult, and in some cases impossible, for the evaluation committee to respond directly to aspects of the evaluation as requested in the Lantos-Hyde Act of 2008.
The programmatic indicators that are reported centrally to OGAC across the entire PEPFAR program provide only limited answers to the evaluation charge. There are only nine indicators that are routinely reported centrally to OGAC and that have had stable, consistent indicator definitions since the inception of PEPFAR (see Table 2-2). Therefore, these are the indicators that are available across the whole scope of countries and duration of PEPFAR. These indicators represent limited aspects of PEPFAR’s programmatic areas. They also primarily represent outputs, which can serve to assess program implementation through the volume of services provided, but are limited in terms of outcomes and impact to assess those services in the context of the population in need, to assess the quality of the services provided, and to assess PEPFAR’s effectiveness in achieving measurable effects on health.
Most evaluation questions required the evaluation committee to draw on data that went beyond the indicators that are routinely reported to OGAC. Data from PEPFAR beyond the centrally-reported indicators, such as recommended indicators collected by country programs but not reported to OGAC, data collected independently by the major USG implementing agencies and other implementation partners, financial data by type of partner and expenditures by program activity, results of PEPFAR-supported evaluations, and publications from PEPFAR-supported programs are not managed through processes that allow for ready cataloguing or ready access to what is available. Accessing these data comprehensively would have required a more intensive and significant data-mapping, data-gathering, and data-analysis effort than was possible given the time and resources available for the IOM evaluation. The necessary requests from the IOM also would have imposed a significant burden of time and resources on staff at OGAC and other implementing agencies as well as on mission teams and implementing partners while they simultaneously continued to oversee and implement the program. Therefore, data requests and data gathering were done strategically within the limitations of what could be responded to and completed in a timely manner. In addition to challenges related to feasibility, for some implementing partners, concerns about sacrificing the