2

Evaluation Scope and Approach

CONGRESSIONAL CHARGE

As described in Chapter 1, the U.S. government (USG) currently supports programs to combat global HIV/AIDS through an initiative known as the President’s Emergency Plan for AIDS Relief (PEPFAR). This initiative was originally authorized in the U.S. Leadership Against HIV/AIDS, Tuberculosis, and Malaria Act of 20031 and subsequently reauthorized in the Tom Lantos and Henry J. Hyde United States Global Leadership Against HIV/AIDS, Tuberculosis, and Malaria Reauthorization Act of 2008 (hereinafter, the Lantos-Hyde Act of 2008).2 A description of the history and evolution of the PEPFAR initiative can be found in Chapter 1.

In the Lantos-Hyde Act of 2008 which reauthorized PEPFAR, the U.S. Congress mandated that the Institute of Medicine (IOM) conduct a study that includes “an assessment of the performance of United States-assisted global HIV/AIDS programs” and “an evaluation of the impact on health of prevention, treatment, and care efforts that are supported by United States funding, including multilateral and bilateral programs involving

__________________

1 United States Leadership Against HIV/AIDS, Tuberculosis, and Malaria Act of 2003, P.L. 108-25, 108th Cong., 1st sess. (May 27, 2003).

2 Tom Lantos and Henry J. Hyde United States Global Leadership Against HIV/AIDS, Tuberculosis, and Malaria Reauthorization Act of 2008, P.L. 110-293, 110th Cong., 2nd sess. (July 30, 2008).



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 39
2 Evaluation Scope and Approach CONGRESSIONAL CHARGE As described in Chapter 1, the U.S. government (USG) currently sup- ports programs to combat global HIV/AIDS through an initiative known as the President’s Emergency Plan for AIDS Relief (PEPFAR). This initiative was originally authorized in the U.S. Leadership Against HIV/AIDS, Tu- berculosis, and Malaria Act of 20031 and subsequently reauthorized in the Tom Lantos and Henry J. Hyde United States Global Leadership Against HIV/AIDS, Tuberculosis, and Malaria Reauthorization Act of 2008 (here- inafter, the Lantos-Hyde Act of 2008).2 A description of the history and evolution of the PEPFAR initiative can be found in Chapter 1. In the Lantos-Hyde Act of 2008 which reauthorized PEPFAR, the U.S. Congress mandated that the Institute of Medicine (IOM) conduct a study that includes “an assessment of the performance of United States-assisted global HIV/AIDS programs” and “an evaluation of the impact on health of prevention, treatment, and care efforts that are supported by United States funding, including multilateral and bilateral programs involving 1  United States Leadership Against HIV/AIDS, Tuberculosis, and Malaria Act of 2003, P.L. 108-25, 108th Cong., 1st sess. (May 27, 2003). 2  Tom Lantos and Henry J. Hyde United States Global Leadership Against HIV/AIDS, Tuberculosis, and Malaria Reauthorization Act of 2008, P.L. 110-293, 110th Cong., 2nd sess. (July 30, 2008). 39

OCR for page 39
40 EVALUATION OF PEPFAR joint operations.”3 The legislation further specified that the study include the following: (i) an assessment of progress toward prevention, treatment, and care targets; (ii) an assessment of the effects on health systems, including on the financing and management of health systems and the quality of service delivery and staffing; (iii) an assessment of efforts to address gender-specific aspects of HIV/AIDS, including gender-related constraints to accessing ser- vices and addressing underlying social and economic vulnerabili- ties of women and men; (iv) an evaluation of the impact of treatment and care programs on 5-year survival rates, drug adherence, and the emergence of drug resistance; (v) an evaluation of the impact of prevention programs on HIV incidence in relevant population groups; (vi) an evaluation of the impact on child health and welfare of in- terventions authorized under the Act on behalf of orphans and vulnerable children; (vii) an evaluation of the impact of programs and activities authorized in the Act on child mortality; and (viii) recommendations for improving [United States-assisted global HIV/AIDS] programs.4 PLANNING PHASE FOR THE EVALUATION In the first phase of the study, the IOM formed a multidisciplinary ad hoc committee to develop a strategic plan for the assessment and evalu- ation of U.S.-supported global HIV/AIDS programs as requested in the Lantos-Hyde Act of 2008. In developing the plan, the planning committee engaged in deliberations through three in-person meetings, two committee teleconferences, and telephonic and electronic communications as needed among working groups composed of subsets of the committee. To inform these deliberations, the planning committee held public sessions to solicit input and gather information from a broad range of stakeholders involved in and affected by PEPFAR. Delegations from the planning committee and IOM project staff also held information-gathering meetings with a 3  Ibid., at §101(c), 22 U.S.C. 7611(c). 4  Supra, note 2 at §101(c), 22 U.S.C. 7611(c)(2)(B).

OCR for page 39
EVALUATION SCOPE AND APPROACH 41 range of global stakeholders, including the Global Fund, the Joint United Nations Programme on HIV and AIDS (UNAIDS), the World Health Or- ganization (WHO), and the United Nations Children’s Fund (UNICEF) to discuss potential data sources, methodologies, and lessons learned from experiences with large-scale evaluations. The planning committee and staff also explored potential data sources for the evaluation by consulting and reviewing a range of resources, including documents from the Office of the U.S. Global AIDS Coordinator (OGAC) and other bilateral and multilat- eral agencies, relevant published literature on PEPFAR and global HIV/ AIDS, available literature on large-scale program evaluation, and commu- nications with staff from OGAC, implementing partners, and multilateral stakeholders. The planning committee used the information gathered to assess the methods and anticipated data sources that could potentially be employed to respond, to the extent possible, to the charge in the statement of task. The planning committee focused on identifying data and methodology that would be robust, available, feasible, and appropriate. This was a preliminary exploration of the identified data sources, carried out within the time and resources available for the planning phase. The planning committee could not make conclusive determinations about the suitability of some data sources and therefore, the feasibility of some methodological approaches. Therefore, as described in the sections that follow and through- out the report, a more thorough examination and assessment to make these determinations was carried out as data were requested, reviewed, and col- lected in the subsequent phases of the evaluation. Through this information gathering and deliberation, the planning committee developed a conceptual framework for the evaluation that was based on both the committee’s expertise and current standards in evaluation methodologies for large-scale programs. The planning phase culminated with the publication of a report describing this conceptual framework and the proposed strategic approach to the evaluation, taking into consideration the requirements for the congressional mandate (IOM and NRC, 2010). The following sections of this chapter describe how the evaluation itself was subsequently implemented. To conduct the evaluation, the IOM convened an evaluation committee whose members represented the ap- propriate expertise for the evaluation scope and approach as defined and articulated in the planning phase. There was significant overlapping mem- bership between the planning committee and the evaluation committee. More information about the members of the evaluation committee can be found in Appendix D.

OCR for page 39
42 EVALUATION OF PEPFAR INTERPRETATION OF THE CHARGE Scope of the Evaluation As part of the planning phase, the scope of the evaluation was deter- mined based on the planning committee’s interpretation of the legislatively mandated statement of task. The primary content areas for the evaluation were specified in the Lantos-Hyde Act of 2008.5 As described above, this legislative mandate requested an assessment of the performance of PEPFAR and an evaluation of the impact on health of PEPFAR’s prevention, treat- ment, and care efforts. More specifically, the mandate requested that the evaluation include an assessment of progress toward meeting PEPFAR’s performance goals and targets, which are laid out in the legislation and the PEPFAR Five-Year Strategies (OGAC, 2004, 2009b). The mandate also spe- cifically requested that the evaluation of PEPFAR include the impact of HIV treatment, care, and prevention programs; the effects on health systems; the efforts to address gender-specific aspects of HIV/AIDS; and the impact of programs on child health and wellbeing. In the Strategic Approach, the planning committee identified three additional content areas that were not explicitly identified in the legisla- tion but were determined to be critical elements underlying the assessment of the specific content areas requested by Congress. First, the committee deemed it important to review PEPFAR funding in order to determine the level of PEPFAR’s investment and to gain insight into how financial sup- port for programs and activities has been determined and distributed over time. Second, it found it essential to assess PEPFAR’s progress in tran- sitioning to a more sustainable response in partner countries, given that this was a major goal set forth in the Lantos-Hyde Act of 2008 and the second PEPFAR Five-Year Strategy. Finally, the committee determined that assessing the performance of PEPFAR’s activities to collect, manage, use, and share data, information, and knowledge was an important evaluation component because it is critical not only to the IOM’s evaluation process but also to PEPFAR’s own ability to successfully monitor and evaluate the activities and effects of its programs as well as to guide policies, priorities, and programmatic decisions (IOM and NRC, 2010). The legislative mandate also requested recommendations for improv- ing the USG’s bilateral programs as part of the U.S. response to the global HIV epidemic. Informed by its findings with regard to PEPFAR’s progress toward its stated goals and the effects of the supported programs on health, the overall aim of the evaluation committee in its major conclusions and recommendations was to be forward-looking and anticipate the evolution 5  Supra, note 2 at §101(c), 22 U.S.C. 7611(c).

OCR for page 39
EVALUATION SCOPE AND APPROACH 43 of the U.S. response to global HIV and therefore, to be positioned to in- form the USG response to key issues under consideration at the time of the report release. Further parameters for the scope of the evaluation were interpreted by the planning committee, informed where needed by clarifications discussed with congressional staff and OGAC. The evaluation was defined as an as- sessment of the performance of PEPFAR and of the contribution of PEPFAR to changes in health outcomes and health impact. As described in more detail below, it is not feasible or appropriate to determine the direct attri- bution of PEPFAR funds to effects on health outcomes because PEPFAR is implemented in partner countries within the complex and diverse context of other funding sources, other HIV and health programs, and other factors that affect health outcomes. As an assessment across the whole of PEPFAR, the evaluation was not intended to be an assessment of or a comparison among specific countries, agencies, programs, or partners. It was also not intended to be an assess- ment of the organizational infrastructure and management of PEPFAR, or a financial audit or assessment; these areas fall under the scope and mandate of other organizations external to PEPFAR that have issued reports of their assessments, including the U.S. Government Accountability Office and the Inspectors General of the Department of Health and Human Services, the U.S. Agency for International Development, and the Department of State (GAO, 2009; OIG, 2008, 2009, 2010). As described in Chapter 1, PEPFAR has provided support to more than 100 countries over time. However, in order to represent the greatest intensity of PEPFAR’s investment, the scope of this evaluation was defined to focus on the 31 partner countries submitting an annual Country Opera- tional Plan (COP) at the time of the initiation of the planning phase for this evaluation in 2009.6 In FY 2011, these 31 countries represented 96 percent of PEPFAR’s planned funding (OGAC, 2011).7 As specified in the chapters 6  The 31 PEPFAR countries submitting Country Operational Plans at the time of the plan- ning phase include the original 15 focus countries (Botswana, Republic of Côte d’Ivoire, Federal Democratic Republic of Ethiopia, Cooperative Republic of Guyana, Republic of Haiti, Republic of Kenya, Republic of Mozambique, Republic of Namibia, Federal Republic of Ni- geria, Republic of Rwanda, Republic of South Africa, United Republic of Tanzania, Republic of Uganda, Socialist Republic of Vietnam, and Republic of Zambia) as well as the following additional countries: Republic of Angola, Kingdom of Cambodia, People’s Republic of China, Democratic Republic of the Congo, Dominican Republic, Republic of Ghana, Republic of India, Republic of Indonesia, Kingdom of Lesotho, Republic of Malawi, Russian Federation, Republic of the Sudan, Kingdom of Swaziland, Kingdom of Thailand, the Ukraine, and the Republic of Zimbabwe. 7  Planned/approved funding as reported in the FY 2011 PEPFAR Operational Plan. See Chapter 4 for more information on PEPFAR funding.

OCR for page 39
44 EVALUATION OF PEPFAR that follow, in some cases, the data presented in this report represent only a subset of these COP countries. The legislative mandate describes an assessment of programs and ef- forts “that are supported by United States funding, including multilateral and bilateral programs involving joint operations.”8 This was clarified by congressional staff as a request to focus on the performance and impact of bilaterally funded PEPFAR programs, including those activities that are operated jointly with both bilateral funding through PEPFAR and funding through the Global Fund, which also receives a substantial proportion of its funding from the USG.9 As described in the Strategic Approach, “con- sistent with the clarified congressional intent, U.S. contributions to the Global Fund that are not a part of activities jointly funded or implemented by PEPFAR will not be the focus of the evaluation, and the evaluation will not compare the performance of bilateral PEPFAR programs to that of Global Fund programs” (Bressler, 2009; IOM and NRC, 2010, p. 19; Marsh, 2009). The new U.S. Global Health Initiative (GHI) was launched subsequent to the Lantos-Hyde Act, which mandated this evaluation (OGAC, 2009c). The scope of this evalaution does not include an evaluation of the GHI itself or of the incorporation of PEPFAR’s strategic cumulative goals within the GHI as a comprehensive U.S. global health policy approach (DoS, 2010). Timeframe of the Evaluation The evaluation encompasses PEPFAR’s efforts since PEPFAR funding first became available in 2004 (OGAC, 2005a). The timeframe of the data collected and assessed by the evaluation committee varies by data type and data source. This is described in brief in the methods section of this chapter and in Appendix C, and details are also given at points throughout the report where the analysis and interpretation of the data are presented. The majority of data collection was completed before June 2012; however, some primary data collection through interviews was conducted as late as September 2012. Data requested from OGAC, implementing agencies, and implementing partners were received as late as October 2012. Recent de- velopments in PEPFAR that were introduced since the main data collection period could not feasibly be assessed by the committee, although key recent developments are noted for context in Chapter 1 and in relevant content areas of subsequent chapters of this report. 8  Supra, note 2 at §101(c), 22 U.S.C. 7611(c)(2)(A)(ii). 9  Personal communications from Congressional Staff of the U.S. House Committee on For- eign Affairs and U.S. Senate Committee on Foreign Relations and OGAC, 2009b.

OCR for page 39
EVALUATION SCOPE AND APPROACH 45 OPERATIONAL PLANNING PHASE The Department of State, as the study sponsor, agreed contractually that a transitional period for operational planning should take place be- tween the delivery of the report describing the strategic plan and the imple- mentation of the evaluation itself, which began in the fall of 2010. During this operation planning phase, OGAC partnered with the IOM to dissemi- nate information about the purpose and process of the evaluation and to facilitate introductions to field, headquarters, and agency staff. The primary purpose of the operational planning phase was for IOM staff, planning committee members, and consultants to carry out activities to inform and prepare for the implementation of the evaluation. The operational plan- ning activities focused on data mapping (to continue to identify and assess sources and availability of relevant data); mapping of methods and data sources, including key indicators, to the mandated evaluation tasks; devel- oping procedures for data requests; initiating data requests; designing and initiating data quality review methods for data collected directly or received from outside sources; preparing background materials; and continued re- lationship building with relevant stakeholders such as contacts in PEPFAR countries and at implementing partner organizations. In addition, a major focus of the operational planning phase was to develop and refine processes, frameworks, methods, and instruments for qualitative data collection. This also included early planning of logistics for field work and training for IOM staff by expert consultants in qualitative methods and the use of qualitative analytical software. Additional pilot testing and refinement of field research methods and data collection instruments occurred during pilot visits to two PEPFAR countries, which took place in late 2010 and early 2011. CONCEPTUAL FRAMEWORK FOR THE EVALUATION The following section describes the conceptual and methodological ap- proach taken for evaluating the performance and impact of PEPFAR, while reiterating the context for reasonable and appropriate expectations for an evaluation of this kind as originally articulated in the report of the strategic approach to the evaluation (IOM and NRC, 2010). Program Impact Pathway The planning committee developed an overall conceptual framework that was subsequently used to carry out the evaluation. In this framework, a program impact pathway guided the assessment of the contribution of PEPFAR. The program impact pathway illustrates how PEPFAR-supported programs are intended to ultimately translate into health impact. It rep-

OCR for page 39
46 EVALUATION OF PEPFAR resents the theory of change that underlies PEPFAR—in other words, the rationale for how the combination of activities supported by PEPFAR are logically expected to produce intermediate outcomes, which are then expected to collectively contribute, along with programs funded by other sources, to the desired individual and population health impact. The use of a program impact pathway, which is also referred to as a logic model or results chain, is a well-established method for evaluating complex, large- scale development assistance programs and is becoming widely accepted as a standard in the global HIV/AIDS community (IOM and NRC, 2010; Leeuw and Vaessen, 2009; MERG, 2010). Figure 2-1 shows the program impact pathway developed to guide the assessment of PEPFAR. The pathway begins with the investments and other inputs to the program. For PEPFAR, inputs include not only funding and other resources but also strategic planning, programmatic and policy guid- ance, and technical assistance. These inputs support activities to provide services and support to children, adolescents, and adults in need. Although services are described by PEPFAR in categories like prevention, treatment, and care and support, the conceptual framework acknowledges that they are part of an interrelated and overlapping approach, which also includes activities around gender issues and capacity building. These activities result in outputs that are measurable proximal effects. When PEPFAR-supported programs are implemented well, these outputs are expected to produce outcomes as intermediate effects on the pathway to the ultimate goal of health impact. These intermediate outcomes include, for example, the de- livery of high-quality, efficient services that are available and accessible to the targeted populations and that are achieving the intended and appropri- ate coverage. Other target outcomes include, for example, health systems strengthening; changes in individual risk behavior; and changes in knowl- edge, norms, and attitudes that affect sexual behavior, stigma, and gender issues. Ultimately, PEPFAR-supported programs are intended, through this pathway, to contribute to an impact on individual and population health and well-being, including HIV incidence, HIV prevalence, morbidity, and mortality (IOM and NRC, 2010). Among the inputs to this program impact pathway is the evolving evidence base. This is derived from evidence gener- ated outside of PEPFAR and used to inform PEPFAR-supported programs as well as from data, information, and other forms of knowledge that are generated through PEPFAR-supported activities in monitoring, evaluation, epidemiological data collection, and research, and through the experiences of those implementing PEPFAR-supported programs. Although not directly represented in the pathway shown in Figure 2-1, this knowledge is also an output of PEPFAR-supported activities that underlies the rest of the path- way by serving to monitor, inform, and improve the supported programs.

OCR for page 39
Inputs Activities Outputs Outcomes Impact (proximal effects) (intermediate effects) (distal effects) Strong Health Capacity Strategies Building Systems Individual and Population People Health Impact Trained Service on: Guidance Delivery: Prevention Goods Integration Purchased/ Pyschosocial Coverage Well-Being Planning & Distributed Quality Modeling Cost Efficiency Treatment Services HIV Incidence Access Provided Evolving Evidence Base HIV Beneficiaries Prevalence Served Changes in: Care and Funding & Support Resources Laws and Knowledge Morbidity Policies Attitudes Implemented Behaviors Technical Norms Mortality Addressing Assistance Gender Process & Implementation Results FIGURE 2-1 Program impact pathway for evaluation of PEPFAR’s effects on HIV-related health impact for children and adults. SOURCE: IOM and NRC, 2010. 47

OCR for page 39
48 EVALUATION OF PEPFAR By identifying intermediate steps, the program impact pathway allowed for the evaluation to consider more than the starting point of the PEPFAR investment and the ultimate endpoint of impact on health. Rather, this framework supported an assessment of the performance of PEPFAR along the full range of its implementation and its intended effects. Although it was a major challenge to directly assess health impact, the evaluation com- mittee was able to use the framework of the program impact pathway to state credible findings about the effects of PEPFAR-supported programs. Assessment of Contribution Although it provides a critical guide for the evaluation, the program impact pathway is of course a simplified view of the implementation of PEPFAR-supported programs. Of particular importance is the reality that PEPFAR-supported programs in partner countries operate within the con- text of a wide range of other factors that affect implementation as well as health outcomes (see Figure 2-2). These other factors include the presence of HIV programs supported through the Global Fund and other external and partner country funding sources, as well as other health and devel- opment programs funded through both the USG and other sources. As described in the strategic approach to the evaluation, “Investments from a range of other sources support programs that are aimed at the same desired outcomes [as PEPFAR], and the proportion of total HIV/AIDS support that is provided by PEPFAR varies from country to country. In some cases, multiple funding sources may be co-mingled to support the same programs. Therefore, changes in population health that can be used to reflect program impact cannot be separated by specific programs or investments. Even individual measures can be difficult to attribute directly, as an individual or household may be receiving different services from different programs funded through different sources, all of which have an impact on the health outcomes of the beneficiary” (IOM and NRC, 2010, p. 25). In addition to the influence of other health and development programs, health outcomes are also influenced by cultural, societal, geographical, and political factors that vary by country and are not within the control of PEPFAR-supported programs. In addition, as noted in the Strategic Approach, “As PEPFAR programs increasingly operate with an emphasis on country ownership and harmonization with national plans, the extent to which central USG guidance and authority can influence all levels of priority setting, decision making, and implementation can be quite limited” (IOM and NRC, 2010, p. 25). Ultimately, with a foreign assistance program that is implemented as broadly as and on the scale of PEPFAR, there is not an appropriate comparison available to allow direct attribution of outcomes based on what would have happened in the absence of the investment.

OCR for page 39
EVALUATION SCOPE AND APPROACH 49 Other Donors HARMONIZATION PEPFAR Multilaterals “Three Ones” USG Bilaterals Others Partner Countries • Strategies and policies • Public spending CONTEXTUAL BACKGROUND FACTORS: CULTURE, EPIDEMIOLOGY, POLITICS CONTEXTUAL BACKGROUND FACTORS: CULTURE, EPIDEMIOLOGY, POLITICS • Regulation Total Inputs Private Sector Public Sector Health Other Public Sectors System • NGOs/Nonprofits • Institutional capacity • Child welfare • Firms/For-profits (including M&E) • Primary, Secondary, • Academia • Public goods and Higher Education • Human workforce • Public goods • Financing • Food security quality, cost, access quality, cost, access quality, cost, access AIDS prevention, Other health treatment, and care services Communities Cultural and Social Norms Households and Families Rural and Urban Gender Settings Individuals Income At-Risk Behavior Risk Assets Populations HEALTH IMPACT FIGURE 2-2 Context for PEPFAR contribution in partner countries. NOTES: M&E = monitoring and evaluation; NGOs = nongovernmental organizations; USG = U.S. government. SOURCE: IOM and NRC, 2010.

OCR for page 39
52 EVALUATION OF PEPFAR TABLE 2-1 Country Visit Interviews by Stakeholder Type Stakeholder Type and Sub-Type Number of Interviews U.S. Government (USG) Stakeholders 147 Mission Leadership In-Briefings and Exit Meetings 26 PEPFAR All-Staff Mission Team Briefings 16 PEPFAR Country Coordinator 13 Agency Leadership 26 Technical Staff and Working Groups 66 U.S.-Based Stakeholders with Operations in Partner Country 62 NGO 41 Academia 11 Private Sector (for-profit) 10 Partner Country Stakeholders 156 Government, National 53 Government, Sub-National (province, district, facility) 40 NGO 51 Academia 6 Global Fund CCM 4 Private Sector (for-profit) 2 Other Stakeholders 16 NGO (other country-based) 4 Other Bilateral Government Donors 1 Multilateral 11 Mixed (Stakeholders from USG, Multilateral Organizations, Other 2 Bilateral Donors, Partner Country Government, U.S. Private Sector) TOTAL 383 NOTE: This does not represent the total number of interviewees, because the majority of interviews were with groups of interviewees. In some cases, the same interviewees participated in multiple interviews. For example, there was usually participant duplication between the PEPFAR all-hands interview and subsequent USG interviews. Repeat participation also happened occasionally across multiple interviews with partner country governments. CCM = country coordinating mechanism; NGO = nongovernmental organization. that work in the global response to HIV, such as multilateral organizations, NGOs, and another bilateral donor. As with the country visit interviews, non-country visit interviewees were selected through purposeful sampling, prioritized based on targeted focus areas within the evaluation and on the process of mapping data sources for evaluation questions. Interviews were conducted using the same methodology as for country visit interviews, utilizing semi-structured interview guides with questions and prompts ap- propriate to the interviewee(s).

OCR for page 39
EVALUATION SCOPE AND APPROACH 53 Secondary Data Sources The secondary data sources used in this evaluation included financial data, programmatic monitoring data, clinical data, global indicator data, and publicly available documents. These are described briefly here, with additional information provided where data are presented in the subse- quent chapters as well as in Appendix C. Financial data were received from OGAC, extracted from publicly available PEPFAR documents, and gath- ered from other external sources including the Organisation for Economic Co-operation and Development and the Center for Global Development. PEPFAR’s program monitoring indicators were received from OGAC, and additional clinical data representing programs implemented by Track 1.0 partners were received from the CDC. Some additional data analyses were provided directly by one Track 1.0 partner. Another source of information was global indicator data, primarily from UNAIDS. Finally, document review drew upon a wide range of publicly available sources, including PEPFAR documents, reports from PEPFAR-supported activities and evalu- ations, reports from organizations external to PEPFAR, and published literature. Data Analysis and Interpretation As described in more detail in Appendix C, primary and secondary data were analyzed, using appropriate methodologies, by the members of the evaluation committee, the study staff team, and consultants with specialized knowledge in both qualitative and quantitative methodologies. The committee, staff, and consultants took steps to assess and ensure the quality and completeness of the data used for the evaluation, and they took into account these factors in the interpretation of the data. The methods used to assure the quality of the primary data collected by the committee and the secondary data received through data requests are described in Appendix C. When existing data analyses were used, the committee and consultants reviewed and assessed the methodology and quality of the data in the original analyses. The mandate of the committee was to draw conclusions and to make recommendations across the whole of the program. Wherever possible, data presentations, analyses and interpretation are presented in this report across all of the 31 PEPFAR partner countries defined as the focus of the evalu- ation. However, data sources with comparable and comprehensive data across all of these countries were very limited. To ensure that this constraint would not overly limit the scope of the evaluation findings, the committee also identified subsets of countries and components within programmatic areas for which more robust data could be gathered to contribute to the

OCR for page 39
54 EVALUATION OF PEPFAR PEPFAR-Wide COP Countries (n=31) Select Country Subsets Globally Reported Indicator Data Interview Data (n=13) Interview Data • National prevalence and • Country visit interviews & site visits • USG HQ interviews coverage data • Exit syntheses • Global stakeholders Data Collection Financial Data Financial Data • Planned/approved funding to prime Financial Data • Global donor data partners (n=13) • Obligations and outlays • Planned/approved funding • Planned/approved funding Program Monitoring/Clinical Data Program Monitoring Data • Track 1.0 Partner data (n=13) • Centrally reported PEPFAR Document Review indicators • Legislation and Five-Year Document Review Strategies • Reports and published literature • Programmatic guidance Document Review • USG Agency and Track 1.0 Partner • PEPFAR reports • Country Operational Plan publication lists • External reports guidance • Country Operational Plans Interim Iterative Analyses and Interpretation Within Each Data Type Interim Iterative Interpretation and Triangulation Across Data Types Whole of PEPFAR Interpretation and Deliberation Final Report FIGURE 2-3 Overall data collection and analysis process. assessment. Therefore, some of the data presentations and analyses in this report represent only a subset of countries and were interpreted with care to inform conclusions about the whole of the program. For example, analysis of country visit interview data was limited to the countries selected for visits by the committee. In addition, some analyses drew on existing data sources that were available only for some countries, programs, or partners. Some evaluation questions were most applicable only for a subset of countries, such as countries with concentrated epidemics driven by injection drug use. Finally, the time and resources available limited the scope of some analyses, such as those involving review of COPs for which the sheer volume of the documents over all countries and years limited the feasibility of comprehen- sive review across all countries. Throughout the report, where data analyses that do not represent the whole of the program are presented, the scope of these data is described. Because the committee was not charged to draw conclusions or to make recommendations at the level of specific countries, partners, or programs, analyses of data from subsets of countries or part- ners are presented in a manner designed to maintain anonymity.

OCR for page 39
EVALUATION SCOPE AND APPROACH 55 In summary, the committee applied a mix of methods and layers of investigation and analysis using a range of available primary and secondary data sources, summarized in Figure 2-3. Using this approach, the committee was able to arrive at findings that could be triangulated to draw conclu- sions about the performance and impact of PEPFAR even when any one data source was not sufficient or any one methodological approach was not feasible. Building on the interpretation of the available data, the conclu- sions and recommendations presented in this report represent the consensus reached through the deliberations of the evaluation committee. OVERARCHING EVALUATION CHALLENGES AND LIMITATIONS There were a number of challenges to carrying out this evaluation. The overarching challenges and limitations are described here, while more specific challenges and limitations are described in Appendix C and in the subsequent chapters of this report. Limitations to Evaluation Design The Strategic Approach included a robust discussion of different evalu- ation designs and the limitations to applying these designs to this study (IOM and NRC, 2010). In summary, one major limitation for the design of this evaluation of PEPFAR is that it was not feasible to identify an ap- propriate comparison or control, which would typically be the approach used to answer the underlying question of what would have happened if the program had not existed or if it had been implemented differently. A main reason for the lack of an appropriate comparison is that PEPFAR is widely implemented across many partner countries, which were not selected at random but rather for specific strategic reasons. In addition, an ideal evaluation would use a prospective design, in which data for both intervention and comparison groups would be col- lected from the beginning of the evaluation. When it is not possible for ethical reasons or practical considerations to have a comparison group, a prospective design can at least allow for the planning of a before-and- after comparison of the intervention group. However, the timeframe of this mandated evaluation begins with the initiation of PEPFAR-supported programs, which took place before the evaluation was mandated, planned, and carried out. It was not feasible to carry out complex intervention and evaluation designs or new data collection in order to make prospective comparisons within the time period and resources for this evaluation. Therefore, the questions asked in this evaluation can only be answered retrospectively.

OCR for page 39
56 EVALUATION OF PEPFAR Limitations on Data Availability A primary and very concrete challenge to the evaluation was the lim- ited availability of data to address health outcomes and impact across the whole of PEPFAR, a limitation that was revealed by the data mapping and data collection process for this evaluation. The lack of relevant available measures made it difficult, and in some cases impossible, for the evaluation committee to respond directly to aspects of the evaluation as requested in the Lantos-Hyde Act of 2008. The programmatic indicators that are reported centrally to OGAC across the entire PEPFAR program provide only limited answers to the evaluation charge. There are only nine indicators that are routinely reported centrally to OGAC and that have had stable, consistent indicator defini- tions since the inception of PEPFAR (see Table 2-2). Therefore, these are the indicators that are available across the whole scope of countries and dura- tion of PEPFAR. These indicators represent limited aspects of PEPFAR’s programmatic areas. They also primarily represent outputs, which can serve to assess program implementation through the volume of services provided, but are limited in terms of outcomes and impact to assess those services in the context of the population in need, to assess the quality of the services provided, and to assess PEPFAR’s effectiveness in achieving measurable effects on health. Most evaluation questions required the evaluation committee to draw on data that went beyond the indicators that are routinely reported to OGAC. Data from PEPFAR beyond the centrally-reported indicators, such as recommended indicators collected by country programs but not reported to OGAC, data collected independently by the major USG implementing agencies and other implementation partners, financial data by type of part- ner and expenditures by program activity, results of PEPFAR-supported evaluations, and publications from PEPFAR-supported programs are not managed through processes that allow for ready cataloguing or ready ac- cess to what is available. Accessing these data comprehensively would have required a more intensive and significant data-mapping, data-gathering, and data-analysis effort than was possible given the time and resources avail- able for the IOM evaluation. The necessary requests from the IOM also would have imposed a significant burden of time and resources on staff at OGAC and other implementing agencies as well as on mission teams and implementing partners while they simultaneously continued to oversee and implement the program. Therefore, data requests and data gathering were done strategically within the limitations of what could be responded to and completed in a timely manner. In addition to challenges related to feasibility, for some implementing partners, concerns about sacrificing the

OCR for page 39
EVALUATION SCOPE AND APPROACH 57 TABLE 2-2 PEPFAR Indicators Consistent Across the Duration of PEPFAR NGIs Reporting Area Indicator Definition Indicator Level Care Number of HIV-positive patients in PEPFAR Output HIV care or treatment (pre-ART or ART) who started TB treatment Health Systems Strengthening Number of testing facilities PEPFAR Output (laboratories) with capacity to perform clinical laboratory tests Prevention Number of HIV-positive PEPFAR Output pregnant women who received antiretrovirals to reduce risk of mother-to-child-transmission Percent of HIV-positive National Outcome pregnant women who received antiretrovirals to reduce the risk of mother-to-child-transmission Number of individuals who PEPFAR Output received testing and counseling services for HIV and received their test results Treatment Number of adults and children PEPFAR Output with advanced HIV infection newly enrolled on ART Percent of adults and children with PEPFAR Output advanced HIV infection receiving antiretroviral therapy Percent of adults and children with National Outcome advanced HIV infection receiving antiretroviral therapy Percent of adults and children PEPFAR Outcome known to be alive and on treatment 12 months after initiation of antiretroviral therapy NOTE: Indicator level classified according to 2009 Next Generation Indicator Guidance. ART = antiretroviral therapy; NGIs = Next Generation Indicators; TB = tuberculosis. SOURCES: OGAC, 2005c, 2007f, 2009e. right to first publication also represented a barrier to data sharing for the evaluation. More information about the PEPFAR-specific data considered, requested, and used by the committee can be found in the more detailed description of the methods in Appendix C. The PEPFAR indicators are also discussed in Chapter 11 on PEPFAR’s knowledge management.

OCR for page 39
58 EVALUATION OF PEPFAR Some global data sources available through multilateral organizations, such as UNAIDS, contributed to the committee’s assessment, but these data are not PEPFAR-specific, which limited their utility in evaluating PEPFAR’s effects. In general, for efforts to collect similar data across multiple coun- tries there remains variability by country in the quality and availability of data. Also, in some critical areas that are increasingly a part of PEPFAR- supported programs, such as gender-related efforts, policy efforts, health systems strengthening, capacity building, technical assistance, and bench- marks for sustainability and country ownership, consensus measures have not been developed or implemented either globally or within PEPFAR, and therefore are not available systematically across countries. Challenges with assessing effects in these areas are discussed in more depth in the relevant subsequent chapters of this report. As described above, several sources of data were available for only a subset of countries. The data collected for this evaluation through semi- structured interviews were extensive and systematic, yet the country-level data from these interviews were limited to a subset of 13 PEPFAR partner countries. Available publications provided some useful data for the as- sessment; however, they did not capture information across the whole of PEPFAR but instead represented different subsets of countries and pro- grammatic areas. Given the considerable heterogeneity in PEPFAR imple- mentation across various countries and programs, using data not collected systematically to represent all PEPFAR countries limited the evaluation committee’s ability to generalize findings to the whole of the program and required careful analysis and interpretation, especially because the commit- tee was not charged to draw conclusions at the level of countries, partners, or programs. In summary, the extent to which the goals of this evaluation were met depended on the timely availability of relevant data. As a result, the data used in this evaluation came from a range of disparate sources, and the availability depended in part on the feasibility of access within the evalu- ation’s timeframe. There were, therefore, challenges of interpretation due to heterogeneous data sources with different sampling frames and different data collection systems and criteria, as well as the potential for reporting bias in the responsiveness to data requests from the committee. Chapter 11 presents a discussion of the collection and use of data and information to assess and improve PEPFAR programs and activities, including a forward-looking framework for knowledge management and suggestions for how to develop the means to answer questions posed in the mandate for this evaluation if they are found to be important for future ongoing evaluation.

OCR for page 39
EVALUATION SCOPE AND APPROACH 59 Timing of the Evaluation This evaluation was conducted early in the implementation of changes to the program in response to the reauthorization legislation10 and the new PEPFAR Five-Year Strategy (OGAC, 2009b). “These changes reflect a pro- gressive transition to a new era of challenges and goals for the program, which include efforts to improve sustainability of the response over time, to enhance coordination with partner governments and other global funding partners, and to support accountable ownership of HIV program delivery by countries themselves. They also reflect efforts to give greater consider- ation to the relationship of PEPFAR to broader health and development needs in partner countries” (IOM and NRC, 2010, p. 21). The timing of this evaluation made it difficult to assess the outcomes or impact of these recently implemented changes. For example, the full effect from some ef- forts to strengthen health systems might not be realized for several years or even decades, such as the training and retention of new health care workers or the strengthening of health information systems. There will be a similarly long timeframe required to assess the effects of recently instituted processes being implemented by PEPFAR, in partnership with partner countries, to increase sustainability and country ownership. Nonetheless, the evaluation assessed efforts in these areas in order to understand whether PEPFAR is making reasonable progress toward these new goals and to lead to recom- mendations for how the program can be improved to ensure that these evolving goals can be met. PEPFAR is dynamic, and even as the evaluation was being carried out, it continued to evolve with new goals, new guidance, and new efforts and activities, within the context of newly available evidence. This change in the program over time is a beneficial necessity, but makes evaluation difficult as it presents a “moving target” during the timeframe of the evaluation. Recent changes and new initiatives are not a part of the core content and scope of the evaluation, which was focused on PEPFAR as implemented under the Lantos-Hyde Act of 2008 and the second Five-Year Strategy, but they are acknowledged where relevant throughout the report and serve as context for the ultimate major messages and recommendations. SUMMATION PEPFAR is large, multifaceted, and complex, and it supports a wide range of activities that are carried out by many different partners in a diverse group of countries alongside programs supported by other funders that share the same ultimate aim. Through the conceptual framework of 10  Supra, note 2.

OCR for page 39
60 EVALUATION OF PEPFAR the program impact pathway and contribution analysis described above, the IOM endeavored to conduct a rigorous assessment of PEPFAR that took into account the complexities of implementation and that maintained the flexibility necessary to adapt to the information gathered as the evalu- ation proceeded and to the programmatic evolution occurring within the evaluation timeframe. To conduct a rigorous and thorough assessment, given the limitations, the evaluation committee used a mixed methods ap- proach guided by the program impact pathway framework, drawing on a range of available quantitative and qualitative data sources and using a combination of analytical techniques appropriate to each type of data. By assessing convergence and consistency among different yet complementary data sources and methods, each with different strengths and limitations, the evaluation committee was able to triangulate or cross-examine findings to support reasonable conclusions. When taken together, the totality of evidence allowed the evaluation committee to make recommendations for the program as a whole. REFERENCES Bressler, S. 2009. Presentation. Paper read at First Committee Meeting on Planning the Assesment/Evaluation of HIV/AIDS Programs Implemented Under the U.S. Global Lead- ership Against HIV/AIDS, Tuberculosis, and Malaria Reauthorization Act of 2008, Washington, DC, November 23. DoS (U.S. Department of State). 2010. Implementation of the Global Health Initiative: Con- sultation document. Washington, DC: DoS. GAO (U.S. Government Accountability Office). 2009. Foreign aid reform: Comprehensive strategy, interagency coordination, and operational improvements would bolster current efforts. Washington, DC: GAO. IOM and NRC (Institute of Medicine and National Research Council). 2010. Strategic ap- proach to the evaluation of programs implemented under the Tom Lantos and Henry J. Hyde U.S. Global Leadership Against HIV/AIDS, Tuberculosis, and Malaria Reauthori- zation Act of 2008. Washington, DC: The National Academies Press. Leeuw, F., and J. Vaessen. 2009. Impact evaluations and development NONIE guidance on impact evaluation. Washington, DC: Network of Networks for Impact Evaluation. Marsh, P. 2009. Presentation. Paper read at First Committee Meeting on Planning the Assessment/Evaluation of HIV/AIDS Programs Implemented Under the U.S. Global Leadership Against HIV/AIDS, Tuberculosis, and Malaria Reauthorization Act of 2008, Washington, DC, November 23. MERG (UNAIDS M&E Reference Group). 2010. Strategic guidance for evaluating HIV pre- vention programmes. Geneva: UNAIDS. OGAC (Office of the U.S. Global AIDS Coordinator). 2004. The President’s Emergency Plan for AIDS Relief: U.S. five-year global HIV/AIDS strategy. Washington, DC: OGAC. OGAC. 2005a. Engendering bold leadership: The President’s Emergency Plan for AIDS Relief. First annual report to Congress. Washington, DC: OGAC. OGAC. 2005b. The President’s Emergency Plan for AIDS Relief: Indicators, reporting require- ments, and guidelines for focus countries. Washington, DC: OGAC.

OCR for page 39
EVALUATION SCOPE AND APPROACH 61 OGAC. 2007. The President’s Emergency Plan for AIDS Relief: Indicators, reporting require- ments, and guidelines. Indicators reference guide: FY 2007 reporting/FY 2008 planning. Washington, DC: OGAC. OGAC. 2009a. The President’s Emergency Plan for AIDS Relief: Next generation indicators reference guide. Version 1.1. OGAC: Washington, DC: OGAC. OGAC. 2009b. The U.S. President’s Emergency Plan for AIDS Relief: Five-year strategy. Washington, DC: OGAC. OGAC. 2009c. The President’s Emergency Plan for AIDS Relief: Five-year strategy. Annex: PEPFAR’s contribution to the Global Health Initiative. Washington, DC: OGAC. OGAC. 2011. The U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) FY 2011: PEPFAR operational plan. Washington, DC: OGAC. OIG (Office of Inspector General). 2008. Review of the office of the U.S. Global AIDS Co- ordinator. Washington, DC: U.S. Department of State and the Broadcasting Board of Governors. OIG. 2009. The exercise of chief of mission authority in managing the President’s Emergency Plan for AIDS Relief overseas. Washington, DC: U.S. Department of State and the Broadcasting Board of Governors. OIG. 2010. Review of the President’s Emergency Plan for AIDS Relief (PEPFAR) at select embassies overseas. Washington, DC: U.S. Department of State and the Broadcasting Board of Governors.

OCR for page 39