This study was carried out by the Institute of Medicine (IOM) Committee on the President’s Emergency Plan for AIDS Relief (PEPFAR) Implementation Evaluation, three subcommittees, several consultants, and staff, all of whom are listed in the front matter. Short biographies for each person can be found in Appendix G. The Committee is composed of 13 members. Committee members were selected for their international experience in low- and middle-income countries, as well as their individual expertise in the following areas relevant to the Committee’s charge: behavioral science, bioethics, biostatistics, community nursing, community development, economics, epidemiology, infectious disease (adult), informatics, maternal and child health, modeling, monitoring and evaluation, operations research, professional training/education, public health program management, quality of care, and social services. Three advisory subcommittees comprised of six to seven members each, are focused on prevention, treatment, and care, including orphans and other vulnerable children. Additional members who serve only on the subcommittees provide expanded expertise in the following areas: child psychology/psychiatry, child welfare/services, demography, health communication, health education, infectious disease (pediatric), laboratory quality, logistics, palliative care, and pharmaceutical care.
The Committee began its work on this report in February of 2005 with its initial meeting and concluded in February of 2007 with finalization of this report in response to review. The Committee employed a wide
range of methods, including information-gathering and deliberative meetings; review of the scientific and other literature; review of PEPFAR and other documentation; analysis of PEPFAR budget and performance data; discussions with a wide range of PEPFAR staff, participants, and stakeholders; and visits to the PEPFAR focus countries. Some Committee members and staff participated in both PEPFAR annual meetings—in Ethiopia in May of 2005 and in South Africa in June of 2006—including the U.S. government–only sessions, as well as the Office of the U.S. Global AIDS Coordinator’s (OGAC) monthly outreach meetings, relevant Congressional hearings, meetings of the Center for Strategic and International Studies Task Force on HIV/AIDS, the Partnership for Supply Chain Management launch meeting on May 24, 2006, in Washington, DC—“Delivering HIV/AIDS Commodities to Customers Insights and Partnerships for Seamless Supply Chains”—and the PEPFAR Gender and HIV/AIDS Consultation on June 1, 2006, also in Washington, DC.
On October 31, 2005, the Committee published its plan for carrying out this short-term evaluation of PEPFAR. The plan outlined the Committee’s evaluation questions and approach to the study (IOM, 2005). This report is included as Appendix C and is also available free of charge at the following web address: http://www.nap.edu/catalog/11472.html.
Evaluating Within the Framework of Harmonization
Figure 1-1 in Chapter 1 summarizes the Committee’s evaluation plan, and shows the focus countries and “harmonization” at the center of the plan. In April 2004, the Joint United Nations Programme on HIV/AIDS (UNAIDS), the United Kingdom, and the United States co-hosted a high-level meeting at which donors reaffirmed their commitment to harmonization—that is, to strengthening national AIDS responses to be led by the affected countries themselves. They endorsed the “Three Ones” as guiding principles to improve the country-level response (UN, 2003; UNAIDS, 2004; OGAC, 2005):
One agreed upon HIV/AIDS Action Framework that provides the basis for coordinating the work of all partners.
One National AIDS Coordinating Authority, with a broad-based multisectoral mandate.
One agreed upon country-level Monitoring and Evaluation System (UNAIDS, 2004).
PEPFAR is fully committed to harmonization and the Three Ones principles (OGAC, 2005). Although these principles were not formally in place at the time of the passage of the Act, the legislation calls for the United States to be coordinated with other donors, and this commitment is consistent with the legislation, as well as other U.S. policy (UN, 2003). All of the PEPFAR focus countries have national AIDS authorities, and thus with this commitment, harmonization became the centerpiece of the structure of PEPFAR (OGAC, 2005). As such, it is central to the structure of the IOM evaluation. The imperative to operate within the framework of harmonization adds complexity to both the implementation of PEPFAR and the evaluation of it—it requires that PEPFAR implementation be judged primarily in terms of how effectively it is working through, with, and in support of countries and whether, in the final analysis, countries are becoming more effective at addressing their HIV/AIDS epidemics. Figure B-1 illustrates an example of this complexity and how the Committee approached the analysis. Examples of the challenges at the nexus of PEPFAR’s imperatives to meet its targets, be harmonized, evidence-based, and in compliance with U.S. law and policy are discussed throughout the report.
The Committee held a total of six meetings, three of which were partly for the purpose of information gathering and thus these portions of the meetings were open to the public. All three of the meetings with open sessions were held in 2005 and involved the full committee. A brief description of each open meeting follows, and the agendas are in Appendix G. Available slide presentations can be viewed on the project website at www.iom.edu/pepfar.
The first meeting was held in Baltimore, MD, with open sessions on February 23 and 24, 2005. The Committee heard from officials from the Office of the U.S. Global AIDS Coordinator, as well as some of the implementing agencies, including the Office of Global Health Affairs at the Department of Health and Human Services, the Global Health Bureau at the U.S. Agency for International Development, the Global AIDS Program at the U.S. Centers for Disease Control and Prevention (CDC), and the HIV/AIDS Prevention Program at the U.S. Department of Defense. The Health Attaché from the Embassy of South Africa (one of the PEPFAR focus countries) and the Senior Program Director and Chief of Party from Catholic Relief Services (one of the initial PEPFAR implementing partners), also addressed the Committee. In addition, the Committee heard from
senior officials from the monitoring and evaluation units of the World Bank (in person), the Global Fund, the World Health Organization (WHO), and UNAIDS (all by phone).
The second meeting was held in Washington, DC, with open sessions on April 19 and 21, 2005. During these sessions, the U.S. Global AIDS Coordinator and other OGAC officials gave presentations to the Committee, as did officials from six of PEPFAR’s implementing partners: the Centre
for Infectious Disease Research in Zambia (CIDRZ), the Elizabeth Glaser Pediatric AIDS Foundation, Family Health International, Haitian Group for the Study of Kaposi’s Sarcoma and Opportunistic Infections (GHESKIO), Harvard PEPFAR, and Save the Children. A panel of Chiefs of Party from the Global AIDS Program at the U.S. Centers for Disease Control and Prevention—representing the programs in Botswana, Democratic Republic of Congo, Uganda, Vietnam, and Zimbabwe—addressed the Committee via videoconference.
The fourth meeting was held in Washington, DC, with an open session on September 15, 2005. The Committee heard from two people who were senior Congressional staff involved in the initial development of the legislation that was eventually passed as the United States Leadership against HIV/AIDS, Tuberculosis, and Malaria Act of 2003 and from several OGAC officials. The meeting included a panel of ambassadors with experience in one or more of the PEPFAR focus countries—including Kenya, Nigeria, South Africa, and Uganda—with additional perspective from the Executive Director, Task Force on HIV/AIDS and Director of the Africa Program at the Center for Strategic and International Studies, who had led delegations to several of the PEPFAR countries. The Committee also heard from a panel of three PEPFAR implementing partners—Pact, CARE, and the International Rescue Committee.
LITERATURE AND DOCUMENTATION
The Committee searched widely for literature relevant to the standards for and implementation of programs to address HIV/AIDS epidemics. The Committee searched and retrieved references primarily from the PubMed and Cochrane bibliographic databases, as well as the websites of key organizations such as WHO, UNAIDS, the United Nations Children’s Fund (UNICEF), World Bank, Global Fund, the Kaiser Family Foundation, the Global Health Council, the Center for Global Development, and the Center for Strategic and International Studies. The Committee reviewed extensive documentation of PEPFAR implementation including the authorizing legislation—the United States Leadership against HIV/AIDS, Tuberculosis, and Malaria Act of 2003—and its legislative history; the websites of OGAC, CDC, the U.S. Agency for International Development (USAID), implementing partners, the U.S. embassies in the focus countries, and the focus country governments and/or lead HIV/AIDS entities; the PEPFAR strategy, operational plans, and first and second annual reports; the 2005 and 2006 Country Operational Plans for all of the focus countries as well as the
guidance and technical considerations for developing these plans; the work plans and technical considerations of all of the Technical Working Groups; the HIV/AIDS strategies and operational plans for all of the focus countries when available; all guidance, reports, and other documents published by OGAC as well as those directly relevant to PEPFAR published by implementing agencies such as CDC and USAID; all of the abstracts and presentations from the 2005 and 2006 PEPFAR Annual Meetings; presentations and Congressional testimony given by PEPFAR officials; and documentation provided by implementing partners and other interested parties.
BUDGET AND PERFORMANCE DATA
With the intention of respecting global efforts at harmonization of monitoring and evaluation—in which PEPFAR is a participant (Rugg et al., 2004; OGAC, 2005)—the Committee relied on existing indicators and data sources to the greatest extent possible. The Committee reviewed and analyzed all publicly available PEPFAR budget data and performance data, as well as information about HIV/AIDS funding, epidemiology, and activities in the focus countries. The primary sources for PEPFAR data were PEPFAR Congressional Notifications, Country Operational Plans, Annual Reports and other interim reports, and analyses of the Country Operational Plan Reporting System provided by OGAC. The primary sources of data on the focus countries were their own websites and publications, OGAC, UNAIDS, WHO, the World Bank, and Kaiser Family Foundation. The Committee did not audit or independently verify these data; however it did some checks for internal consistency as well as congruence with external sources. The Committee did not audit any aspect of the program and thus is unable to address issues such as contract compliance, diversion of funds, or corruption.
DISCUSSIONS WITH PARTICIPANTS AND STAKEHOLDERS
Especially because many of the aspects of program implementation that the Committee reviewed had a significant qualitative component, the Committee sought to have discussions with as many relevant participants and stakeholders as possible. For example, with respect to coordination, in addition to examining whether there were structures and processes in place to facilitate coordination, the Committee also solicited the perspectives of all the major parties that were intended to be coordinated.
Throughout the process—during the development of the evaluation plan, in response to the publication of the plan, in preparation for the visits to the PEPFAR focus countries, and after the visits—the Committee had discussions with a wide variety of people, in Washington, DC, and Geneva, Switzerland, both in person and by phone, including
OGAC and implementing agency officials
Staff of the U.S. Congress
Groups of implementing partners and other stakeholders organized by the PEPFAR Implementer Roundtable of InterAction and the Global AIDS Roundtable of the Global Health Council
Officials of the Global Fund, UNAIDS, WHO, and the World Bank
In order to encourage the participants in these discussions to speak candidly, the Committee members assured them that they would not attribute statements made in these discussions to individuals by name, organization, or country. The people and organizations with whom the Committee had discussions are listed in Appendix A. Discussions were grouped as follows and summarized across the various perspectives provided:
OGAC and implementing agency officials
Miscellaneous implementers and stakeholders
Other HIV/AIDS donors and programs
Focus country visits (see below for description of process and analysis)
In order to facilitate consistency across discussions, the Committee developed generic guides for each type of meeting or site visit to be conducted. These guides were based on the questions outlined in the evaluation plan, identified a standard set of issues to be addressed, and also allowed for additional issues to be explored depending on the circumstance. These guides were tailored to the extent possible based on what the Committee was able to learn in advance about the particular people or program involved.
FOCUS COUNTRY VISITS
As a result of PEPFAR’s structure and commitment to harmonization, the majority of implementation activities are occurring in the focus countries. Thus, visits to the focus countries to directly observe implementation activities were a critical part of the Committee’s evaluation plan. The Committee anticipated that these country visits would provide insight into the programmatic successes and challenges through concrete examples and first-hand accounts of how PEPFAR was working on the ground, and found that they did.
In recognition of the unique nature of the HIV/AIDS epidemic and the country response to it in each of the 15 focus countries, PEPFAR has been designed to support national leadership and to adapt to the specific needs of each country. The Committee therefore thought it important to
observe how the conditions prevailing in each focus country were affecting the implementation of PEPFAR. The Committee—in small delegations of four to seven people—visited 13 of the 15 focus countries between late October of 2005 and late February of 2006. The Committee cancelled a planned visit to Côte d’Ivoire and was unable to plan a visit to Haiti, both due to security concerns. In lieu of a visit, the Committee conducted several conference calls with the Country Teams and implementing partners in Côte d’Ivoire and Haiti. Because the delegations could not be large enough to include all areas of expertise required, all delegation members were expected to familiarize themselves with all of the discussion guides and to serve as “generalists” during the visit. Each country visit lasted for 1 week, giving the Committee a total of 13 weeks in the focus countries. Most of the delegations were split for part of the agenda in order to be able to visit more sites. Of a total of 27 Committee members, consultants, and liaisons, 20 participated in country visits together with five staff. The majority of
Generic Agenda for Focus Country Visits
Purpose: Understand PEPFAR Mission staff’s perspective on program implementation (especially coordination and harmonization)
Purpose: Understand perspective of country officials, other donors, and partners on PEPFAR implementation (especially coordination and harmonization)
Purpose: Examine PEPFAR-supported prevention, treatment, and care programs
people participated in more than one country visit, resulting in a total of 68 person-weeks of observation in the focus countries.
Agendas for the Country Visits
The Committee developed a generic 1-week agenda for the country visits that was intended to provide as comprehensive an overview of PEPFAR implementation as possible in a short time (see Box B-1). The Committee tailored the agenda to the particular circumstances of each focus country in concert with PEPFAR staff, PEPFAR implementing partners, national officials, other donors, community leaders, and others. The agendas included some discussions that were consistent across all countries—such as with the U.S. Country Team and focus country government HIV/AIDS officials, and some that varied widely depending on the country. The Committee selected a purposive sample of sites to both illustrate the breadth of the program in that country and exemplify PEPFAR’s successes and challenges there. The
Purpose: Examine PEPFAR-supported prevention, treatment, and care programs
Purpose: Examine PEPFAR-supported capacity-building efforts
Committee approached the country visits as a continuous learning process, and attempted to leave some flexibility in the agendas to allow the visiting teams to adapt to what they learned as the visit progressed. In practice, because the agendas were ambitious and logistic challenges were numerous, there was little flexibility to be exercised.
Country Visit Process
Prior to each visit, the Committee sent letters to all parties with whom they were scheduled to meet to confirm the appointment, to provide information about the study and the delegation members who would be coming, and to outline the major topics of discussion.
The Committee endeavored to create an environment in which people felt free to speak frankly and thus—in addition to assuring participants that the Committee would not attribute remarks to individuals, organizations, or countries—requested of OGAC that no OGAC staff be present during discussions with the PEPFAR Country Teams and that no PEPFAR staff be present during visits to sites that PEPFAR was supporting. PEPFAR staff honored this request. In a few instances, it was necessary for diplomatic purposes to be introduced by PEPFAR staff, but they left after the introductions had been made.
The Committee explained at every meeting that all materials given to the delegation would have to be made available in the public domain, that is, in our public access file. After each visit was completed, the Committee also sent follow-up letters to every party with whom it had met to thank them and to reiterate that any documents they had given to the delegation would be available in the public domain.
Especially for the first several trips, each delegation was expected to produce a brief report on Thursday night to go to the next delegation on Friday before its visit started. The purpose of the report was to highlight any major considerations or lessons learned with respect to logistics, diplomacy, or the substance of the discussions. In practice, no major unanticipated issues were identified and later delegations did not produce these reports.
Analyses of Country Visit Information
To guide the development and conduct of each country visit, the Committee reviewed and analyzed information available about the country’s demography and general and AIDS-specific epidemiology, as well as national AIDS strategies and plans, PEPFAR focus country profiles, and PEPFAR Country Operational Plans for fiscal year 2005.
Individual Country Visit Analysis
By the end of the visit, each delegation was expected to develop a summary of its consensus on the key observations, findings, and conclusions about the implementation of PEPFAR in that particular country, by synthesizing the information from the many meetings and site visits. For example, after having discussed harmonization with a range of parties with whom PEPFAR is involved, the delegation endeavored to “triangulate” among the various sources to reach a conclusion about how well PEPFAR was harmonizing in that country (see Figures B-1 and B-2). Most teams continued to work on these analyses after the visit was completed.
Cross-Country Visit Analysis
To reach conclusions across the focus countries, the Committee conducted several types of analyses. First, the Committee compiled all of the summary reports by topic or issue. For example, findings related to coordination were pulled from each of the summary reports to provide a view
of PEPFAR’s coordination efforts across all of the focus countries. Second, several committee members who served on more than one country visit developed individual cross-country conclusions based on the unique set of countries that they had visited. Third, the Committee synthesized all of the other analyses as well as notes from all of the visits to produce a set of findings and conclusions about the overall perspective from the focus countries on PEPFAR. Finally, the Committee “triangulated” this synthesis of information from the focus countries with other sources of information—including documentation and other interviews—to reach conclusions about key aspects of PEPFAR implementation such as harmonization (see Figure B-2).
A great deal of information about the focus countries is a matter of public record—for example, information about the nature of their HIV/AIDS epidemics, their national AIDS strategies and sometimes their operational plans, and their PEPFAR Country Operational Plans. When discussing this kind of information in the report, the Committee identifies specific countries by name. However, when discussing information that is based on discussions had in the focus countries, the Committee avoids attribution of comments even by country. As discussed earlier, the Committee assured all parties with whom it spoke that it would not attribute comments and believes that this assurance helped to facilitate candid discussions.
The focus country visits were not designed to allow the Committee to go deeply into and reach definitive conclusions about any one focus country, program, or aspect of implementation—with perhaps the exception of overall PEPFAR management, coordination, and harmonization. Thus, the Committee did not attempt to reach conclusions about specific countries or programs and has not based conclusions about any aspect of PEPFAR solely on the visits. However, the Committee believes that the cumulative information from all of the visits—effectively 13 weeks on the ground in PEPFAR focus countries, discussions with hundreds of people, and visits to dozens of sites—gave it a comprehensive and detailed picture of PEPFAR implementation overall as viewed from the focus countries.
The Committee provides examples of PEPFAR-supported programs throughout the report. The Committee selected these examples from the Country Operational Plans simply to illustrate the types of activities included in the various PEPFAR categories. The Committee neither visited or reviewed the details of all of the programs described in the examples provided nor evaluated any of them.
Given the nature of the enterprise and the kinds of information collected—comprehensive but not exhaustive looks at each country, discussions with people who were not required to provide documentation of
their assertions, site visits that did not necessarily permit the delegations to get “behind the scenes” and were largely dependent on collective personal impressions, a great many compelling sights and people—the Committee endeavored to remain cognizant of its limitations and avoid going beyond these. The Committee sought to grapple with and synthesize the entirety of the information and to guard against the natural human tendency to let one example, or anecdote, or personality carry too much weight in the final analysis. Information from the country visits that is cited in the report relates to points where there was the greatest commonality of observation and agreement across countries and among Committee members.
In compliance with the Federal Advisory Committee Act Amendments of 1997 (P.L. 105-153), all materials presented to the Committee must be publicly available via our public access file. Because OGAC determined that some portions of the Country Operational Plans are procurement sensitive and therefore could not be in the public domain, they provided the Committee with redacted versions of the full Country Operational Plans, in addition to the Country Operational Plan information that is available on the OGAC website. Because the Committee’s interest was not in appreciating the finest level of detail in the Country Operational Plans, it accepted OGAC’s assertion that the missing material would have little effect on the Committee’s ability to develop a general understanding of the programming in a country or its alignment with national plans and priorities. Further, because OGAC considers some draft guidance and other documents to be sensitive until finalized, the Committee has only reviewed those that OGAC will allow into the public domain and thus has not been privy to the guidance development process or guidance which has yet to be finalized.
IOM (Institute of Medicine). 2005. Plan for a short-term evaluation of PEPFAR implementation. Washington, DC: The National Academies Press.
OGAC (Office of the U.S. Global AIDS Coordinator). 2005. The President’s Emergency Plan for AIDS Relief: Defining US support for national strategies (PowerPoint Presentation, July 2005). Washington, DC: OGAC.
Rugg, D., G. Peersman, and M. Carael. 2004. Global advances in HIV/AIDS monitoring and evaluation. Minneapolis, MN: Wiley Periodicals, Inc.
UN (United Nations). 2003. Monterrey Consensus of the International Conference on Financing for Development: The final text of agreements and commitments adopted at the International Conference on Financing for Development, Monterrey, Mexico, 18–22 March 2002. Geneva, Switzerland: UN.
UNAIDS (Joint United Nations Programme on HIV/AIDS). 2004. “Three ones” key principles. Conference paper 1, Washington consultation of April 4, 2004. Geneva, Switzerland: UNAIDS. http://www.unaids.org/NetTools/Misc/DocInfo.aspx?LANG=en&href=http://gva-doc-owl/WEBcontent/Documents/pub/UNA-docs/Three-Ones_KeyPrinciples_en.pdf (accessed September 20, 2006).